Great thinkwork about “agile”: presentation by David Anderson¶

Via Bill de hÓra I found a great 1.5 hour presentation by David Anderson about agile practices and the changes we ought to make in them. Bill: I think maybe this is the best material I've come across on Agile since v2 of Extreme Programming. It floored me. I liked David's presentation so much that I've made a summary of it.

What does it mean to be agile?

Read the agile manifesto and the principles behind it. This stuff is useful for other disciplines than just programming. You cannot learn a marketeer to pair program. Well, you can but they would be doing pair programming, not pair marketing. So get back to the manifesto. What does it mean to be agile? How can we scale it to enterprise scale?

A problem is that the manifesto is written as an article of faith: principles are stated, but not really explained scientifically. So what are the origins and actual basics?

The big heavy contract movement was born out of big government projects where risk aversion was paramount and where trust was low. The agile community was born in a much more liberal culture and partially in the dot-com boom where the turnaround time was quick and errors not-too-dangerous. So it payed off to trust eachother and to be flexible. Two processes, oil and water. They just don't mix. That's the background behind the manifesto's third principle ("individuals and interactions over processes and tools").

There used to be an assumption that to ensure good software, you needed to achieve perfection in specification. And you needed a repeatable process to reliably get that perfect specification and to reliably turn that perfect specification into software. This resulted in, for instance, UML diagrams (UML as such can be pretty useful). But there are a whole lot of diagrams needed to catch all requirements. Often you still need more detail and more diagrams. So you got a huge pile of diagrams and documentation. And the diagrams were, at the time that agile started, not automatically synced by tools with the code but basically just drawings. So they were at the very least partially out of date by the time they were finished. Good specifications sure help, but the process was over the top and the analysis work didn't give any good return on investment anymore. The agile response was that we didn't need 14 layers of diagrams before writing actual working code.

A repeatable process means a well-defined process. Thousands of pages. That's unwieldy, so many teams could not work with that. Which means hiring one of the large consulting firms to ensure the process is adhered to. Problem is that it didn't work. (In the Netherlands there's now again a big 87 million Euro government IT project down the drain: with one of the big consulting firms as the software partner and presumably a big huge spec and a big huge process (what else would you manage to spend 87 million Euro on?)). "Working software over comprehensive documentation" is another agile manifesto value.

So "individuals and interactions over processes and tools" and "working software over comprehensive documentation" really boil down to perfect is the enemy of good enough. That's the background of that part of the manifesto.

In the current business climate, lots of things are changing fast. New competitors, new products, changing environment. So if you have a new idea, it should be executed quickly: short lead time. And your current knowledge and your organization's current knowledge is a perishable good. Like bananas, it goes bad over time. So you need a shorter cycle time. All that combines to the 4th value of the manifesto: "responding to change over following a plan".

Agile basically dumped some 20th century business myths and replaced them. David Anderson summarized the changes that Agile made as follows (copied mostly verbatim from his slides):

• Perfect was the enemy of good enough and we stopped trying to be perfect, accepted that we had to move forward with imperfect information and refactor as we learn more (it's much cheaper, faster and better).
• Self-organization (or: "empowerment, delegation and failure tolerance" as he tells his enterprise clients) through adoption of a high trust culture was better than contracts, commitments, audit and arbitration in a low trust culture.
• System requirements were not assets, they were liabilities. So we spent less time writing things down, relied a lot more on tacit knowledge and focused on reducing cycle times.

Apart from these three he also extracted four more items from the agile manifesto. Together with the above three points, they recast the agile manifesto in somewhat of a model:

• Perfect is the enemy of good enough.
• High trust, high social capital, highly collaborative culture.
• Knowledge work is perishable.
• Reflect and adjust.
• Sustainable pace.
• Craftsmanship.
• (A hint of) wast elimination.

Scaling "Agile" to enterprise scale

The agile community experiences change as people are trying to adapt it to enterprise scale. And we've been adapting Lean techniques. And "just" following the agile practices isn't enough, you need to have organizational maturity. From these three drivers for change, David extracted three he's going to talk about: kanban (part of the "lean" theory), real options and CMMI .

Lean has its own set of principles. Value is the core. Value can change regarding context and time. So deliver on time at the right place. Focus on flow, removing stumbling blocks, lowering inventory (as it is static), etc. Pull limits work-in-progress. Work is "pulled in" as required, one of the advantages is that it prevents overload.

"Relentless pursuit of efficiency" is a core element of western management. Bigger sized work batches (or firms, or tasks) is traditionally a way to get efficiency benefits: economy of scale. Overhead was assumed as inevitable, so they tried to spread it out over as big an area as possible. But, as the mythical man month by Fred Brooks tells us, coordination costs increase non-linearly with batch size in knowledge work as the coordination costs increase so much. In Japan, they simply didn't have the space in the 1950's to go the way of economy of scale. Their answer was to reduce the batch size to reduce the coordination costs. This turned out to be a better answer as it allows you to respond more quickly and capture more value out of the market.

Kanban (literally "signal cards") is a lean solution for value/flow/pull. Cards flow from for instance design to developing to testing. Every phase is limited in number of cards it can hold. So it demolishis some sacred XP practices like the planning game and time-boxed iterations. If you focus on practices instead of paradigms (where the real value lies according to David). He revisited some of the paradigms mentioned above and found that kanban supports all the paradigms. For instance, limited number of cards in a phase means everyone gets to focus on just a few items and that those items HAVE to be good, so it supports good craftsmanship.

One of the ways kanban allows enterprise-scale software engineering is that it allows division of labor. Much more so than regular agile projects where you need very good generalists. Kanban is friendlier in allowing business analists or UI designers to do their specialized jobs. On the other hand, you don't want too much division of labor, as hand-offs mean transaction costs. So you have two opposing forces that ought to keep each other in check. Kanban gives Agile a nice way of embracing division of labour without losing the agile values.

More thinkwork

David thinks more things will change that will force us to embrace lean. Like reusable components and cloud services.

For reusable components and so to work, there has to be a means of exchanging economic value between partners in a value chain (a chain as reusable components by definition have to be tied together, btw). When building a cell phone, you've got to concentrate your efforts. You'll buy a lot of software components without doing any refactoring or so on it. You lose a lot of what agile holds dear there. One way to deal with that is to differentiate the various lifecycles. The telephone subsystem changes only every 5 years or so: just buy it. The user interface on the other hand should be build in the most agile way possible: that's where the money and the diffentiatory power is. Reuse of existing stuff could drive a next wave of lean productivity in the areas where it actually matters most.

Another point: cmmi . Maturity. Levels 2 and 3 have some agile anti-patterns in them which makes cmmi a bad word in most of the agile community. When big enterprises describe what they want out of their software engineering departments, they mostly describe what would be cmmi level 4. They don't need the certification, but they do need/want the associated maturity. Agile teams with their agile processes mostly aim/reach the agile equivalence of levels 2 or 3. 2 is pretty easy for us, 3 is pretty hard. Agile-ers are pretty smart and clever, so we could develop our own maturity model. We could also choose to learn from the experience of others (which is wise). So David actually says the cmmi is pretty good. Even the organization behind cmmi is thinking about agile methods.

Cmmi level 4 has predictability as the big keyword. Predictability could allow a team to be even more agile and to respond to changes faster. Level 4 also means long-term institutional learning. Don't lose knowledge. Don't make the same mistakes twice. Level 5 aims at continuous improvement, which is something the agile community aspires. "Kaizen".

Track raw data about your process, otherwise you won't mature beyond the equivalent of level 3. Quantitative, objective management. You need it. You won't get there with subjective retrospectives. Gather data and use it. David uses operational reviews that are data-driven.

The more mature an organization, the easier it can use agile methods. Nice insight. This means that he focusses more on maturity than on productivity in his consulting jobs.

In David's opinion, kanban is the method that can help us achieve both agility and high maturity. It will push us forwards. It creates a cultural shift. A shift that allows some teams to reach cmmi level 4 certification.

One last thing he talked about was "real option theory". Real options change everything in David's opinion. Project planning/scheduling. Risk management. Portfolio management and governance. "Throw it away and start again". Changes for the better, but still changes. Options have value; options expire; never commit early unless you know why. Push back decisions as far as possible. We like to make decisions: better a wrong decision than indecision. Instead, figure out what the last moment is that you need the decision. You're way more flexible and way more certain of a right decision that way.

Something he mentioned that I want to check out: Linda Rising is hunting the 8 hour day: why are we working according to a workday that was calculated to be optimal for 19th century coal workers?

Embrace management and organizational skills as essential to furthering agile's goals. Programmers can be learned the agile way in a few weeks or months. But if the organization around it doesn't match that it is all for nought.