The Truth or a Comforting Lie?

Long time no blog. I've been distracted in recent months learning about non-geek stuff like investing in the stock market, which I may blog about at some point. But what has inspired me to pick up my virtual pen again is the apparent mainstream rise in popularity of agile development. Bear in mind that I'm in Australia so your time lines may vary, but these days there are very large organisations stating agile development as a key aspiration and sending head hunters around about scrounging for anyone who can spell it correctly to lead them to the promised land. Five years ago we relatively early adopters of eXtreme Programming would band together and tell each other stories of what a wonderful world it would be when major banks and telcos saw the light and changed their evil waterfall ways. Now it seems we may have gotten what we wished for and perhaps we should have been more careful.

Refer to Martin Fowler's recent post on how top-down imposition of agile is pretty un-agile by definition; he's far more calm and articulate then I am. But as someone who has seen agile succeed and fail, I thought I'd throw my opinion into the ring for a bit of perspective for the recently converted. The greatest success with which I have been involved personally was simply awesome. A true agile poster child project: the brochure for it would tell you that we estimated it almost perfectly, delivered it on time and on budget, have had so few bugs in two years of production use that you'd have several fingers left counting them digitally, and the system transformed the business in a truly fundamental way. And we're not talking about a small project here: 20 people all up for 18 months and several million dollars. This is all absolutely true. What the brochure would fail to mention most likely is that we clashed with several internal stakeholders to the point where I was excluded from the building for a week in the middle of the project, that we needed the best team of developers I've ever seen to pull it off, the best project manager I've ever seen, the best iteration manager, customer representative and analysts and we had to fight off any number of die-hard process nazis who did not share our value system of actually delivering something of value (think Vogons). The brochure would also fail to mention that the organisation that witnessed this startling occurrence was quite committed to hiding this anomalous project like an embarrassing relative; you see, we outsiders had been brought in as the third attempt at this project in three years, and were set up to fail. The brochure probably would not mention that either.

Where I've seen agile fail is quite simply when you don't have all of these things working for you. Like having a distracted customer who can't spend the time with the team. Like having developers who really don't care about the craftsmanship of their work. Like having analysts who just type requirements into documents without any actual analysis. Like testers who just do the happy scenarios because the other ones are too hard. Like having a project manager who focuses on avoiding failure rather than achieving success. The problem with agile is that any one of these can severely damage or even kill your project despite the health of the others.

So the good news is that agile can and does work. The bad news is that agile can and does fail. So what are you to do? Looking back on that successful project, what made it work was, unfortunately, probably the same old things that have made any project work in any human endeavour since human endeavours began; highly skilled and committed people who bust a gut, put their necks on the line and get really lucky as well. And guess what? If you took these circumstances and did waterfall, you'd be just fine. What agile is great at is failing fast and eliminating waste. The feedback cycles are there to detect and correct the course of the project almost as soon as it starts. I'm not sure how common this expression is, but when someone asks you a question to which the true answer will be very upsetting for them, the initial response is often, "Do you prefer the truth or a comforting lie?" That successful project had been estimated prior to my team's arrival. When we took a look, our estimate was 3.5 times higher and the duration was twice as long. We had extreme pressure to just hide our estimate and do the typical budget blowout thing later on. Not doing so and being honest nearly killed the project before it started. Agile is for those who can handle the truth; waterfall is for those who prefer the comforting lie. Either one may be right for you and your organisation.