Long time no blog. I've been distracted in recent months learning about non-geek stuff like investing in the stock market, which I may blog about at some point. But what has inspired me to pick up my virtual pen again is the apparent mainstream rise in popularity of agile development. Bear in mind that I'm in Australia so your time lines may vary, but these days there are very large organisations stating agile development as a key aspiration and sending head hunters around about scrounging for anyone who can spell it correctly to lead them to the promised land. Five years ago we relatively early adopters of eXtreme Programming would band together and tell each other stories of what a wonderful world it would be when major banks and telcos saw the light and changed their evil waterfall ways. Now it seems we may have gotten what we wished for and perhaps we should have been more careful.
Refer to Martin Fowler's recent post on how top-down imposition of agile is pretty un-agile by definition; he's far more calm and articulate then I am. But as someone who has seen agile succeed and fail, I thought I'd throw my opinion into the ring for a bit of perspective for the recently converted. The greatest success with which I have been involved personally was simply awesome. A true agile poster child project: the brochure for it would tell you that we estimated it almost perfectly, delivered it on time and on budget, have had so few bugs in two years of production use that you'd have several fingers left counting them digitally, and the system transformed the business in a truly fundamental way. And we're not talking about a small project here: 20 people all up for 18 months and several million dollars. This is all absolutely true. What the brochure would fail to mention most likely is that we clashed with several internal stakeholders to the point where I was excluded from the building for a week in the middle of the project, that we needed the best team of developers I've ever seen to pull it off, the best project manager I've ever seen, the best iteration manager, customer representative and analysts and we had to fight off any number of die-hard process nazis who did not share our value system of actually delivering something of value (think Vogons). The brochure would also fail to mention that the organisation that witnessed this startling occurrence was quite committed to hiding this anomalous project like an embarrassing relative; you see, we outsiders had been brought in as the third attempt at this project in three years, and were set up to fail. The brochure probably would not mention that either.
Where I've seen agile fail is quite simply when you don't have all of these things working for you. Like having a distracted customer who can't spend the time with the team. Like having developers who really don't care about the craftsmanship of their work. Like having analysts who just type requirements into documents without any actual analysis. Like testers who just do the happy scenarios because the other ones are too hard. Like having a project manager who focuses on avoiding failure rather than achieving success. The problem with agile is that any one of these can severely damage or even kill your project despite the health of the others.
So the good news is that agile can and does work. The bad news is that agile can and does fail. So what are you to do? Looking back on that successful project, what made it work was, unfortunately, probably the same old things that have made any project work in any human endeavour since human endeavours began; highly skilled and committed people who bust a gut, put their necks on the line and get really lucky as well. And guess what? If you took these circumstances and did waterfall, you'd be just fine. What agile is great at is failing fast and eliminating waste. The feedback cycles are there to detect and correct the course of the project almost as soon as it starts. I'm not sure how common this expression is, but when someone asks you a question to which the true answer will be very upsetting for them, the initial response is often, "Do you prefer the truth or a comforting lie?" That successful project had been estimated prior to my team's arrival. When we took a look, our estimate was 3.5 times higher and the duration was twice as long. We had extreme pressure to just hide our estimate and do the typical budget blowout thing later on. Not doing so and being honest nearly killed the project before it started. Agile is for those who can handle the truth; waterfall is for those who prefer the comforting lie. Either one may be right for you and your organisation.
Subscribe to:
Post Comments (Atom)
I'd put this in a different way.
ReplyDeleteAgile is the recognition that if you create certain properties in your project (for example, http://alistair.cockburn.us/index.php/Crystal_Clear_distilled), the likelihood of success is significantly greater. If you fail to create those properties, your likelihood of success is significantly less.
If you failed to create the properties, does that mean that Agile failed? No, it means that you failed to introduce Agile. Does that mean that having Agile properties guarantees success? No, other factors may come into play. Similarly, not having Agile properties does not guarantee failure.
As for starting with good circumstances, I would say that in that case, a waterfall process will more likely create a distracted customer, developers who don't care, etc. and therefore still be more likely to fail.
What I'm saying is that given *any* problem, I will bet on bringing people together, empirical feedback, and adaption OVER formality, contracts, and sticking to the plan.
Jason,
ReplyDeleteNow, keep in mind that I'm a self-proclaimed agile fan and practitioner, but that sounds a bit like defending communism: Did communism fail? No, it just wasn't implemented properly. You can pretty much defend _anything_ with that argument--beta-max; IBM's micro-channel architecture; OS/2; the list goes on.
Hi James,
ReplyDeleteI'd like to examine the idea of the same team doing the same project with waterfall being as successful. Let's take this to the extreme by putting the following 'waterfall' constraints on your project and see what you think:
- Analysts are the only people allowed to talk to the business - no development team allowed.
- Analysts produce use cases in the requirements phase, then move off the project, so cannot be contacted any more - the development team just have the use-cases.
- No testing is allowed until all development is complete, and it is under no circumstances to be performed by the development team. That's the QA team's job. And they can't be involved until a month before the end of the project, as they are currently allocated to another project.
- There are office rules – there are desks allocated for your development team, These are not close together, are on a different floor to the PM, the testers and the business, and can only fit one person in front of the computer. The office has a 'clean wall' policy. Anything stuck on the wall will be cleared by the cleaners at lunchtime and at night. All whiteboards must be cleaned every day.
- Only the team-lead and architects can attend meetings - they can communicate to the development team. The developers will be too busy writing code.
- Development practices : No TDD / automated testing allowed. Testing is for the testers! No CI - we can't buy you a server, and you are not allowed to run a server on our internal network. Refactoring has been banned by the "Architect", as it involves too much re-work.
(I’m sure I could go on…, but you get the idea)
What are your chances now of delivering a successful project with the same team? Would it be on time, on budget, with a handfull of bugs?
This is, of course, rather silly, and I'm sure you could argue that lots of the above statements are just bad development practices and the solution is not specifically agile, but as you know these kind of situations that have to be overcome everyday (maybe not all on one project!). To me, agile development is a collection of the successful project and development values and practices that are considered, through experience, to produce successful projects.
However, I'd be interested what you consider "if you took these circumstances and did waterfall" actually means? Is it any of the above? Or is it really having great people that force through agile practices and values without calling it “Agile”?
Simon,
ReplyDeleteCommunism did fail because it wasn't implemented properly... :) Of course, I'd also argue that even implemented properly it would have failed and the process of implementation that communists will likely take is doomed to failure.
The point is that a collection of practices is not Agile (nor any other thing you want to introduce). I can teach you how to punch and kick, but that doesn't make you a martial artist.
Darren,
ReplyDeleteExcellent comment. I have been lucky never to have had it so bad as what you describe, though all of those kind of things have happened on different projects. You've made me think a bit more about what I think, and what I think I think is that while I remain a hard core agile fan, I think I'd rather be on a waterfall project with awesome people than a top-down imposed supposedly agile project where nobody on the project knew or cared what agile was and don't play their part to a highstandard. This is the dangerous thing from my perspective; that the years of good people doing good agile projects in small numbers will be numerically insignificant compared to the mass of people who have done crappy projects for years only to try agile in desperation, fail, and blame agile. I think you hit the nail on the head in your last sentence; I think good people on a waterfall project would do agile on the sly to get the result because they care about the result.
I think Kent Beck said something like "XP is what good people do to make their project successful", the point being that he was documenting stuff that good people do. My corollary would be that if you don't have good people, knowing what good people would do may or may not be useful to you.
ReplyDeleteG'day James!
ReplyDeleteFrom my limited experience in software development I can't say I'm much of a fan of the waterfall approach.
Throughout Software Design and Dev class in high school and even at university now the teachers and textbooks drum into our heads that waterfall and structured development is the only way to develop software appropriately and that other methodologies such as my beloved RAD or agile time-release development are simply too unreliable for anything other than small projects. They claim less structured approaches are child's play and that nobody in the real world with a decently sized cranium would consider using them.
The problem I see with the waterfall approach is that there's simply too much overhead and fluff which gives the illusion that work is being done despite the fact nothing constructive has been achieved. While I agree that going headfirst into a project is foolhardy, I don't think sitting in a boardroom in suits presenting pretty graphs in Keynote and PowerPoint for months really actually serves a practical purpose other than to make the boss happy; which may or may not be fulfilling your objective! A feasibility study should be produced, a rough schedule hammered out and work should start. Doing what needs to be done versus doing what you want to get done I think can be two very different things.
I'd consider using the Sashimi Waterfall approach I learned about last month, but only because I think its more flexible and sounds like Japanese food of which I am quite partial to.
Microsoft supposedly uses the strictest form of the waterfall approach, and look at Windows Vista! Perhaps when I join the workforce and the weight of the world sets in I'll change, but for now I'll just bask in my inexperience and continue to look critically at rigid, structured software development.
Love reading the blog!
Cheers,
Ruben