Smell the Tests!

When you are faced with a team that's new to agile development, it's common to spend a lot of time driving home the need to test everything. Along with refactoring, test-driven (or at least test-conscious) development is the most basic skill the aspiring extreme programmer must acquire. We strive to testability in design and the cheapest, most effective quality we can get. So is there such a thing as bad testing? Can tests hinder more than help? Can tests just be a downright pain in the rear?

The answer is a resounding YES! YES! YES!

Just because you code in an object-oriented language does not guarantee that what you write will contain any object-oriented characteristics whatsoever. Similarly, just because you write JUnit test cases does not guarantee that what you produce will display any of the qualities of an effective and adaptable test suite. Naive application of testing frameworks has the same potential for developers to exercise their bad judgement and bad taste as any other technology. Now while I'd certainly rather have a project with lots of less than ideal tests than no tests at all, we need to learn to 'smell the tests' to really get the most out of our (and our customers') investment in testing.

So here are some of the smells that tests can emit. I'm sure you can think of more, so please drop me a line so I can share your pain...:P

• No statement of intent. If I have a Widget with a getPrice() method, I often see a testGetPrice() test method, and my first thought is "test getPrice() does what?". Prefer names for tests that express the intention of the test, such as testGetPriceReturnsZeroWhenNotInitialisedExplicitly(). There are tools that can turn your test names into human readable documentation, so don't be afraid of full-length sentences as test method names.
• Irrelevant assertions. It is common (particularly in functional tests) for a test that is concerned with proving a small incremental behaviour to make irrelevant assertions about all sorts of other fields, even on other screens. Prefer semantic generalisations over value-specific assertions. For example, if your test is not concerned with the actual price algorithm, don't assertEquals("46.12", widget.getPrice()) when you can say assertNonZeroMonetaryValue(widget.getPrice()) - even though it's a bit more work at first.
• Setup leakage. It is extremely common for developers new to testing to not easily be able to distinguish the concerns of their test from the setup steps required to get them to the point where their test can begin. The newly test-infected developer can thus create maintenance burdens and simply waste time reinventing the testing wheel with every test case. Strive for a clear separation between setup and test, and look for reuse in the setup code via fixtures and scenarios.
• Overtesting. Many developers find it difficult to focus only on what is relevant to the test at hand. It is not only more efficient to assume the rest of the system works when writing a new test, but it also improves maintainability as each test is more isolated and targeted. It is common for developers to turn themselves into knots to get controlled input and output moving between different components when a mock would be faster and more effective. Leave it to the other tests to do their job and assume away responsibilities that are not yours right now for the code under test.
• Multi-layer tests. Unit tests should rearely (if ever) traverse multiple layers in your design. A sure sign that they do is when they run slowly. Unit tests should run in the thousands per minute kind of rate, while functional tests tend to be in the 30-50 per minute range, so prefer unit tests to achieve coverage and functional tests to prove wiring of configured components.
• In-container unit tests. T'ain't no such thang. Don't make sense. If you think you have some, that's a smell. :)
• Fractured functional tests. Adding functionality a small bit at a time leads to functional tests with a lot of overlap and there is no force to encourage the consolidation of these tests other than slow build speed. I'm not sure how to fix this one other than slow slog refactoring. We have a lot of this joy ahead on our current project!

None of this alters my belief that test-driven development is the best method I know for building quality software. But it does mean that (once again) we need to be careful about silver bullet syndrome when it comes to testing.

 

Have Hair Dryer, Will Boot

Having moved house recently, I was dismayed when one of my PC's (the one that's built for music and video editing) refused to boot. The primary hard disk was not being recognised by the BIOS. Dammit. Now yes, I have all the important data backed up, but it's pretty tough to backup full quality video footage at the best of times, and this box has had years of tweaking put into it, so rebuilding it from scratch was a rather sphincter-puckering concept. Having a couple of weeks paternity leave to sort it out, I started ringing around the pro data rescue market, and the prices were all in the $2,000 range. Hell, I could just buy a whole new box for that.

So being a software guy, I rang my buddy the hardware guy and told him my story, of how the box had been running happily for 18 months without being turned off, then it was shut down for a couple of weeks during the house move/new baby transition. "Ah-hah", says he, "just get out the hairdryer and heat that drive up for 20 minutes; it'll boot first go". And bugger me if he wasn't right on the money! A quick plug in of a new drive onto which I ghost-ed the dying one, switcheroo and off we go.

Phew!

Who gives a toss about software anyway?

I mean, really, I'm as guilty as the next geekblogger of making software and its myriad cultural spawn the centre of the universe, but when you hold your new baby girl in your arms it really does seem all a little silly. So here's to the arrival of Ella Rose....and yes, I did buy her a domain name before she was born!

IntelliJ as.....Writing Tool!

Trying to collaborate on some technical writing with Simon Harris led us to figure out a way to get the type of collaboration we are used to while coding when we're writing. this means having the content in a textual format so it can be merged, ruling out MS Word and other binary formats. We've been using a wiki a lot for this recently, but reading Martin Fowler's post about writing using XML made us want to give it a try. Now before you say DocBook, let me just say that that's just a little tooooo heavyweight for our needs. But the idea is right. So we knocked up a little DTD that incorporates a hierarchy of sections, with figures, code listing, cross references, external references, etc, and then used IntelliJ IDEA's DTD-driven XML editing to start writing the content. With the code folding and a few live templates, we pretty quickly had a nice intuitive editing experience that allows the separation of the creation of the content from the marking up thereof. With the addition of a little XSLT that automatically numbers the nested sections, voila! IntelliJ as author's workbench! Cool!

When Prevention's Not Better Than Cure

When doing agile software development, our main challenge is having little enough process to go fast, but enough process to avoid crashing. The desire to go fast is all well and good, but we don't want to have our only feedback mechanism on whether we're going *too* fast to be blunt force trauma.

I have no doubt that application of agile methods such as XP without equal application of both the freedoms and the responsibilities they afford will end in tears. This is often the biggest challenge with developers new to XP. They love the freedom to refactor and design on the fly, but their enthusiasm sometimes wanes when confronted with the discipline of maintaining test coverage under schedule pressure.

So how do we know when we're going too fast? One problem with perceiving the team's speed (as opposed to velocity :P) is that different members of the team will have different comfort levels with the shared rate of progress. Several times recently I've found myself responding to feedback from team members that would like a little more process to avoid some issue or another, be it screen rework or the presence of defects during QA. I try to be careful to get enough feedback to find out if what we have is just turbulence or the swelling sound of a wing shearing off the fuselage. Most times it's the former, and in these cases I find there's a recurring theme to my response: prevention isn't always better than cure.

For example, in our current system we have 500+ story cards. If about 10% of these result in a defect (which looks about right at the moment) and each of these cost a day for a pair to fix (no metrics but I think that's more than the reality), that's 50x2 days of rework. If we were to put in place a process to prevent all these defects, it would probably cost the team at least two hours per story (that's only two people talking for an hour) to do the extra analysis and testing for EVERY story card. That's 1,000 hours extra work, or comfortably more than the 100 days of rework we have to do to cover off the defects. That's also on the unrealistic assumption that you do actually prevent all the defects! It's just a numbers game in which you win if the quality is sufficient that the amount of rework on the defects is less than the amount of effort to prevent all those defects plus the extra effort you put in to the majority of cards that would never have had defects in the first place.

So I find I'm spending a bit of time reassuring the passengers that a little turbulence is normal, and preventing it is not only unnecessary, it's counter-productive. At the same time I try to maintain a healthy paranoia about the process. Of course, the danger lies in not being able to tell turbulence from a tailspin, so if anyone has ideas on how to do this, bring it on!

 

Conversation Is Not Understanding Either!

I have been as guilty as the next geek of rationalising my distaste at the tedium of documentation. Many of my agile friends and colleagues are fond of saying that 'Documentation is Not Understanding' when promoting the 'people over process' bit of the Agile Manifesto. Now while I do indeed hold this truth to be self-evident, a few recent experiences find me not so dismissive of the value of documentation as once I was.

Circumstances led me to leave my role in a nice fun agile project for a few months. When I returned, I struggled to get back up to speed for a couple of weeks - these agile folks will just go and change things while you're not looking :) When trying to reconstruct my mental model of the project, I found the pieces I had to put together out of conversations with analysts, customers and developers didn't always fit together snugly, and I found myself needing some sort of point of reference that was indeed shared understanding to work forward from. Anyway, I haven't really worked out what this means, but I have to conclude that while documentation is not understanding, neither is conversation. Another assumption bites the dust!

The Need for Speed

At the Melbourne eXtreme Programming Enthusiasts Group the other night we had a big rantorama about TDD. One of the points in my presentation was that Kent had said that unit tests had to be fast, and he cited one project in which there were 4,000 tests that ran in about 20 minutes. A few guys from my team were there and guffawed out loud. Our current project (4 months old) has 1,300 unit tests that run in about 10 seconds, so they weren't impressed with Kent's numbers at all. Even allowing for Moore's law over a few years, we'd run about 4,000 tests in 30 seconds, which two speed-doublings ago would take 2 minutes. That's still an order of magnitude faster...so what's the deal?

This led to lots of enthusiastic debate about what the hell a unit test actually is. For what it's worth, here's my simplistic test categorization model:

1. A unit test is a test that requires only the class under test and its immediate dependencies (or mocks thereof). This means no deployment, no packaging, nothing. That's why they're fast.

2. A functional test is a test you can't run until the app is packaged and deployed in some way. So you "in-container unit test" fans aren't doing unit tests by this model. I'm hard on this stuff because anything requiring deployment puts you in a different order of magnitude speed wise; this doesn't mean these tests aren't valuable, just that I don't count them as _unit_ tests, even if they only test a single unit of functionality.

3. An integration test is a test of your external dependencies, not necessarily requiring your app to be deployed at all, but requiring your external dependencies to be available. This is where databases come in, or other services provided by EJB's or anything else. These tests are also usually way too slow for me to want too many of them, and I want to isolate them from my app as much as possible. This means a separate test harness for the external stuff and leave my app just unit testing against the assumptions proven by the integration test suite.

The point is that I want to push unit tests as far as I can because they just provide so much more performance than the other types of tests. That's the only reason. I want as few of the others as possible because I spend enough of my life waiting for computers to get done with stuff as it is. Yes, you _can_ unit test custom tags to within an inch of 100% coverage!

So what's everyone else getting speed-wise???