At the Melbourne eXtreme Programming Enthusiasts Group the other night we had a big rantorama about TDD. One of the points in my presentation was that Kent had said that unit tests had to be fast, and he cited one project in which there were 4,000 tests that ran in about 20 minutes. A few guys from my team were there and guffawed out loud. Our current project (4 months old) has 1,300 unit tests that run in about 10 seconds, so they weren't impressed with Kent's numbers at all. Even allowing for Moore's law over a few years, we'd run about 4,000 tests in 30 seconds, which two speed-doublings ago would take 2 minutes. That's still an order of magnitude faster...so what's the deal?
This led to lots of enthusiastic debate about what the hell a unit test actually is. For what it's worth, here's my simplistic test categorization model:
1. A unit test is a test that requires only the class under test and its immediate dependencies (or mocks thereof). This means no deployment, no packaging, nothing. That's why they're fast.
2. A functional test is a test you can't run until the app is packaged and deployed in some way. So you "in-container unit test" fans aren't doing unit tests by this model. I'm hard on this stuff because anything requiring deployment puts you in a different order of magnitude speed wise; this doesn't mean these tests aren't valuable, just that I don't count them as _unit_ tests, even if they only test a single unit of functionality.
3. An integration test is a test of your external dependencies, not necessarily requiring your app to be deployed at all, but requiring your external dependencies to be available. This is where databases come in, or other services provided by EJB's or anything else. These tests are also usually way too slow for me to want too many of them, and I want to isolate them from my app as much as possible. This means a separate test harness for the external stuff and leave my app just unit testing against the assumptions proven by the integration test suite.
The point is that I want to push unit tests as far as I can because they just provide so much more performance than the other types of tests. That's the only reason. I want as few of the others as possible because I spend enough of my life waiting for computers to get done with stuff as it is. Yes, you _can_ unit test custom tags to within an inch of 100% coverage!
So what's everyone else getting speed-wise???
Is speed really the only reason? It's one hell of a big reason but what about reducing the effort to setup infrastructure and just overall good design and de-coupling, etc. that comes from "properly" unit testing?
ReplyDeleteJames,
ReplyDeleteon my machine the unit tests (now up to 1390 of them) now run in 4.5 seconds. You should upgrade your machine if you want avoid that extra 5 second wait :p
I have to admit that the unit test suite for this project runs the quickest (by huge amounts) of any project I've ever worked on. Well done to those that set the foundations.
Hi James,
ReplyDeleteWe have 445 test cases at the moment and they take 10 seconds.
The only reason they are that slow is because someone accidentally put calls to the ASP.Net membership API inside some of the tests. We are now in the process of mocking this out and are seeing a .1 second drops in running time.
Some have called me crazy for caring about the difference between 5 seconds and 10 seconds. I simply tell them that we need to set a good example for others and that if we take the easy road now, we'll just have to pay for it later when we have 10-20 times as many test cases.
- Chris