Don't delete unit tests even if you have acceptance test coverage
I think it is well understood that the sooner you find a defect the less it will cost. A defect costs the most once into production whether that be a car that needs to be recalled or a software bug that stops your companies orders being processed.
If you use TDD and CI then you get to find some defects at the development stage where it is clearly a lot cheaper to fix them than finding those same bugs in QA, UAT or worse still in production.
What is really odd then is that people delete unit tests and while there can be legitimate reasons such as code no longer being used this is not why most tests get deleted. Some bogus reasons for deleting tests are
a) "They keep failing when I change things" - Well that's because you just broke some existing functionality! And guess what: it's a lot cheaper to fix that defect now at development time than once it gets into production. Of course maybe someone else has to deal with production bugs so you could just be selfish and ignore the cost to them and your company and go ahead and delete that failing test.
b) "They are very hard to maintain" - This is more insidious and is a symptom of something that actually happens quite a lot: developers forget that tests are code too. So for some reason once developers are working on unit testing all those things we know are bad practice are allowed. Unsurprisingly the end result is poorly factored code that is hard to maintain. So should you delete those tests? No! You should do the same as you would when you find hard to maintain code anywhere else in the codebase (that is slowing you down), after all test code is still just code.
c) "I have coverage of that code from acceptance tests" - So what if you have coverage of that code from a higher level acceptance test? Is it ok to delete the unit test then? I think the answer must still be no and the reason is to do with the cost of finding and fixing a defect. A while ago I was working on a big complex distributed application with great acceptance level tests but relatively poor coverage around the messaging layers. These acceptance tests failed frequently and developers would spend a lot of time trying to track down the problem. We added units tests to flag those issues much earlier and in a far clearer way. So while we were adding test coverage to code that was already tested we did so to lower the cost of finding and fixing defects.
Deleting test code should only be done if it will not increase the cost of finding a defect.
If you use TDD and CI then you get to find some defects at the development stage where it is clearly a lot cheaper to fix them than finding those same bugs in QA, UAT or worse still in production.
What is really odd then is that people delete unit tests and while there can be legitimate reasons such as code no longer being used this is not why most tests get deleted. Some bogus reasons for deleting tests are
a) "They keep failing when I change things" - Well that's because you just broke some existing functionality! And guess what: it's a lot cheaper to fix that defect now at development time than once it gets into production. Of course maybe someone else has to deal with production bugs so you could just be selfish and ignore the cost to them and your company and go ahead and delete that failing test.
b) "They are very hard to maintain" - This is more insidious and is a symptom of something that actually happens quite a lot: developers forget that tests are code too. So for some reason once developers are working on unit testing all those things we know are bad practice are allowed. Unsurprisingly the end result is poorly factored code that is hard to maintain. So should you delete those tests? No! You should do the same as you would when you find hard to maintain code anywhere else in the codebase (that is slowing you down), after all test code is still just code.
c) "I have coverage of that code from acceptance tests" - So what if you have coverage of that code from a higher level acceptance test? Is it ok to delete the unit test then? I think the answer must still be no and the reason is to do with the cost of finding and fixing a defect. A while ago I was working on a big complex distributed application with great acceptance level tests but relatively poor coverage around the messaging layers. These acceptance tests failed frequently and developers would spend a lot of time trying to track down the problem. We added units tests to flag those issues much earlier and in a far clearer way. So while we were adding test coverage to code that was already tested we did so to lower the cost of finding and fixing defects.
Deleting test code should only be done if it will not increase the cost of finding a defect.
posted by Ian Cartwright at 10:56 AM
2 comments
