Dale Emery influenced this blog entry with his paper on Writing maintainable automated tests. At the essence I’m going to compare several test approaches: Unit Testing (i.e. Test-Driven Development), Functional or Acceptance Testing and Exploratory Testing. I’ll look at means necessary for regression capability (checking), costs for testing and costs for adaptation to changing requirements.

Unit Testing

When you’re writing your code test-first with unit tests in place before you develop your code around them, you have near to perfect regression capability. You need to keep these tests up-to-date though. The Continuous Integration practice from Extreme Programming teaches us to automate these unit tests as much as possible. This results in near to zero tests for having these checks run over and over again. If you immediately stop the line when the tests start to fail and fix the problem behind it – i.e. a not adapted test data or a software bug – then you have a good balance in your regression ability and your tests.

There might be a problem, when you run into the situation that your customer adapts to changed business conditions. Facing the unavoidable requirements change, your code and your unit tests need to be adapted to the changes. Since there is no free lunch, you have to take the bitter pill and adapt your tests, but you still know in the end whether or not you broke any unrelated functionality in other modules with your changes by doing so.

Functional or Acceptance Testing

In functional or acceptance testing (I confuse the two, since we mostly use both at work in a similar manner, forgive me Michael), when using frameworks like FitNesse, Robot Framework or Concordion (to name just a few) we have some test data in some textual form together with some glue code that makes the transition between some keywords in the textual test descriptions and the system under test. There are two dimensions to consider here.

First the textual description of your tests should be clear. Dale also suggests to clean them up to not repeat themselves. In fact your tests should focus on the business requirements at hand. These frameworks suggest to focus on a single business rule per test case. This might seem contraditing, especially when transitioning from a legacy test base or code base, but it is essential for managing the tests.

Second, the glue code needs to be maintained just like any other source code on your project. It’s as critical if Technical Debt lurks in your fixture code base as if it lurks into your production code base. This does not necessarily mean that just developers should write the code for your fixtures.

So, getting back on costs, you have the costs of writing down the business rules in clear language, maybe using the constructs available in the framework at hand. Then there are the costs of writing the glue code. Running the tests is usually also automatable to the degree, that you can run them in your Continuous Integration server. Comparing this to the costs of unit tests, then Functional or Acceptance tests are more expensive to develop. Running them over and over is a bit more expensive, since they are likely to take a longer time to run. When requirement changes get incorporated and you need to adapt your tests to it, then you’re also in the unlucky position, that some of your tests might change. This might result in changes in the data only, or changes in the glue code only, or both. Thus, overall these kind of tests are more expensive than unit tests, while providing a more composite view on the product at hand than unit tests.

Exploratory Testing

Exploratory Testing is a human approach. The human sets up her own charter for the test session at hand. Exploratory Testing is an approach, which uses simultanous test design, test execution and feedback. The costs boil down to the stuff time you set up for Exploratory Testing – and this shouldn’t be too few in most contexts.

Running the tests is therefore as costly as paying your tester per hour. Running the same test again is a way to demotivate your testers on the other hand. So if you plan for regression ability, you should better set up with a combination to some of the automated approaches mentioned early. The costs for humans repeating the same tests over and over again are non-linearily higher due to the negative impact on motivation. On the other hand a human tester can adapt to changed requirements for example in the user interface easily. The costs for changing requirements are near to zero. You simply plan a charter for Exploratory Testing of the changed area of the product.

You will find a more thorough elaboration of testing costs in a series from Michael Bolton: Why is testing taking so long?.

Conclusion

Like any three-course meal you need to arrange your approaches to software testing well. Balancing one over another can lead to higher maintenance costs. On one hand you don’t want to spent too much time adapting your test code to new requirements, on the other hand you need to have a safety-net of regression tests to make sure your latest code change did not break any critical area of the product. Overall I prefer to plan time for all three activities if I can. But make sure to balance them. You wouldn’t start with a big dessert up-front in your three-course meal. This wouldn’t be healthy over time. Instead having some slices of tests may lead in the end to a thoroughly tested product. Reflect for bugs creeping up where these are best handled and take care of them accordingly.

Unfortunately most test managers can’t decide on the degree of unit testing in their company. If you’re in a lucky position, you can help your company and your product by balancing the meal. If you’re in the unlucky position, you need to collaborate with your developers more on this. Overall, test strategy decisions are best made with the whole team buy-in. So, make sure to involve anyone in setting up your individual meal.