In the past week Ulrich Freyer-Hirtz on the Agile Testing mailing list asked about different testing levels in an Agile project. He found the definition of unit, component, and acceptance tests appropriate based on several books. This discussion reminded me about the ambiguity that we face today in terms of testing. There have been several renaming attempts in the past few years. For example Dale Emery refers to a blog entry from 2004. Gojko Adzic made a similar renaming attempt. Let’s take a step back, and see what all those names really want to tell us. Afterwards we will be able to make a more informed decision about names.
Test Levels
Why do we need to think in different test levels? I think we don’t need to do that anymore. Back in the 1970’s when the distinction between programmers and testers had to do with different responsibilities most of the terms floating around today have been coined. Back then, it mattered whether you< were testing on the unit level, on the component level, or on the system level. You had different missions for either of these levels. In 2010 Mike Hill commented on my chapter in How to reduce the cost of software testing that he prefers the term microtest for unittests. The term unittest it seems has different meanings for different developers. Microtests make it transparent that we try to tests a little code as possible.
I like this term. It makes transparent that we aim to test as little as possible with these unittests. When I work with programmers, I often find the practice of unittesting producing tests focused on the functional requirements. The application with its database, mostly enterprise application servers, and any additional component is utilized. While this sort of tests is necessary as well, these are certainly no microtests. Microtests focus on one class of the production code – if possible. But what about that area of functional tests? And what about the remaining levels of testing, i.e. component, integration, system, sub-system, etc.? Let’s take a look on other heuristics that might reveal insights about those. Enter the testing quadrants.
The Testing Quadrants
The testing quadrants as written down by Brian Marick provide a classification scheme for different tests as well as for the needs you might face. Rumor has it that most of the thoughts came actually from Cem Kaner, but Brian was the first to write them down, followed by Lisa Crispin and Janet Gregory in their Agile Testing book.
The testing quadrants describe a consultant’s universal tool: four quadrants in a rectangular shape. The dimensions of the rectangle are technology-facing vs. business-facing on the x-axis, and team-support vs. product-critique on the y-axis. I think the term team-support does not ring immediately for certification-harmed testers. Yet, the dimensions describe quite well what helps a team, and makes a profound testing approach.
Microtests are found in the quadrant describing technology-facing tests that support the team. These clearly help drive forward the development. Over time they become change detectors, or checks as Michael Bolton would put it.
Functional tests usually lie in the quadrant with business-facing tests that support the team. These tests help the team, too. But they are written on a different level for a different purpose. They also help drive forward development, but are focused on a different level. Rather than making sure that you can exercise a network connection or connect to a database, these tests express business expectation and requirements. Some teams call them integration testing, acceptance tests, or system tests.
Whatever you call them, they address the issue that you don’t necessarily know that you fulfilled the requirements for your customer. Acceptance tests may not reveal that information as well, but it informs you definitely that you fail to fulfill them in the future if one of them fails.
On the product critique side of the quadrants, performance, load, and the *ilities are found in the quadrant with technology-facing tests, while User Acceptance tests, usability testing, and the like are found on the business-facing quadrant.
With all these tests at my finger tips, what should I do in an Agile project? Enter risk analysis combined with the Testing Quadrants.
Risk analysis
In an Agile project, you face two constraints. On one hand, you want to serve the project as a tester. You want to provide feedback on time, to meet the two week iteration length, or whatever your time constraint there is. On the other hand you want to test the software thoroughly so that end users will not be harmed or confronted to a bug. So, you face time vs. test coverage – like in most traditional projects as well, but with a shorter timeframe.
So, you have to pick tests that you can fulfill now within the given timeframe. You want to make that as transparent as possible. So, depending on where your team currently is, you might be able to run tests for every quadrant, or you might be able to cover just the microtests and the functional tests.
But you have to make sure to include compensation for the tests that you currently cannot run in one iteration. This will take down your velocity, or you will have to plan from time to time for performance, load, or usability tests. However, over time you should be able to expand the tests that you can perform in one iteration, and fill these gaps. Over time you will gain a more balanced test set within your iteration.
Test Levels are an anti-pattern
Test levels are an anti-pattern. They are remains from a time long gone. On an Agile team everybody has the same responsibility to make the team successful. If you think about test levels, you have a more profound problem in your team. How come your team is not thinking as a team about issue? Why don’t all team members contribute to the testing necessary? I claim thinking about test levels on an Agile project as an anti-pattern. When transitioning to Agile, you often start with explanations that testers are used to. So, that’s where we start to classify different testing practices on the quadrants. But over time the team should make a mind-shift towards a Whole Team Approach to testing. If it’s not, you should fix that problem, not the problem about levels or terms.
HI Marcus
The quadrant is an interesting view of testing. The main point that I get from your post is the important attitude change that testers need in order to work well in an agile team. I agree! However I think this has more to to with team cooperation, responsibility and flexibility than the concept of testing levels.
Just to play devils advocate and test your theory. When you describe the content of each quadrant, to me it sounds very much just like.. test levels. Acceptance tests in top right quadrant (test level AT), microtests lower right (unit/dev test) etc
Oops train arriving in Stockholm, got to go.
Hi Tobbe,
the examples I gave are useful to make a connection between the usually known test levels in pre-Agile companies and a more useful scheme for Agile testing.
It’s true that your team continues to think in test levels form quite some time. And as the responsibility for testing gets more and more shared among team members, you start talking less and less in levels, and rather in different approaches.
Hi Markus,
I agree that we’re still struggling with the remainders of the times past, especially by holding to the importance of these concepts we inherited.
While I think it’s still worth while using, say, the concept of distinguishable test levels, in my opinion they’re just not quite as important and central to testing anymore.
The same holds true for the very detailed test plans … and long lists of test cases.
The levels of testing as originally proposed by Boris Beizer have nothing at all to do with distribution or delegation of work, but are abstract concepts that can help focus the purpose and/or design of a test. For example, unit tests are generally very focused tests on individual methods or functions. System level tests tend to be very broad in scope, include greater variation (hardware, data, environmental, etc), and often from the customer perspective (finished product in Marick’s lingo).
I also think the quadrant schema is just as valuable in helping people think about test designs and or the purpose of their tests.
I teach both, but personally like the layered model because it works well when used in conjuntion with typical layered architectures used in todays OOP software.
Neither model speaks to the delegation or responsibility of work; however both can be used to help people think of how to better design tests based on the purpose or intent of the test.
What I object in my blog entry, is not how the models were originally intended to work, but how they are in widespread use. Changing that needs to think differently.
Another objection I have is the “term” Integration Test. I think it should be banned from our vocabulary completely as it means so many things to so many people. It just confuses me each time I hear it to the degree, that I ask people what they mean by this confusing term.
Hi Markus,
The theory here is really interesting. On a practical level the boundaries can be fuzzy, of course.
We are approaching the end of a major rearchitecting project. The lead developer and myself both constructed test suites very early on and developed them in parallel. They’re handled in an external test harness that drives the product and in JUnit and they probably correspond to system-level and unit-level tests (i.e. not micro-level in your terminology).
The relevant part here is that we made a decision to share test data whenever we could and so we agreed a data format and implemented readers that could drive our suites with the other’s data. This has worked really well for us despite crossing over the traditional levels.
Hi Markus,
Nice post about the discussion around test-levels!
Personally, I am in favor of using the agile testing quadrants to see whether user stories are covered by testing as well. However, besides focusing on the risks that you’ve found during risk analysis, I think that testing should also be focused on value. Since teams practicing agile aim to deliver business value as soon as possible, it should also be ensured that the business can actually generate this value when a potentially shippable products goes into production.
As an example: there may be user stories that entail little risk but can deliver high value for the business. Once time pressures get to the team, and this user story is not included (or not enough) in testing efforts, the business will not be able to immediately benefit from the release.
The combination of risks and value should, in my opinion, be the input for an iteration’s test plan and the test effort should be focused on the high risk – high value combination (I like the visualization of Gojko Adzic: http://gojko.net/2011/11/28/5-key-challenges-for-agile-testers-tomorrow-slides/). Knowing the test priorities, we can use the agile testing quadrants to map test types in order to identify the effort required for each story.