Lisa Crispin gave the first keynote in the morning of the Agile Testing Days in Berlin. She talked about an approach to defect management and tracking.
Crispin stated that in order to deal with defects, one had to lower the bar for defects. She used a limbo analogy for this, where the bar gets lowered and lowered. She presented form her team which had strived for zero bugs for several years, and when and how to log defects and track them.
On the traditional view on defect tracking, these systems are used for metrics and to help the team to know what to fix. Customers also may want to get an insight into the defect tracking system in order to understand which errors they might have to deal with when buying the software.
In the Agile view defect tracking is a waste in the Lean sense. Striving for zero defects is the goal to try to achieve there. Instead of filing bug in the tracking system, Agile teams work on creating an automated test first, which reproduces the problem in the system, and then to have any regressions being dealt with by the continuous integration framework.
On the advantages of using a tool to log bugs Crispin mentioned that the tool might provide the metrics necessary on more traditional teams. Also, the customer can get a hindsight of the quality of the system, and managers may use it to control the project.
On the downsides of a defect tracking system, it takes usually more time to file a bug than to write an automated test which exposes the problem. On an Agile team you can use color-coded story cards to track defects instead of having a second backlog from which work is taken by the team. Color-coded bug cards reveal problems in the process as well.
Crispin presented the approach her team takes in order to deal with defects. For unit level bugs they most of the time write an automated unit test instead of filing a bug, and fixing the problem immediately. The same holds for functional regressions, which then yields in a functional tests on the story-level. Some bugs they file for later, putting them effectively into their product backlog as a new story, which gets estimated and prioritized together with the rest. In addition Crispin mentioned that her team tries to avoid to put anything into the defect tracking system which they know they are not going to fix any way.
Finally, Crispin mentioned that getting a team to zero defects takes years, especially when dealing with legacy and fragile code. Her team achieved this, but it’s a journey worthwhile to take on.
Good post, Markus. Did Lisa mention anything about how the team chooses which defects to put on the backlog as stories and which to reproduce via automation? I’m thinking about those bugs that need a human brain to recognise as being wrong, odd or questionable.
Anything that can be fixed right away, should be fixed was her rule of thumb. More complex defects that might introduce rippling effects should maybe put on the backlog. Gojko also has a write-up on this: http://gojko.net/2010/10/05/bug-tracking-for-agile-teams/
Stephen has a great point – what about bugs that can’t be recognized by an automated test? Seems like you need another way to keep track of them. Right after my talk, a fellow named Brett (sorry, forgot his last name, it wasn’t Brett Schuchert) talked to me, he works in the games industry. The games require human visual testing – for example, if two textures don’t line up correctly, that would annoy the user and look bad. I don’t know how to write a test for that, though one of my tutorial participants yesterday also works in the games industry and says they have an automation tool developed in-house that saves them much tedious manual testing. Must get these two together and learn more about testing games.
This sounds interesting, though I think that filing an exploratory test charter for these areas is the better trade-off to do so. Read a book chapter about an approach for an upcoming book which I am not allowed to mention yet, but there might be more to come.
Thanks Lisa and Markus. That’s the answer I was hoping for. Most of my testing is data driven but there is a very strong human element. We ask questions like “Does the postal address that has been selected as ‘preferred’ match the input address well enough taking account of spelling errors or would another address altogether have been a better match (either on the list of possibles or not)?” Like the games analogy these are very human brain dependent. Some people have more tollerance of ‘error’ than others in this context.
I keep track of such defects using Mantis and good old fashioned Word documents.
Stephen
good post!
I like her (“tabula rasa”) idea of starting an experiment without a DTS (defect tracking system), limit the number of bugs by say 10, fix all bugs immediately (if they are important), and use cards on the story boards for the bugs.
I love the game of cat and mouse which resemble developer running away with the bugs and tester chasing him. In the good new world of Agile they are bed fellows and they raise and fix bugs as they are found without any tracking system in sight.Tracking system is helful for later stages of testing. Looking at colour coding bugs from colour coordinated Lisa it is a recipe for success. As I commented on Gojko blog, I attended one presentation 10 days ago where one project manager followed similar technique but they used Kanbas board to progress bugs in a very effective way. The thing that surprised me was that these guys new very little about Kanban just the notion but were able apply the technique so successfully that they are planning to train their developers in Kanban for all future projects.The work was carried out by the development team using this approach where testing team was completely separeate entity.They are re-thinking their strategy for future projects.
Similarly we used a Kanban board to track change requests from the customer last year during a project. Kanban is considered for operations to be very worthwhile to look into, but there might be different ways to get on top of your bugs or change requests as well.
As stated, visual and/or subjective issues may be impossible to verify automatically.
And you also miss the benefits of manual verification, that looks further and on side-effects.
Zero defects is self-fulfilling if you only report the issues you intend to fix?
I agree that not writing bugs, and fixing them right away, is the fastest way forward.
At the same time, a well-managed bug system can be your best source for quality-related information.
You might want to log bugs so they are visible and searchable to others (and yourself in the future) when generating test ideas, fixing similar issues, planning regression tests, investigating (customer) issues etc.
Hi, thank you for this post I agree with you that In the Agile view defect tracking is a waste in the Lean sense. Striving for zero defects is the goal to try to achieve there. very useful information