Category Archives: Testing

Software Testing

Testing and Management Mistakes

This is a response to a recent blog entry from Michael Bolton. He challenges to spot as many management mistakes as possible in a short dialog excerpt between a project manager and a tester. I will quote the dialog and walk through each of the problems I can spot. Though asked differently, I will not only mention those problems related to management, but also to the tester there.

Magnus the Project Manager: “Hey, Tim. Listen… I’m sorry to give you only two days notice, but we’ll be needing you to come in on Saturday again this week.”

Tim the Tester: “Really? Again?”

The manager does not give the reason for this action. He simply states there is the need for the tester. This sets up emotional pressure on the tester, who does not want to let the team fall. I’m not sure if the manager put this up by intention or just by coincidence.

Tim on the other does not start to challenge the claim. The project manager simply says they need him, especially. But maybe another tester could do the job. What’s his mission? How long should he test? To whom should he report his results? Who is the client? Instead, he simply asks from the personal perspective, which might indicate an emotional reaction to the claim from the project manager. So, let’s look how the manager responds to this.

Magnus: “Yes. The programmers upstairs sent me an email just now. They said that at the end of the day tomorrow, they’re going to give us another build to replace the one they gave us on Tuesday. They say they’ve fixed another six showstoppers, eight priority one bugs, and five severity twos since then, and they say that there’ll be another seven fixes by tomorrow. That’s pretty encouraging—27 fixes in three days. That’s nine per day, you know. They haven’t had that kind of fix rate for weeks now. All three of them must have been working pretty hard.”

The build is a replacement for a broken build. The programmers seem to have produced some lousy work the last time. Now, that’s interesting information. Despite this valueable information, the manager seems to rely on the facts as counted by the numbers. There seems to be some corporate metrics program, that let’s this manager look more particularly on the number of bugs fixed than how many further bugs could have been introduced by fixing those. Again, this information is available in any bug database, but the manager simply ignores this valueable information. Last, but not least, his interpretation in the last sentence indicates that “more bugs fixed” means “harder worked”. This mustn’t be the case. Another point that Jerry Weinberg made me aware of, is the fact, that there is nothing stated about how many incidents correlate to the mentioned bugs. There could have been 10 bugs fixed by a single code line change. Similarly there could be a single bug which needs fixing in several components. The numbers alone lie here. A final point: The manager should know if his programmers have worked hard , if he would have paid attention to what they were doing rather than looking at the numbers, only.

Tim: “They must have. Have they done any testing on those fixes themselves?”

Tim clearly should know better here than to give in to the manager. The developers mustn’t have worked very hard as pointed out before. Tim could know this better, if he paid attention to the past of the project. Just by stating that the only outcome here is that the developers were working hard, he proves his manager correct – in the manager’s eyes. If you believe something hard enough, your mind makes it a fact. This is inattentional blindness.

Magnus: “Of course not. Well, at least, I don’t know. The build process is really unstable. It’s crashing all the time. Between that and all the bugs they’ve had to fix, I don’t imagine they have time for testing. Besides, that’s what we pay you for. You’re quality assurance, aren’t you? It’s your responsibility to make sure that they deliver a quality product.”

Of course the developers did not test, how could they deliver that many fixes otherwise? There are several pointers that this managers ignores relevant information. He doesn’t know if they tested their code, the build prociess is unstable, but that doesn’t give him a clue, either. In combination with the Assurance Fallacy of Testing (“You’re quality assurance, … It’s your responsibility…”) there really is more need to fix bugs, than to fix them properly. Of course, the build process crashing all the time could indicate a problem in the quality of the code these programmers seem to produce.

Tim: “Well, I can test the product, but I don’t know how to assure the quality of their code.”

When someone is arguing irrationally, don’t try to argue with logic. “I know that you have many duties as a project manager, but doesn’t the broken build process point out, that there might something wrong?” could be a response to get the attention of the manager back to quality rather than delivery. The hidden message Tim wants to send here just doesn’t work in this context.

Magnus: “Of course you do. You’re the expert on this stuff, aren’t you?”

Without knowing this particular project manager for sure, I sense that he is basically just trying to sell the idea on testing to this poor tester. For any reasonable tester it’s obvious that he does not know anything about testing at this point. Instead, by giving kudos (“expert”), he tries to let the tester feel comfortable. This is the “washing machine selling management”. In this style management acts as if they were selling you a washing machine. Here, the selling involves the tester to be the right person to do overtime on the weekend. Anyone who thought about testing at least a tiny bit, should get to the conclusion, that testers don’t assure any code quality. But as pointed out, the manager simply wants to get this tester to work over the weekend by sacrificing even his own credibility.

Tim: “Maybe we could arrange to have some of the testing group go upstairs to work more closely with the programmers. You know, set up test environments, generate data, set up some automated scripts—smoke tests to check that the installation…”

This is a brilliant idea. Let testers and programmers work together to fix the team problem, help programmers get started with unit testing and the like. Too bad his claim is interrupted here by the manager, thereby undermining the testers management capabilities. Of course, the tester shouldn’t make suggestions on how to manage the project here.

Magnus: “We can’t do that. You have high-level testing to do, and they have to get their fixes done. I don’t want you to bother them; it’s better to leave them alone. You can test the new build down here on Saturday.”

At this point, the manager fires back by telling the tester how to test. Personally, I think this is a response to the tester’s attempt to manage the project. Of course, the fallacy of testers interrupting and interfering with the development is inherent here. Testers help developers get going. If the priority is to fix the bugs, rather than to fix them properly, then it’s more important to get some software. If the software does not need to work, I can hit any other requirement (The Zeroth Law of Software taken from Weinberg’s Quality Software Management Volume 2).

Tim: (pauses) “I’m not sure I’m available on Sa…”

Magnus: “Why not? Listen, with only two weeks to go, the entire project depends on you getting the testing finished. You know as well as I do that every code drop we’ve got from them so far has had lots of problems. I mean, you’re the one who found them, aren’t you? So we’re going to need a full regression suite done on every build from now until the 13th. That’s only two weeks. There’s no time to waste. And we don’t want a high defect escape ratio like we had on the last project, so I want you to make sure that you run all the test cases and make sure that each one is passing before we ship.”

Noticed that this manager does not even let the tester finish? Again? This indicates that the manager does not have any interest at all to get to know what the opinion from his tester is. It’s more important to have him working on Saturday. Opinions do not count. In addition. he puts more pressure on the tester here. The schedule is of course pressuring this manager, but it’s his job to manage to get a quality release.

In addition his claims in this response are not congruent to his earlier message. He cares about quality from the tester, but not about the quality from this who arrange quality in the code itself. This is ridiculous and I really do not want to be in Tim’s situation at this point. The manager again tells the highly-educated tester what he has to do. Mary Poppendieck explained to me that this is really a problem.

Once again the manager is ignoring the facts. Every “code drop” has had problems in the past. So, where are the code reviews? Where are the unit tests and walkthroughs? Where are the design reviews? Given the information the manager seems to have, he could really do better.

Tim: “Actually, that’s something I’ve been meaning to bring up. I’ve been a little concerned that the test cases aren’t covering some important things that might represent risk to the project.”

This response again is good, but he could do better. Telling the manager that the problem indeed might be in the way he manages the project with the information at hand, is ridiculous. The Candidate Product Rule says, “Actually, it ain’t nothin’ ’til it’s reviewed” (Weinberg, QSM Vol. 2, page 290). This manager seems to manage by the rule, that it’s nothing until it’s tested, but we can skip review easily. The claim Tim makes here is still valid, but he doesn’t respond to the things Magnus just raised. He could bring his point out more clearly by doing so.

Magnus: “That might be true, but like I said, we don’t have time. We’re already way over the time we estimated for the test phase. If we stop now to write a bunch of new test scripts, we’ll be even more behind schedule. We’re just going to have to go with the ones we’ve got.”

Ah, the common time creation myth. Time is not created, you have to take your time. Especially for quality products there is no free lunch. You have to plan accordingly, incorporating the time it takes. The problem that the project is already over estimate, might as well depend on the manager of the project. And the manager is falling for the test phase fallacy. Testing is not a phase, bugfixing might be. Last, not adapting to the project as it unfolds by creating new test cases may as well produce the problems of tomorrow. This vicious cycle the Magnus manages to get in here is very common.

Tim: “I was thinking that maybe we should set aside a few sessions where we didn’t follow the scripts to the letter, so we can look for unexpected problems.”

Magnus: “Are you kidding? Without scripts, how are we going to maintain requirements traceability? Plus, we decided at the beginning of the project that the test cases we’ve got would be our acceptance test suite, and if we add new ones now, the programmers will just get upset. I’ve told them to do that Agile stuff, and that means they should be self-organizing. It would be unfair to them if we sprang new test cases on them, and if we find new problems, they won’t have time to fix them. (pause) You’re on about that exploratory stuff again, aren’t you? Well, that’s a luxury that we can’t afford right now.”

Well, the claim from Tim to do some exploratory testing to fill the gaps for the uncovered areas is great. Magnus’ response to it is fatal to the project. Again the project manager does not adapt to the situation at hand, after the project started to unfold itself. By getting risk-averse coverage of those areas which are not covered by the initial test cases, the manager sets himself up for tomorrow’s problems. Oh, and of course, it’s better if the customer finds the bugs, rather than have the programmer fix them now before the loss of reputation. Last, but not least, exploration isn’t a luxury, neither is testing itself. They’re both a necessity for quality software. Skipping either or both is not only undermining testers’ efforts, but also comprising developers who very well observe what managers do and don’t do. Finally, the manager has misunderstood Agile in the sense to develop software without the weight of documentation, but has not understood that Agile development needs a team. By separating the testers from the developers the manager undermines every benefit he could get from really doing Agile development.

Tim: (pauses) “I’m not sure I’m available on Sa…”

Magnus: “You keep saying that. You’ve said that every week for the last eight weeks, and yet you’ve still managed to come in. It’s not like this should be a surprise. The CFO said we had to ship by the end of the quarter, Sales wanted all these features for the fall, Andy wanted that API put in for that thing he’s working on, and Support wanted everything fixed from the last version—now that one was a disaster; bad luck, mostly. Anyway. You’ve known right from the beginning that the schedule was really tight; that’s what we’ve been saying since day one. Everybody agreed that failure wasn’t an option, so we’d need maximum commitment from everyone all the way. Listen, Tim, you’re basically a good guy, but quite frankly, I’m a little dismayed by your negative attitude. That’s exactly the sort of stuff that brings everybody down. This is supposed to be a can-do organization.”

If it’s natural for Tim to come back all the weekends, he will be asked again. Of course he shows his struggle during this conversation, but it’s really for the manager to influence to get what he particularly wants him to do. Again, in the example stated, the manager ignores very well information which is available to him beside the information he gets from the testers. The pointer to “bad luck” just reminds me on Weinberg, who stated to replace the word “luck” in the phrase with “management”, when a managers refers to bad luck. We might as well prey to the software gods, if a whole company can be built upon management by luck. Of course, the manager’s claims about commitment and can-do organization are just institutionalized to make his argument and get Tim motivated. The problem lies in the decrease of motivation after this conversation. Tim’s trust for proper management has surely decreased, and a few more of these instances may let Tim leave the company searching for a better job opportunity.

Tim: “Okay. I’ll come in.”

Finally, Tim gives up. Never do this, though it might mean to continue fighting for ages or to lose the job. This is ok, as you didn’t like the job, anyways, now do you?

That’s my more elaborate response to Bolton’s challenge. Now, which parts did I miss?

Documentation fallacies

Today, I started off a series of documentation fallacies. Gerald M. Weinberg inspired this series. Since I didn’t have the initial inspiration quote handy at the time, I looked it up right now. As it is the initial one, I call it the documentation fallacy #0. It’s a citation from the reminders in Perfect Software… and other illusions about testing.

The Documentation Fallacy #0:

Believing that the mere existence of documents has some value.

Just because there is a document, means nothing. The document might just simply plainly lie around and start to get dusty. When no one is reading your document, then you have basically wasted all the time that you spent on preparing to write it, write it, review it, publish it. Period. Well, maybe you learned something as a side-effect, but you could have learned that one without writing a document that no one reads in first place. Ok, maybe you could have learned something by writing your thoughts down on the learning topic, sure, but still, the writing isn’t worth the paper that it is going to full. Save the rain-forest, remember?

So, just writing a document means nothing. It could be of no value because it has low quality, it could be of no value as it describes the life of Alfred E. Neumann, or it could be of no value as it describes how to the 8086 CPU was invented. If your project is not about any of these, you will simply not care about the content. More dramatically, your document could distract the people on your project and mislead. In the examples before I exaggerated to make a point, but what if the document was plainly wrong, and people started to work after it? Maybe you will end up with a project way behind schedule when you find out the problem right after delivery of the system. Doesn’t sound good, does it?

The Documentation Fallacy #1:
Thinking that more documentation means more thorough.

More documentation doesn’t mean anything. Personally, I find great value in those documents, that are as crisp as possible and point me out where to find the details if I need them. Usually I started to dig into a topic by getting a rough overview. After that I start to dig deeper and search for more details. This is the reason why to write requirements in a rough form before the work on the implementation begins. This concept can be found in use cases. Goals build the high level overview. There are different goal levels, and for more complicated business flows you write down an overview use case description alongside with finer-grained use cases on the subsystem or function level as you may see fit. Similarly when working on a user story, you start with a high-level overview and then start to dig deeper.

So, starting with a crisp overview is great. But when digging deeper you don’t need to document everything. The trivial cases are mostly irrelevant to document, because everyone already knows what shall be done in the login use case for example. Well, maybe you have a more complicated login use case, then document it, but if it, refuse to document the very obvious. Simplicity – the art of maximizing work not done – is the key here. But this does not mean that you shouldn’t document anything at all, of course. That would be stupid.

The Documentation Fallacy #2:
Thinking that since it’s documented, it gets read.

Just because it is written down somewhere, does not mean that anyone will read it. Maybe you’re lucky enough that someone reads it, but you can’t rely on it. Similarly you cannot rely on the fact that the people who actually need to read the document, have read it. Most of the time, documents simply lie around with no on reading them, covering dust – even virtually on the hard-drive of your computer. In the waterfall model we introduce the biggest problem here, since the documents don’t necessarily get read. I remember a case where we run into a problem during the user acceptance testing resulting from customer stakeholders not having read (properly?) the documents that we had sent them. Obviously this was the wrong time to find out that the software didn’t realize the functionality properly.

The Documentation Fallacy #3:
Thinking that since it gets read, it gets understood properly.

The Satir Interaction Model taught me, that not necessarily everything that gets read is understood in the same way for all readers of the document. The model splits up communication into four phases: intake, meaning, relevance, and response. During the intake each person takes in some parts of the message that she just heard. Therefore it gets distorted during the reception already. The meaning phase then interprets the message based on the personal rule model and previous knowledge of the receiver. This might distort the message even further. Overall, this may result in necessary details left out, or unnecessary details added to that message.

That said, it’s obvious that everyone may understand something different when reading a particular document. As the message and the meaning are distorted from what the receiver takes in (partially) and what she adds to the message based on previous experience, the understanding may well end up as being something completely different. So, just barely reading a document does not mean that the author’s intentions were understood.

The Documentation Fallacy #4:
Thinking that since it gets read, it actually gets used downstream.

Thanks for Paul Boos, who added this to my initial list. The two parts from the Satir Interaction Model left out of the discussion so far are the relevance and the response parts. On the relevance I assign to the distorted message how important I think that interpreted message is to me. If it is irrelevant, I might end up ignoring the message overall, thinking “fine, so what?”, or “don’t bother me.” I might also end up giving the relevance to completely ignore the message without responding to it. In the response I can decide how to react to me interpreted message, that I assigned some relevance. I can say something completely different, even out of context, like “nice weather outside, isn’t it?” This means that the mere existence of a document does not mean that I will follow it’s contents when reading it. It’s important what relevance I assign to the message and how I decide to respond to that message.

The Documentation Fallacy #5:
Thinking that since it is understood properly, it is used properly

Thanks for Jeroen Rosink for this addition. As derived from the Satir Interaction Model again, the response I take to my understanding of the document may vary to a great deal. This means that I might understand the document, but this does not necessarily mean that I will apply the message accordingly. In fact, most of the software I crossed was used unintentionally, in different ways, that the developers initially did not intent. The Windows scripting capabilities are one example of this coincidence, but there are many, many other examples out there. I’m sure you can name at least five on your own.

Please let me know of any additions I could make to that list. Hopefully, I raised some of your thoughts by now, so share your stories on misused or unread documentation with me.

Quality Software Management

Recently I finally finished the fourth volume of Jerry Weinberg‘s Quality Software Management series. As a wrap-up I decided to write a summary of what I learned and why you should read it yourself, too, if you haven’t already.

Introduction

In summary, the first book introduces into the cultural patterns that Jerry has found during his work with software companies. These cultural patterns include the Oblivious (Pattern 0), the Variable (Pattern 1), the Routine (Pattern 2), the Steering (Pattern 3), the Anticipating (Pattern 4) and finally the Congruent (Pattern 5) culture. He then digs deeper in the first three volumes on how to create a Steering culture. In the fourth volume he goes into the details to reach an anticipating culture.

Jerry provides a compelling view on the act of creating software, testing it and delivering it. Interestingly his view is right up-to-date, although he wrote the four books more than a decade ago. Next, I’m going to get into detail on each of the key lessons I took with me while reading through them.

Quality Software Management Volume 1 – Systems Thinking

In the first volume Jerry provides the cultural pattern model and relates it to the CMM levels and other approaches taken. Interestingly his view includes the culture around the people actually producing the software in first place, while other models focus on anything else but the people. First and foremost, Jerry defines quality as “value to some person”. Based on this definition, the question we have to ask in software development – which I define as programming, testing and delivering – is whose values counts most to us.

Additionally Jerry introduces the reader to systems thinking. That is a model to think through difficult and maybe complex situations in software projects. He shows many graphs applying systems thinking, thereby creating deep insights into each of the cultural models, as well as the dynamics around the creation and maintenance of software.

Using systems thinking Jerry introduces the non-linearity of aspects in software development. He revisits Brook’s Law and generalizes it to hold not only for late project, but for any project, when adding people, you’re making it more complicated, and maybe even later.

Jerry provides a compelling view on software error, distinguishes between faults and failures, and thereby provides a model for measuring, which is the topic of the second volume.

Quality Software Management Volume 2 – First-order measurement

In the second volume of the series, Jerry continues to dig deeper into the cultural model of software development organizations. Jerry presents a thorough discussion of the Satir Interaction Model. The Satir Interaction Model splits up communication into four major steps: intake, that is how I take in information; meaning, which is derived by making interpretations from the received message; significance, that is whether I start to bother or not; and finally the response on the initial message. All these steps happen in a fraction of a second in human communication.

The Satir Interaction Model is useful in situation where you start to make meaning too soon based on fractions of the data that you got. Often, this leads to mis-interpretations about the intentions of your communication counter-part and therefore not only to a poor dialogue, but also to a big misunderstanding. That’s why Jerry raises the Rule of Three:

If I can’t think of at least three different interpretations of what I received, I haven’t though enough about what it might mean.

In parallel to the discussion of the Satir Interaction Model, Weinberg continues to show the relevance for first-order measurements in management. Using systems thinking as introduced in the first volume of this series, he explains management actions and their results. He continues to dig into metrics that are useful for managing. Finally, he discusses zeroth-order measurements, that each management should be aware of. These measurements tell you on which things to keep an eye in order to manage your software project.

How to act upon these data is the topic of the third volume in this series.

Quality Software Management Volume 3 – Congruent action

In Congruent action Jerry introduces congruent communication. For any effective communication one has to respect the self position, the other position and the context about the communication. When leaving one or all out, you’re going to find yourself in an incongruent communication style. Leaving out the self-position leads to placating behavior, missing the other-position leads to blaming, leaving self and other out is a super-reasonable response, finally leaving all of them out is just an irrelevant coping style.

Additionally Jerry discusses the Myers-Briggs type indicator system. He introduces the four dimensions in that model. The first of the four letters says something on how I refresh my energy – introverts (I) for seeking self-reflection or extrovert (E) for interacting with people. The second letter describes how I take in information – either by facts from the sensation type (S) or by grasping the abstract concepts via intuition (N). The third letter says something how I make meaning – by pure logic in the thinking preference (T) or by grasping the feelings of the humans around me (F). The fourth and last letter states how I prefer to take actions – the judging (J) style prefers to settle decisions, while the perceiving (P) style prefers to leave options open. So, as I’m an INTJ, I prefer to take energy from being alone, intuitively grasp the abstract concepts, making meaning of them through logical thinking, and prefer to have things settled.

So, a congruent manager should consider their preferences and see the preferences from others using the Myers-Briggs type indicator model. Whenever I am aware that my message was not receipt correctly at my communication partner, I can take the choice to reflect and maybe present the information in a different format for my audience.

In addition Weinberg concludes in this third volume in the series how to reach a Steering culture built upon systems thinking, first-order measurements and congruent responses while leaving the transition to an anticipating culture for the fourth and last volume.

Quality Software Management Volume 4 – Anticipating change

In the fourth and last volume Jerry introduces the Satir Change Model. Since an anticipating culture relies heavily on change artists, this model comes in handy in forming such an organizational culture. Weinberg explains in great detail how people react upon foreign elements in the organization, and that too many threating external influences may result in people hiding in their basement. In addition he walks the reader along the change model and explains how to get from an old status quo to a new status quo and actually make and foster the necessary change.

Weinberg continues further how an anticipating organizations works through the overall software development process. Starting from meta-planning, tactical change planning, over to planning as an software engineer, this volume conducts how to make change stick in the organization. In addition he writes about process and process improvements. He lists processes commonly in practice by the time he wrote the book (1997), and discusses why it is important to know several process models as a change artist. Weinberg describes the differences between a process vision, the process model, and the process.

Finally, he gives a compelling view on things that are still relevant nowadays. Taking a closer look on how to terminate projects, and how to know that you should terminate or re-plan them, he gives a thorough overview of do’s and don’t’s in software development for the project manager. He discusses that requirements documents and design documents should be placed in a version control system alongside with the code. Considering that he wrote this book more than ten years ago, it struck me when I encounter multiple teams which are still not using any version control mechanism at all. Finally, Weinberg takes a look on tools, and how and when to introduce them.

The biggest gift in this volume are the lists of do’s and don’t’s in software projects alongside with the eleven commandments of Technology Transfer. His comparison of the waterfall model to other software development models may be seen as a bit outdated, but most of it is still relevant. Weinberg even touches the spiral model in this discussion, so portions of it also apply for modern Agile methodologies. Of course, the principle to know about many models and to know when they apply and should be used, is the key lesson to take away here.

Summary

Overall, Jerry provides many techniques and models, which come in handy at understanding the dynamics at play during a software project. Systems thinking helps you to analyze complex social systems in your project. The cultural model provides you with a useful view on how quality is seen in a particular organization, and which steps you might take for improvements. The Satir Interaction Model is useful in critical project situations, where your own measurement system is likely to be collapsed under the pressure of the situation.

Last, but not least, I’m glad that I got my copy from QSM Vol. 1 signed by Elisabeth Hendrickson. It was a coincident during last October, but she told me she felt honored to sign it for me.

The Deliberate Tester

Some while ago I started to write a tester’s novell. Heavily inspired by Elisabeth Hendrickson‘s article in the January 2010 issue of the Software Test and Performance magazine. Personally, I wanted to try myself on a tale of a tester after having read her article, and got in touch with Elisabeth. We exchanged some thoughts, some words, and voila, there was it.

I hoped it was that easy. Basically, what I wanted to tell, is the story about a new tester in a larger corporation. Based on my personal experience four years ago, I realized there was little previous knowledge that I could have taken with me for the job. Between the job interview and starting as a tester my father had died suffering from cancer for a little more than a decade. One day after the job interview I brought him to hospital. During family affairs I got the job position offering call. Having got this hard set-back, anyways, I started to dig into the field and learned. First by getting experience from colleagues, later by reading the classics and reflecting constantly on the practices I followed.

After having read a blog entry from Anne-Marie Charrett and one from Rob Lambert earlier, I decided to tell a tale about a tester getting introduced into our work. Alongside I want to spread the word on what we’re acutally doing, though there may be just few outside the testing field who will actually care. Therefore, I realized I needed to write it as an authentic, still fictive story, just like “The Craftsman” from Robert C. Martin.

So, I realized that I might split my work up into pieces. I decided to come up with three pieces so far from the original article. There is more room for further articles in the same manner. I’m still working on the story-line, so I would be glad to get some feedback on future episodes from the readers, which I might incorporate. The Software Testing Club put the first episode of the Deliberate Tester up on their blog. You can read it here, it’s called Session based exploration.

Big Test Design Up-Front

In yesterday’s European Weekend Testing session we had a discussion upcoming on whether or not to follow the given mission. The mission we gave was to generate test scenarios for a particular application. During the debriefing just one group of testers had fulfilled this mission on the letter generating a list of scenarios for testing later, the remaining two had deviated from the mission and tested the product, thereby providing meaningful feedback on the usefulness of the product itself. Jeroen Rosink already put up a blog entry on the session. He mentions that he define his own mission, and that it’s ok to do so during his spare time.

As mentioned I made my own mission also. I believe I am allowed to it because it is my free-time and I still kept to the original mission, define scenarios. for me the questions mentioned in the discussion were a bit the scenarios.

Of course, Jeroen is right, and he provided very valuable feedback in his bug reports. But what would I do at work when faced with such situation? Should I simply just test the already available application? Or should I do as I was asked? Well, it depends, of course. It depends heavily on the context, on the application, on the project manager, on the developers, on your particular skill level, maybe even on the weather conditions (nah, not really). This blog entry discusses some of these aspects I did not want to include generally in the chat yesterday.

Model building

One week earlier, Michael Bolton attended our session. He explained his course started with building a model of the application under test. Then he started to exercise the product based on that model thereby refining his own mental model of the application.

Jeroen had picked this approach. He also built a mental model of the application and went through the application with the model in order to refine it. On the other hand, Ajay Balamurugadas had build his model and translated that model into test scenarios completely.

To be clear here, both approaches are reasonable. Indeed, knowing when to use which is essential. The software engineering analogy teaches us, that we have to make decisions based on trade-offs. The trade-offs at play here are model building (thinking) as opposed to model refinement (doing). The more time I spent on thinking through the application in theory, the less time I have during a one hour session of weekend testing to actually test the application. On the other hand, the more time I spent on testing (and the more bugs I found by doing so), the less time I can spare to refine the model I initially made. Knowing the right trade-off between the two is context-dependent. I summarized this trade-off in the following graph.

In software testing there are more of these trade-offs. You can find most of them on page four of the Exploratory Testing Dynamics where exploratory testing polarities are listed. Thanks for Michael Bolton to point me out on this.

Information

Basically, the main distinction between the two missions followed is the fact, that Ajay used all his imaginary information available to build the model for test scenarios, while Jeroen questioned the available product to provide him some feedback on the course. While, we may not have an application available to help us making informed decisions about our testing activities, the situation in the weekend testing session was constructed to have a product in place, which could be questioned.

Indeed, for software testing it is very vital to make informed decisions. The design documents, the requirements documents, the user documentation rarely satisfies our call for knowledge about the product. Interacting with the product therefore can reveal vital information about the product and its shape. Gathering information in order to refine your model of the application at hand. Of course, for very simple applications or for programs in areas where you are an expert, this may be unnecessary. Again, the contextual information around the project provides the necessary bits of information about which path to follow.

Professionalism

So, professionally Ajay and Jeroen did a great deal of testing. The key difference I would make at work is that I would inform my customer on the deviation from the mission. There might be legal issues, i.e. a call to follow a certain process for power plant testing for example, that asks to follow the approach on the letter. Negotiating the mission with the customer as well as proposing a different mission when your professional sense calls to do so, is essential for an outstanding software tester. Deviating from the original mission is fine with me, as long as you can deal with the Zeroth Law of Professionalism:

You should take responsibility for the outcome of every decision you make.

Since we’re more often than not the professionals when it comes to testing software, we need to inform our customer and our client about deviations to the original missions. We have the responsibility to explain our decisions and to make a clear statement that it’s unprofessional to deliver non-tested software, for example. Of course, they might overrule your decision, but by then they’re taking over the responsibility for the outcome themselves. Of course, just because you lost a fight, this does not mean, that you should give up to raise your point and lose the battle.

Last, but not least, I hope that the other participants, Vijay, Gunjan Sethi, Shruti, do not feel offended, since I just mentioned Jeroen and Ajay here. They also did a great job of testing the product, of course.

FitNesse Search & Replace

Uncle Bob announced today the release of FitNesse 20100303. Please grab it from the FitNesse download page. Since I added a new function which I thought over the course of nearly the whole last year and which is not yet finished, let me try to introduce the idea I have with it.

The present

The feature I’m talking about is not very complex nor is it complicated. It’s a feature I was missing over the course of the last two years while working with FitNesse. It’s a simple regular expression based search & replace functionality. Well, thus far there has been a the possibility to do search & replace on the shell-level with regular tools like sed and others. So, what’s new about this? Well, at work we use perforce for revision control. Some while back I wrote a PerforceCmSystem for FitNesse to deal with opening files for writing, for add and for delete. Thereby we can store the FitNesse pages in our revision control system with real ease. Now, I get back to the need for a search & replace function inside FitNesse itself. On the shell we would need to check out each page to change, make the change, then check the changes back in. With search & replace I can now simply hit a button on a page after filling in stuff in two text fields, and then submit the changes made to the pages by FitNesse itself. Version control is done by FitNesse, as well as the change necessary. Great thing!

How does it work?

First of all consider the function to be experimental. There are no checks for consistency between the regular expression and the replacements. This made the implementation really easy, and there already some changes waiting for the next release to implement. That said, you will find two new text boxes on the refactoring page. In the first box you can enter regular expressions according to the description from the Java API documentation. You may use grouping enclosed with (), etc. On the second box you can insert a replacement, maybe referencing groups from the pattern in the first box. So, let’s walk through an example. Say, I have got some test pages, where Peter is used as the standard user. Now, for equality reasons, I would like to change Peter to Petra. Then I enter Peter in the first box, and Petra in the second one, hit the search & replace button, and watch the result popping up.

There is one thing you have to keep in mind. Search & replace works on the page hierarchy level. This means that any page below the page where the refactoring button was pressed, search & replace will work on. On any level above that nothing will be changed. So, you have to be aware where you start your replacements.

The future

There a two main streams for the future on FitNesse and refactorings. First, search & replace needs further evaluation and needs to grow. Uncle Bob suggested to include a two page flow, where I get all the results from the search on a second page and can choose among the replacements I would like to have – or as it is today, to replace all. This is already planned on pivotal tracker, and I would be happy to learn something new while implementing this.

Second thing I see coming is a refactoring for the tables inside FitNesse. With the underlying classes it’s very easy to write a custom refactoring for FitNesse that exchanges for example two columns from a table. For the following table:

abc
asdfqweryxcv

I might provide to exchange column a and c, resulting in the following table:

cba
yxcvqwerasdf

So, I would be happy on feedback on this feature and the future of refactoring in tools like FitNesse. The column replacer would be a neat feature, but I hope it’s just the beginning of a refactoring tool for wiki-pages. So, please leave me a comment what you think about it, or how you think search & replace might be used as well.

Configuration unit tests

James Bach encouraged me to think more thoroughly about the ideas that I came up with. In general for most ideas I try to give credits to the people who made me come up with it. While thinking over it, I didn’t know whom to contribute the idea of configuration unit tests, and I think I took more than idea to generate this one. So, let me introduce you to the context in which unit tests for product configuration makes sense, and the approach we took on it.

Continue reading Configuration unit tests

Writing automated business-facing tests

Since I work in a more traditional orientated environment, I’m facing some questions regarding the usage of test frameworks such as FitNesse, Robot Framework or Concordion. The biggest problem I hear very often is to start implementing fixture code before any test data in terms of test tables or html pages is available. Directly the scene from J.B. Rainsberger’s Integration tests are a scam talk comes to mind where he slaps the bad programmer’s hand. Bad! Bad, bad, bad! Never, ever do this. So, here is a more elaborate explanation, which I hope I may use to reference to my colleagues.

So, the first thing is to pick up the classics on the topic and check the documentation about it. So, let’s start with the classic FIT for Developing Software. So, the structure of the book is separated into a first part mentioning the table layouts, in the second part goes into detail about the classes for the implementation. So, Ward Cunningham and Rick Mugridge seem to follow this pattern. Great. Next reference, Bridging the Communication Gap. Gojko introduces there specifications workshops and specification by example. Both are based on defining the test data first, and later on automate them. This helps building up the ubiquitous language on the project at hand.

But there is more to it. Since test automation is software development, let me pick an example from the world of software development. Over the years, Big design Up-front has become an anti-pattern in software development. Though there are some pros to it, on the con-side there are that I may try to think about each and every case which I might need for my test data, but I may be wrong about that. So, just in case you are not from Nostradamus’ family, thinking about your design too much up-front my lead to over-design. This is why Agile software development emphasizes emergent designs and the simplest thing that could possibly work. Say, if I work now on ten classes, which I completely do not need when the test data is noted down, then I have spent precious time on building these – probably even without executing them. When later on the need for twenty additional classes arises, the time spent on those initial useless ten classes cannot be recovered to cope up. Additionally these ten classes may now make my suffer from Technical Debt, since I need to maintain them – just in case I may need them later. Maybe the time spent initially on the ten useless classes would have been better spent on getting down the business cases properly in first place – for those who wonder why your pants are always on fire.

Last, if I retrofit my test data to the available functions in the code, I have to put unnecessary detail into my tests. The FIT book as well as the Concordion hints page lists this as a malpractice or smell. For example, if I need an account for my test case and I am retrofitting it to a full-blown function call which takes a comma-separated list of products to be associated with the account, a billing day, a comma-separated list of optional product features and a language identifier as parameters, I would write something like this:

create accountmyAccount product1,product2,product3 5 feature1,feature2,feature3 EN

When I can apply wishful thinking to my test data, I would be able to write it down as brief as possible. So, if I don’t need to care about the particular products and features sold, I may as well boil the above table down to this:

create accountmyAccount

In addition to this simplification think about the implications a change for create account in the example above would have, when I need to a add a new parameter for example a the last billed amount for that account. If I came up with six-hundred test tables by the time of introduction of this additional feature, I would have to change all of those six-hundred tests. This time for changing these six-hundred tests will not be available to test the application. Wasted – by my own fault earlier!

In the end, it boils down to this little sentence I used to describe this blog entry briefly on twitter:

When writing automated business-facing tests, start with the test data (the what), not the fixture to automate it (the how). ALWAYS!

Deliberate Learning

Today after two weeks of reading, I finished Secrets of a buccaneer-scholar from James Marcus Bach. I was amazed on how he describes my attitude towards learning. Over the course of the last ten years I had always believed in more traditional ways to get educated. Well, when looking back at the education I got in school and the education I got at the university, and reflecting it over the stuff I need now on my day-to-day work, there is little that I have learned in those more traditional systems that help me now. Maybe ten percent of it does any service to me today. The remainder I learned myself ever since I started to work at a software company as a software tester – right when I didn’t know anything about testing software and how complex that field even is. This passion for learning has always driven myself and helped me get further ahead. When looking at my colleagues I was amazed that I seemed to be the only one, who was practicing self-education besides their day job. A key attitude I have built upon early on is the attitude for life-long learning – and I live it.

Now, reflecting back on the last four weekend of Weekend Testing in Europe, I noticed a pattern and a relationship to self-education – or buccaneer-scholarship how James calls it. Remembering back on our first session we had an online image processing tool and were asked to test it. There were two or three people who had experience with image manipulation, and the remaining three or four did not know anything about it. Since the mission was provided at the start of the session, this meant that no one really knew what was going to happen to prepare. Since no one dropped out of the session either, all participants were very, very eager to learn something new.

Indeed, when you face a situation which you don’t know anything about, some may react with fear about it. James taught me from his book the most valuable lesson: A buccaneer-scholar is not afraid of new situations. Even when you just now a tiny little piece of information about the product you’re going to test, or the project you’re asked to work upon, your deliberate learning attitude can and will help you. There is no big impediment to get to know something new. Maybe it gets you out of your comfort-zone, but imagine all the stuff you will know in addition to now. You will be able to make associative connections to stuff you already know. This is how your brain works. By making new connections, you may also reflect on stuff you already know and learn new things about that, either.

Deliberate learning can help you become used to learning as a competitive advantage. I see this happening for myself, I see it happening for James Bach (take a look on the videos on his buccaneer-scholar site), and I know that it may very well work for you as well.

Happy learning.