Saying NO to ISTQB Software Testing Boards

Back in 2005, I participated in an initiation discussion of a new "Bangladesh Software Testing Board" under ISTQB. It was a very disappointing discussion. Instead of giving attention to how they would try to improve the skills of the testers or tester wannabes in Bangladesh, they were more concerned about who will be in the board and how they will register it legally. It was obvious that almost all of the individuals involved in this initiative were not testers. I would have said ALL, but maybe I had failed to notice someone. Just maybe. I brushed off this initiative as just a hype. I did not expect much would ever happen, and I was right.

In 2007, two nice gentlemen from this yet to be initiated board came to talk to me about my opinions about this initiative. I am sure I am not the only one they spoke to. I expressed my skepticism, and interestingly they said they understood many of my concerns. They had very little knowledge about how certification actually undermines our testing craft. They even had very little knowledge about the other so-called famous certifiers in the business. Even this time, non of them were testers. But atleast they had the courage to admit that they did not know much about testing.

Finally, a month back, I was called up by one of the Executive Committee members (a good man with good intentions), from this yet to be initiated board, to invite me to their Executive Committee. But I declined. I mentioned my disappointment in the lack of progress they were making (not that I actually wished they would make progress in certification) and that their mission is not clear to me. He spent the next 30 minutes to convince me about their seriousness to move beyond mere certification. He gave me hints of intending to bring certified trainers from abroad, and tried to encourage me to join them. He even suggested to call up a meeting with the other EC members to discuss this further, and clear my doubts.

I was actually a little flattered by all this (the dark side within me). But I still refused to join them and instead told him that I would rather like to be part of their planning on how they intend to train testers in Bangladesh, and what precautions they would take not to undermine the testing craft. I did not have to be in the Executive Committee for that. He finally agreed. To be honest, I doubt this approach will ever work out. I think they would again disappear for another year or so because of their historical lack of commitment.

You see, there is no way I can pursue my thoughts about the dangers of certification, if I am involved with one myself. I would be known as a hypocrite. Here is the hall of fame where I chose not to include my name.

Meanwhile I am continuing my advocacy against certifications in my local forum, and it has grown to be a very hot topic, requiring the full commitment of myself and others. The discussion grew to about 40+ posts in just one week, with experience reports of certified testers and those who suffered its consequences. One thing for sure, it is not easy to go against the tide. It started off with people being disappointed in my opinions about certification, but now things are looking very productive indeed.

I will probably be summarizing some of these discussion points in a blog post soon. But until then here is one of my latest response to the long thread.

Don't test that

Here is the story. We received a customer complaint that she is able to access some data that she was not supposed to. On reading the customer feedback it seemed to me that this was something we usually test for since the access control system is one of our prime sales pitch. So I asked the tester in charge of that particular feature to investigate the allegation. She came back reporting that she was able to reproduce the issue. She seemed tense, realizing the gravity of the situation. By the way, I usually never blame my testers when a problem is spotted in the wild, no matter how serious it is. She knew this, so her nervousness wasn't about fear of blame, but rather about why she overlooked this very important problem. I remained calm to prove to her that we were not playing the blame game. They know the drill for what comes next. I usually ask a series of questions about how we overlooked the issue, and then we start talking about how we can try to avoid missing similar problems in the future. She knew the questions I will be asking, so she wanted to cut the conversation short, by saying that she wanted to test it but the programmer told her not to.

You as a tester just discovered some serious problems with a new feature and report it to the programmer. The programmer shrugs off your claims and tells you not to test those scenarios since those are not important in his opinion. What do you do?

There is a dilemma here. On one hand, I can't accept the fact that I or any of my testers will intentionally blind ourselves from certain problem scenarios. On the other hand, if we do test the forbidden zone, we risk pissing off the programmers, and being accused of not utilizing our time crunch adequately. Being rapid testers, deadline stress is the norm.

There are a couple of ways I play this game. If I feel that these tests are actually very important to test right away, I try to convince my testers or the programmers with scenarios of the kind of problems I expect. Sometimes I would take examples of similar reported scenarios from Customer Support.

If that does not work I go for Plan B. When a programmer says "Don't test that", I take that to mean "Don't test that NOW". So instead of testing those right away, I would wait until things calmed down. What that means is that if we do not anticipate serious problems in the areas we are testing and we still have time before the release (which is pretty rare) we would go ahead and test the forbidden zone. Otherwise we would wait until after the release.

There have been cases where we did not return to these problems due to time constraints or plain forgetfulness and the worse happens, i.e. the problem is reported by the customer. In such cases my testers and I TRY (emphasis on try) not to scream "I told you so". When we decided not to test these scenarios it was a combined understanding between the programmers and us. However it is important to bring this new information to the attention of the programmers and their superiors so that we can avoid making similar strategic mistakes. So instead of gloating over our hunch I would say something like, "Although we agreed that this was not important enough to test, it seems we did not anticipate the current scenario". You have to admit that it was a hunch, since we couldn't come up with any credible scenarios to prove otherwise.

Blame games do not appeal to me. I believe being a Test Manager, it is important to trust the skills of my testers. They must be good since I hired them and have been working with them to improve their craft. I intentionally avoid a signoff process of their test strategies. I play more of an advisory role recommending test ideas I feel were overlooked or complementing them on a plausible intuition. I would even help them in coming up with scenarios to communicate with the programmers or reproducing unreproducible problems or be their peer tester or speed up their testing efforts. They realize I am a facilitator rather than a lawmaker. There will be times I would disagree with them, but my disagreement is not authoritarian. They always have autonomy over their testing. This in no way suggests that if an issue gets overlooked in a product I will use them as a scapegoat. I think that was clearly expressed in this writing.

I understand this is an unconventional way of managing a Test Team. Many managers I know prefer that their testers spell out all their test cases and get those approved by them. They would rejoice when they were right and the programmers were wrong or take punitive measures when issues popup that should have been tested (in their opinion). Many would argue that this is the only way to have control over what is being tested, and to ensure that issues don't go overlooked. However I feel it is only an illusion of control. I fail to see how a Test Manager can know more about the test subject than the tester who spends more time studying it, exploring it and experiencing it. It is inevitable that problems may be overlooked, but when it does I will be there trying to understand it with my testers. I also make it a point to take into account when the stakeholders are satisfied with our testing, and I have found that there are more cases of appreciation rather than dissatisfaction over unnoticed issues.

Faster builds increases testing time

Being a rapid tester, I try to invest in automating areas of my testing effort or using tools where the task requires muscle over mind. This accelerates my testing and allows me to utilize my frequent time crunch more effectively. Since testing time scarcity is a regular phenomenon, the R&D and selection of tools or to write code for such tasks is another challenge. In this post I will outline one case where my efforts paid off. Of course, there were cases where I reached a dead end after weeks of R&D and prototypes, but I will share that in another day.

My company develops a web application, that takes a significant time to build (compile) and deploy it to the application server. Usually a tester would get the latest source from the version control system, build it, and deploy it on the test server. This part was already automated with Apache Ant. Ant is primarily a build tool that runs XML scripts for doing all sorts of stuff, and is platform independent.

Even then, there are a few things the tester needs to do:

1. Inform the other testers that the application is being updated and they will need to temporarily stop the testing on the server.
2. Start the entire build process and monitor the progress at certain intervals, to see if it is completed. It is irritating when you return to the console to find that the build broke 6 minutes ago.
3. Inform the other testers that the application is ready for testing.
   

Although these tasks are not directly linked to testing, they are a prerequisite. If you are doing this a few times during the day, you can imagine the idle time of the tester executing the build process. Also, the longer he delays noticing the end of a build process, the longer everyone has to wait for the deployed application.

It would be great to be automatically notified when a build is starting or ended or broke. Then all the testers can go about their other tasks, such as following up on reported problems, writing session sheets (for Session Based Test Management). So I preferred notification through instant messengers since we all use that for communication.

Some clarifications:

• Jabber = Your google talk IDs are actually Jabber user IDs. So you are basically using a Jabber server to do instant messaging using google talk. You can host your own Jabber servers for intranet instant messaging.
• Spark = A jabber messenger. We use it with our own hosted Jabber server.

The solution I am providing, can even be used for MSN, Yahoo and AIM.

So here is what I did. I found a way that Ant will:

1. Notify all the testers in spark before the build process starts.
2. Sleep for 30 seconds so that it can be terminated if any tester requests it, and wants to delay deploying the latest updates.
3. Execute the entire build process.
4. Notify all the testers in spark that the application is ready for testing with the latest updates.
5. Notify all the testers in spark if there is a build error.
   

To do this I used imtask and ant-contrib, which are 3rd party libraries for Ant. Imtask has a jabber task to be used in Ant, to send instant messages to Jabber. It also supports instant messaging for MSN, Yahoo and AIM users. ant-contrib has the try/catch tasks I used in Ant to handle build errors.

Finally, for those who are interested, here is the code snippet from my build.xml that I run with Ant:

<taskdef name="jabber" classname="com.tfftech.ant.taskdefs.im.jabber.JabberTask" />
<taskdef resource="net/sf/antcontrib/antcontrib.properties"/>

<target name="all.with.notification" description="Let me handle it from here. And I will let you know when I am done!">
 <antcall target="send.message">
     <param name="param.message" value="${message.buildstarting}"/>
 </antcall>
 <echo message="*******Going to sleep for a while so that the build process can be stopped if necessary...*******"/>
 <sleep seconds="30"/>
 <echo message="*******Starting build process...*******"/>

 <trycatch property="failure.message">
     <try>
         <antcall target="all"/>

         <antcall target="send.message">
             <param name="param.message" value="${message.onsuccess}"/>
         </antcall>
     </try>
     <catch>
         <echo message="*******SOMETHING WENT HORRIBLY WRONG!*******"/>
         <echo message="*******PROBLEM: ${failure.message}"/>

         <antcall target="send.message">
             <param name="param.message" value="${message.onfailure}"/>
         </antcall>

         <fail message="*******There was a PROBLEM. Stopping build process...*******"/>
     </catch>
 </trycatch>
</target>

<target name="send.message">
 <jabber host="${jabber.server}" secure="true" from="${login.username}" password="${login.password}" message="${param.message}">
     <to>tester1@myjabberserver.com</to>
     <to>tester2@myjabberserver.com</to>
     <to>tester3@myjabberserver.com</to>
     <to>tester4@myjabberserver.com</to>
     <to>tester5@myjabberserver.com</to>
     <to>tester6@myjabberserver.com</to>
     <to>tester7@myjabberserver.com</to>
     <to>tester8@myjabberserver.com</to>
     <to>tester9@myjabberserver.com</to>
     <to>tester10@myjabberserver.com</to>
 </jabber>
</target>

No such thing as a YES or NO question

Maybe I exaggerated the title of this post, but I tend to use it as a heuristic (rule of thumb) when answering testing status related questions. I never assume that I am being asked a yes or no question. So I don't answer with a yes or no. Here are some examples of the questions I am asked by project stakeholders and how I prefer to answer them at different occasions. Majority of the answers I have already used and some I intend to.


Are you done testing the feature?
Answers:

• We are done testing the cases we wanted to test.
• We assumed certain important failure cases and tried to trigger those problems with our tests, within the time you gave. We did not find any problems with those.
• There are some important problems we found that gave us ideas to do more testing. Will we get more time?
• There are certain risks we still anticipate, and would want to run a few more tests. Will we get more time?
• The setup time took longer than expected and we could not execute all the tests we intended. But we did run the important tests first. Will we get more time?

You can always find more ways to test. The problem is that you are only given a limited time frame. So you need prioritize the important tests to execute. If you answer yes, then don't be surprised if you are chastised later when bugs are discovered after the release. If you answer no, then you may influence his unpleasant attitude, even before you get the chance to explain.


Can we release?
Answers:

• Well, we got some problems that may turn out to be serious issues when we release (I would sometimes give a brief outline of the issues). What do you think? Is it important to give the release anyway?
• We are still testing, but we are not finding many problems with our tests. But we don't anticipate the current problems as a major threat.
• There were some important problems we discovered and resolved, but that gave us new ideas to do more testing. Is it important to give the release anyway?
• I realize the bug count is very low, but we fear that the recent bug fixes may have caused side effects. We have witnessed some of these problems already. Do you think we can get the time to run some regression tests?

I don't think it is wise to actually answer if we should or should not release. It is not up to me to decide. I will give him the information he needs to make up his own mind.


Do you need more time?
Answers:

• More time would certainly help since I got some test ideas that could trigger some important problems.
• I can always come up with more tests if you give me more time.
• We did not find any problems. Do you want us to rethink the risks and try to come up with different tests?

There are cases where I am insisted less time and there are times where the reverse happens. You would assume testers would be happy getting more time, but that is not always the case. James Bach's Pinata and Dead Horse Heuristics sheds some light on how much should we test (thanks to Ben Simo for sharing that). I may be beating a dead horse, that may increase frustration and boredom of the test team. On the other hand, if I simply stop testing because I am not finding many problems, I may be overlooking some critical issues.


You will notice that some of the answers are interchangeable between the questions. My intension was to give an overview. There are other questions that may be addressed with the above answers, such as:

• Is it stable?
• Are there any more bugs?

Sometimes I get carried away by my confidence and answer too briefly. But every time I do, I remind myself to be true to the team. It is important to understand that I am not being difficult by not giving a straight answer. It is just that there is no straight answer. There are too many uncertainties and I feel I am only doing justice to the entire product team by stating the facts.

Converting the nonbelievers

For a long time I have been trying to deal with programmers who are not supportive of the testing cause. I suppose there is at least one in every company. They are usually senior programmers, but younger programmers are quick to learn. Pretty soon their population multiplies. They cannot comprehend how someone, who knows so little about their code, can possibly find important problems better than they can. Ironically, they are usually more cooperative with customers. Even then, I can't blame them since I am certain they had to deal with an abundance of frail misguided testers. To them, testers do not add value to their work. It took me a long time to realize that I need to break this assumption for them to believe.

When I was a programmer I had my share of delusions. My constant complains to management led them to believe that I should guide the testers. That was a tactical mistake (at least at the time). I considered myself a good programmer and so my allegiance remained with the programmers. I assumed that to be a better tester the testers need to be more like programmers, i.e. they need to penetrate the black box and understand the internals of the software. Those who could not meet this condition were not good testers in my judgment. Thankfully the sands of time shaped me to be wiser. As I executed and experienced the testing challenges, I am better equipped to deal the obstacles. I guess I did it the unconventional way, i.e. started as a Test Manager and then tried hard to learn testing. However, I have finally become a tester after a lot of disasters. You truly need to be a tester to lead testers.

Now, getting back to the other nonbelievers, most of whom may never get the opportunity I got. Let me share an account of how I handled such a case in one project. I got debriefed on the project when the project was close to release and was told to test it. Me and my team never met or communicated with the programmers of this project. We were instructed that an online bug tracker is the preferred means of communication, since the programmers were extremely busy. They were very confident about their code and were skeptical about the value our testing would add. We were given a couple of days to test and report the problems. I usually do not negotiate deadlines. Instead I try to understand and prioritize what to test within that time frame. The discussion turned out to be quite interesting, but that is another story.

During the discussion and demonstration of the software we tried to understand why they were so proud of it. It turned out they had already demonstrated the software to the customer and were in the final stage of fixing the bugs reported by them. So the problems we were identifying during the demonstration itself was addressed with, "the customer did not complain about it", or "they will never do that". This response was very important for me to note. We then kick started the testing.

I instructed my team to keep log of every problem they encountered, no matter how trivial it seemed. I knew this approach will be met with sharp criticism by the programmers. So I cautioned them not to report seemingly trivial problems until they discovered an important problem. Then they can report the important problem with high priority, followed by the so called trivial problems, all at the same time. This turned out to be extremely effective. It was as if the programmers were forgiving the testers for reporting low priority problems because they got some important problems to deal with.

We tried to report problems with the customer's perspective, trying to express cases where the customer may find it a problem. Whenever a tester found a problem, he would shout out the problem to the entire fleet. This avoided duplicate reports, hence giving less chance to the programmers to complain. With this method, other testers could analyze if there could be similar problems in their own testing areas, and could check dependencies.

We regrouped every two to three hours and discussed the important problems and our next charter. When we spoke to the envoy of the programmers and they told us that certain problems will not be fixed, we did not complain. If we felt that the programmers failed to realize the seriousness of certain problems, we simply elaborated the scenarios. We did not take the authority to dictate what must be fixed.

It all went pretty well. As the deadline approached, our time frame was increased, and then increased again, and finally increased again, so that we could keep testing and reporting problems they would fix. Some of the programmers finally visited us personally to fix some issues that were blocking our testing progress and appreciated our work. It was interesting how all this was achieved without insisting more time, without taking supreme authority of bug fixes and without any confrontation with the programmers, while the testers had a ball.