October 1, 2007

No such thing as a YES or NO question

Maybe I exaggerated the title of this post, but I tend to use it as a heuristic (rule of thumb) when answering testing status related questions. I never assume that I am being asked a yes or no question. So I don't answer with a yes or no. Here are some examples of the questions I am asked by project stakeholders and how I prefer to answer them at different occasions. Majority of the answers I have already used and some I intend to.

Are you done testing the feature?

  • We are done testing the cases we wanted to test.
  • We assumed certain important failure cases and tried to trigger those problems with our tests, within the time you gave. We did not find any problems with those.
  • There are some important problems we found that gave us ideas to do more testing. Will we get more time?
  • There are certain risks we still anticipate, and would want to run a few more tests. Will we get more time?
  • The setup time took longer than expected and we could not execute all the tests we intended. But we did run the important tests first. Will we get more time?
You can always find more ways to test. The problem is that you are only given a limited time frame. So you need prioritize the important tests to execute. If you answer yes, then don't be surprised if you are chastised later when bugs are discovered after the release. If you answer no, then you may influence his unpleasant attitude, even before you get the chance to explain.

Can we release?
  • Well, we got some problems that may turn out to be serious issues when we release (I would sometimes give a brief outline of the issues). What do you think? Is it important to give the release anyway?
  • We are still testing, but we are not finding many problems with our tests. But we don't anticipate the current problems as a major threat.
  • There were some important problems we discovered and resolved, but that gave us new ideas to do more testing. Is it important to give the release anyway?
  • I realize the bug count is very low, but we fear that the recent bug fixes may have caused side effects. We have witnessed some of these problems already. Do you think we can get the time to run some regression tests?
I don't think it is wise to actually answer if we should or should not release. It is not up to me to decide. I will give him the information he needs to make up his own mind.

Do you need more time?
  • More time would certainly help since I got some test ideas that could trigger some important problems.
  • I can always come up with more tests if you give me more time.
  • We did not find any problems. Do you want us to rethink the risks and try to come up with different tests?
There are cases where I am insisted less time and there are times where the reverse happens. You would assume testers would be happy getting more time, but that is not always the case. James Bach's Pinata and Dead Horse Heuristics sheds some light on how much should we test (thanks to Ben Simo for sharing that). I may be beating a dead horse, that may increase frustration and boredom of the test team. On the other hand, if I simply stop testing because I am not finding many problems, I may be overlooking some critical issues.

You will notice that some of the answers are interchangeable between the questions. My intension was to give an overview. There are other questions that may be addressed with the above answers, such as:
  • Is it stable?
  • Are there any more bugs?
Sometimes I get carried away by my confidence and answer too briefly. But every time I do, I remind myself to be true to the team. It is important to understand that I am not being difficult by not giving a straight answer. It is just that there is no straight answer. There are too many uncertainties and I feel I am only doing justice to the entire product team by stating the facts.


  1. Wonderful post. I believe it is better for testers to provide management with information to support decision making than to make the decisions for management. When testers answer with simple "yes" or "no" answers, they are making decisions and may be usurping someone else's authority.

    There are many managers and executives that are happy to pass the responsibility for quality to testers. When testers make decisions by providing overly simplistic answers to complex questions, they may be taking on responsibility (and blame) for the consequences of those decisions.

    Ben Simo

  2. Ben,

    How true. Yet it is very surprising that many testers and test managers long for this responsibility, not realizing the consequences. It is as if by accepting such authority they have achieved some shred of recognition from management. They fail to realize that there are other forces that influence the success of a product, apart from bug fixing.

  3. >>> More time would certainly help since I got some test ideas that could trigger some important problems

    You sound more like Michael Bolton and James Bach ...(especially like Michael)

    Good to see another Rapid/Context Driven Tester on the horizon ...

    Keep blogging


  4. Shrini,

    Thanks for the complement. I think it will be appropriate to quote from an email conversation I once had with Pradeep:

    "Much of my testing values are based on what I read from blogs, articles and discussions contributed by James, Michael and recently you (Pradeep). The concepts and emotions I reflect are usually not my own creation, but rather the coupling of my acquired knowledge with my own experiences."

  5. I like the way this post is structured and i have my own experience on the title of this post- "No such thing as a YES or NO question".
    What i have experienced during release meetings and the overall status meetings is a kind of collaborative decision making i.e. in which each concerned group involved has to give its independent opinions/decisions on how they feel about the release and trust me how much ever i tried to give generic answers (ones mentioned above and many more), every time it boils down to the bottom line- "Yes" or "No". Probably, giving answers to the questions provides a way to arrive at a final "Yes" or "No". And its same for all the concerned departments i.e. Testing, Development, Technical Writers etc. Each group has its say and the final decision is made after hearing from everyone. After all, Quality is a collaborative responsibility and not just a responsibility of Testing group.
    Note: the above comment is primarily for the Answer to question- “Can we Release ?”

    Anuj Magazine

  6. Anuj,

    To ship or not to ship, that is the question. You gave a good example of how all parties get involved in answering it. That is the way to go.

  7. Great post. Occationally you have to be careful for any late defects that might pop up during our testing. But again its hard to find out all the defects during our testing. We can do as much as we can do and then just wait for something more to come up and be agile to its outcome. Thanks

  8. Chou,

    It is not only hard, but impossible to find all the bugs during testing. I think that is what you meant. :) Thanks.

  9. Excellent writing...Thanks Sajjad bhai to share such an important topic for the SQA/Testing team. I love to read your thoughts and also agree some point of view those you have mentioned in your writing. I also believe that only written test case can not say a software is error free. What I feel there is no alternative of more testing in different way. Also regression cycle is very important to get new issues. Thanks.