No such thing as a YES or NO question

Maybe I exaggerated the title of this post, but I tend to use it as a heuristic (rule of thumb) when answering testing status related questions. I never assume that I am being asked a yes or no question. So I don't answer with a yes or no. Here are some examples of the questions I am asked by project stakeholders and how I prefer to answer them at different occasions. Majority of the answers I have already used and some I intend to.


Are you done testing the feature?
Answers:

• We are done testing the cases we wanted to test.
• We assumed certain important failure cases and tried to trigger those problems with our tests, within the time you gave. We did not find any problems with those.
• There are some important problems we found that gave us ideas to do more testing. Will we get more time?
• There are certain risks we still anticipate, and would want to run a few more tests. Will we get more time?
• The setup time took longer than expected and we could not execute all the tests we intended. But we did run the important tests first. Will we get more time?

You can always find more ways to test. The problem is that you are only given a limited time frame. So you need prioritize the important tests to execute. If you answer yes, then don't be surprised if you are chastised later when bugs are discovered after the release. If you answer no, then you may influence his unpleasant attitude, even before you get the chance to explain.


Can we release?
Answers:

• Well, we got some problems that may turn out to be serious issues when we release (I would sometimes give a brief outline of the issues). What do you think? Is it important to give the release anyway?
• We are still testing, but we are not finding many problems with our tests. But we don't anticipate the current problems as a major threat.
• There were some important problems we discovered and resolved, but that gave us new ideas to do more testing. Is it important to give the release anyway?
• I realize the bug count is very low, but we fear that the recent bug fixes may have caused side effects. We have witnessed some of these problems already. Do you think we can get the time to run some regression tests?

I don't think it is wise to actually answer if we should or should not release. It is not up to me to decide. I will give him the information he needs to make up his own mind.


Do you need more time?
Answers:

• More time would certainly help since I got some test ideas that could trigger some important problems.
• I can always come up with more tests if you give me more time.
• We did not find any problems. Do you want us to rethink the risks and try to come up with different tests?

There are cases where I am insisted less time and there are times where the reverse happens. You would assume testers would be happy getting more time, but that is not always the case. James Bach's Pinata and Dead Horse Heuristics sheds some light on how much should we test (thanks to Ben Simo for sharing that). I may be beating a dead horse, that may increase frustration and boredom of the test team. On the other hand, if I simply stop testing because I am not finding many problems, I may be overlooking some critical issues.


You will notice that some of the answers are interchangeable between the questions. My intension was to give an overview. There are other questions that may be addressed with the above answers, such as:

• Is it stable?
• Are there any more bugs?

Sometimes I get carried away by my confidence and answer too briefly. But every time I do, I remind myself to be true to the team. It is important to understand that I am not being difficult by not giving a straight answer. It is just that there is no straight answer. There are too many uncertainties and I feel I am only doing justice to the entire product team by stating the facts.