Converting the nonbelievers

For a long time I have been trying to deal with programmers who are not supportive of the testing cause. I suppose there is at least one in every company. They are usually senior programmers, but younger programmers are quick to learn. Pretty soon their population multiplies. They cannot comprehend how someone, who knows so little about their code, can possibly find important problems better than they can. Ironically, they are usually more cooperative with customers. Even then, I can't blame them since I am certain they had to deal with an abundance of frail misguided testers. To them, testers do not add value to their work. It took me a long time to realize that I need to break this assumption for them to believe.

When I was a programmer I had my share of delusions. My constant complains to management led them to believe that I should guide the testers. That was a tactical mistake (at least at the time). I considered myself a good programmer and so my allegiance remained with the programmers. I assumed that to be a better tester the testers need to be more like programmers, i.e. they need to penetrate the black box and understand the internals of the software. Those who could not meet this condition were not good testers in my judgment. Thankfully the sands of time shaped me to be wiser. As I executed and experienced the testing challenges, I am better equipped to deal the obstacles. I guess I did it the unconventional way, i.e. started as a Test Manager and then tried hard to learn testing. However, I have finally become a tester after a lot of disasters. You truly need to be a tester to lead testers.

Now, getting back to the other nonbelievers, most of whom may never get the opportunity I got. Let me share an account of how I handled such a case in one project. I got debriefed on the project when the project was close to release and was told to test it. Me and my team never met or communicated with the programmers of this project. We were instructed that an online bug tracker is the preferred means of communication, since the programmers were extremely busy. They were very confident about their code and were skeptical about the value our testing would add. We were given a couple of days to test and report the problems. I usually do not negotiate deadlines. Instead I try to understand and prioritize what to test within that time frame. The discussion turned out to be quite interesting, but that is another story.

During the discussion and demonstration of the software we tried to understand why they were so proud of it. It turned out they had already demonstrated the software to the customer and were in the final stage of fixing the bugs reported by them. So the problems we were identifying during the demonstration itself was addressed with, "the customer did not complain about it", or "they will never do that". This response was very important for me to note. We then kick started the testing.

I instructed my team to keep log of every problem they encountered, no matter how trivial it seemed. I knew this approach will be met with sharp criticism by the programmers. So I cautioned them not to report seemingly trivial problems until they discovered an important problem. Then they can report the important problem with high priority, followed by the so called trivial problems, all at the same time. This turned out to be extremely effective. It was as if the programmers were forgiving the testers for reporting low priority problems because they got some important problems to deal with.

We tried to report problems with the customer's perspective, trying to express cases where the customer may find it a problem. Whenever a tester found a problem, he would shout out the problem to the entire fleet. This avoided duplicate reports, hence giving less chance to the programmers to complain. With this method, other testers could analyze if there could be similar problems in their own testing areas, and could check dependencies.

We regrouped every two to three hours and discussed the important problems and our next charter. When we spoke to the envoy of the programmers and they told us that certain problems will not be fixed, we did not complain. If we felt that the programmers failed to realize the seriousness of certain problems, we simply elaborated the scenarios. We did not take the authority to dictate what must be fixed.

It all went pretty well. As the deadline approached, our time frame was increased, and then increased again, and finally increased again, so that we could keep testing and reporting problems they would fix. Some of the programmers finally visited us personally to fix some issues that were blocking our testing progress and appreciated our work. It was interesting how all this was achieved without insisting more time, without taking supreme authority of bug fixes and without any confrontation with the programmers, while the testers had a ball.

Are you paying attention?

In one of my testing consultancy sessions, I was demonstrating the testing of a JavaScript calendar widget in a web application developed by the client. I started off explaining that we should use the windows calendar as an oracle to help us discover problems in the widget. I opened the webpage containing the widget and the windows calendar next to each other.

I was demonstrating this on a projector which only supported a resolution of 800x600. So if I wanted to keep the windows calendar and webpage always visible, my webpage real-estate becomes considerably reduced. To the surprise of my audience I started resizing the browser window so that I can see more of the region surrounding the widget, which seemed unnecessary since the widget only took a small portion of the entire webpage. I explained that I want to be aware of problems surrounding the widget while I am executing my tests on the widget. I did confess that I do not know what kind of problems could appear in those regions but it doesn't hurt to pay attention.

This was a demonstration of exploratory testing. I was narrating my choices of executing certain tests that I generated on the fly, and why I chose to ignore other tests that I considered as equivalent or not related to my current test objective. A few minutes into the testing, one of my colleagues discovered that a button's outline at the corner of the web page was blinking as I was invoking the widget. The audience appreciated this discovery since they could relate to my initial preemption. I myself did not notice this due to inattentional blindness.

Before we continue, I think a good way to explain inattentional blindness is with the example of a magician's performance. Ever notice how a magician sometimes tries to distract you away from his actual trickery by trying to focus your attention on something else. For example, when a coin that you can swear was in the palm of the magician's hand disappears into thin air when he unfolds his fist. In reality, the coin was swiftly removed while you were paying attention to where the illusionist wanted and hence the perception of magic. This is called inattentional blindness. You can't see what is right in front of you, simply because you were paying attention to something else.

The irony of my episode was that I did not discover the problem even though my intention was to pay attention to such peculiarities. Also, exploratory testing is supposed to be better at minimizing inattentional blindness. This symptom is advocated as being more common during scripted testing. So I tried to analyze this phenomenon and came to the following explanation.

Since I was sitting close to a large projection of the screen on the wall, my visual field was limited. This made me more prone to inattentional blindness. Also I was frequently alternating my attention:

• To find important problems in the widget.
• To explain to the audience the reasons for my actions.
• To make sure my explanations were comprehensible.
• To address queries from the audience.

Now, some may be wondering why I went through the trouble for a seemingly trivial issue. My concern is what if it was not a minor problem and it went undetected. I want to be aware of the constraints that may hinder my discovery of important problems. Being a Tester that is a big deal.

How could so many get it wrong?

I come from the part of the globe called Bangladesh, known by many for its floods, poverty and corruption. They all got it wrong, just like many got testing wrong. But this blog is not about me or my country. It is about my experiences in software testing and how it can inspire others.

Michael Bolton said I sound fascinating. James Bach said I write very well. Pradeep Soundararajan said I should start a blog. So here I am. Even though, this does not mean they endorsed me, it did encourage me.

I like to think that I got testing right from the unlikeliest of conditions, because:

• Fortunately, I was forced into testing since my company was in a crisis and thought I was the best programmer for the job.
• Fortunately, I could not find anyone in my proximity to answer my queries on software testing.
• Fortunately, there were very few, yet boring, books on software testing at local book stores.
• Fortunately, there were no training centers on testing certifications.
• Fortunately, the seminars I attended were pretty shallow in content.
• Fortunately, I had to explore my curiosity on my own.
• Fortunately, I was bored with testing and wondered why others could write about it with so much passion.
• Fortunately, I was always under deadline pressure and was denied my demand for more testing time.
• Fortunately, I had the courage to try out my assumptions and was ridiculed for it.

I questioned the applicability of everything I read, heard, saw and experienced. In my quest I came across the writings of Cem Kaner, Bret Pettichord, James Bach, Michael Bolton and few others who made sense of it all. Yet I questioned their reasoning to my context.

My mistakes taught me what to avoid, while the experiences of others intrigued me to investigate further.

No wonder so many still get it wrong.

"Testing is the INFINITE PROCESS of comparing the INVISIBLE to the AMBIGUOUS so as to avoid the UNTHINKABLE happening to the ANONYMOUS!" -- James Bach

I accept the challenge...