Reconnaissance - A Testing Story

Our application is so huge that we have teams of programmers assigned to separate modules. One day, the development Team Lead of the messaging module came up to me and said...

Team Lead: We are giving a release today with the changes to the messaging module.

Me: Today?

Time Warp -- About Two Months Ago:

The application has a messaging module. We added a feature to enable or disable this module for users. While testing the feature it was found that some user names did not appear in the UI, and so could not be added to the recipient list of the message. On further investigation it was found that this was a side effect of another module; the user creation module. Users are created by entering user profile data. It was assumed that once a user is created it will insert some default user preferences in the database. When implementing the new preference for the messaging module the code was written assuming that the preference values will always be found in the database. But it turned out that the preference data was not inserted when users were created with another feature; i.e. importing all user profiles in bulk from an Excel spreadsheet. This feature did not insert the default preferences. The feature was developed about five years ago. Why was this bug not detected earlier?

There aren't too many preferences. When dependent modules and features look for their respective preferences, the code assumes certain defaults. These defaults are the same as the default preferences that are inserted. So this went undetected all these years, without any noticeable side effects.

Since the release date was close, a quick fix was made that basically checked for the absence of the messaging preferences. This fix was marked to be refactored at a later time.


Time Warp -- About a Week from the Present:

Some performance improvements were being made to the application. Since the messaging module is widely used in the application, some of its database queries were optimized. In one of the optimizations, the extra messaging preference checking was removed. Of course everyone knew that this also required a bug fix for creating new users; i.e. the default preferences should always be inserted in the database. So that fix was done for the Excel spreadsheet import feature. Since it was suspected that the bug in the messaging list (the one where the user names did not appear in the UI) may reappear again, two testers were assigned to test it.


Time Warp -- Back to the Present:

I had been preoccupied the whole week trying to write automated test scripts using a combination of Fitnesse, Selenium RC, JUnit and the test harness I created 2 years ago. I abandoned its progress back then because of lack of resources and the difficulty of maintaining it. I was revisiting the progress with some new ideas I have picked up over the years. Due to this I was not able to give much time to followup on the testing that was done recently. This was an expected side effect and my peers and testers are well aware of it.

So now getting back to the conversation...

Team Lead: We are giving a release today with the changes to the messaging module.

Me: Today?

Team Lead: Yes. They (my testers) said it was ok.

It was a performance improvement. I heard a little about the changes that were made, but did not debrief the testers on their progress. I also play the role of a Test Toolsmith, and there are occasions when I disappear from testing to develop tools to speed up our testing effort. Most of the time these are not automated tests, but are tools that assists in test acceleration. I can probably write about those on another day. This is way more interesting to write about now.

My team uses Yammer to write short notes about what they are working on. Yammer is very similar to Twitter, but it lets you create closed groups with all your coworkers automatically, based on their company email address. It is kind of like twitter for the workplace. Yammer helps to keep me aware of what is going on with my team when I am away. I can interact with the testers later at certain chunks of time during the day about their tasks. Not a foolproof practice (in fact it is far from it), but it is working well for us during such times. When I put aside my Toolsmith hat after a week (or couple of weeks) we become more agile and start interacting all throughout the day.

Me: Ok. Let me talk to them about their tests just to followup.

Whatever hat I was wearing at that time, it was important that I got back to being a Test Manager immediately. I trust my testers, but I am paranoid by nature. Two of my good testers were working on this. So I wasn't anticipating a disaster. Yet I felt curious enough for a debrief with one of the testers (since the other tester was busy working on something else).

The debrief went well, and she seemed to have some good ideas about what to test. I also called one of the programmer's to come over and talk a little bit about any dependencies he was aware of. We didn't go too far with that. So I continued to bounce my test ideas with the tester. At one point I asked her...

Me: What problems did you find?

Tester: I didn't find any problems.

Me: I mean, did you find any problems after they made the changes?

Tester: Actually, I didn't find ANY problems. (She looked and sounded disappointed)

Me: You did not find ANY problems?

Ok, here is the thing. This was not a big issue, and it is not that we always found problems after a code change. In fact, sometime things are pretty rosy. But every time I hear this, I have bells ringing in my head. It was as if "I didn't find any problems" was a heuristic that makes me wonder. Were we looking for the right problems? Were there important tests we didn't think of? So we started talking a little more about other possible tests. She had checked those as well. Impressive. So, I asked her...

Me: How do you know that you did not verify the wrong data?

Illusions happen. I witnessed quite a few myself. In her case, this was a list of user names and titles, that were word wrapped, and the readability was not that great.

Tester: I created the user names and titles by labeling them with the kind of test I was doing. That's how I made sure.

I just love it when they can explain why they did what they did. It may seem that I am just making life difficult for my tester, but hey, things like this can go wrong, and in my opinion we need to be conscious about it. During the discussion I realized that she did not test the other features of creating a user profile. Yes, there are six ways to do this.

This is an application that is being built for more than five years. It has many features and not so obvious dependencies. A thorough regression test before each release is impossible with the limited resources we have, so we have to analyze dependencies to figure out what to test. Since the dependencies are so meshed, checklists haven't been working out that well for us during regression. We figured mind maps would be a useful way to remind us of dependencies. The problem is that once we put down detailed dependencies on the mind map it becomes very difficult to follow. We needed a tool to zoom in on specific nodes, hide the rest of the clutter and show its dependencies. Then zoom in on the specific dependent node and notice if it has its own set of dependencies. Then maybe follow the trail if necessary. An interesting tool that does something similar with mind maps was PersonalBrain. It is not very intuitive and does need some getting used to. Its user interface was not exactly what we had in mind, but it is still helpful when you want to manage complicated dependencies. Here is a snapshot of how the dependencies of creating user profiles looked like (the first one is zoomed out, and the second one is zoomed in on Create User to reveal more branches)...




Getting back to the story. Luckily the Team Lead was walking by, so I told him that we did not test all these other ways to create a user profile. I knew there wasn't much time left for the release, and it will take a while to test all these features, so I wanted to see if he thought we deserve more time. He said, they didn't make any changes there and these were developed a long time ago. There were no signs of any problems. Ok, so clearly this did not have any priority, and with what I had analyzed so far I didn't think it had much priority either.

Back to the debrief with the tester. While I was talking to her, I had asked another tester (who is also a Test Toolsmith) to pull up the source code change logs. Now that we had exhausted our test ideas, I wanted to see if I get any clue from looking at the code differences. It is worth noting that although I was once a programmer, I don't know much about this application's source code (which by the way is pretty huge). I never invested much time reviewing it. I don't usually analyze the code changes, but since this was dubbed as a very small change, and the code differences looked friendly enough, I thought I could dare to investigate.

I could only make out some query changes. The programmer was also kind enough to explain the changes. He calmly reiterated it was simply a performance tweak. He removed a check for a null value in a particular table column. That was basically it. He explained that this value would have been null when importing user profiles from an Excel spreadsheet. Even that was also fixed by another programmer. Also, they will be running a migration script on the database to update these null values. I knew about the migration script before, but now it was all making sense. I realized that everyone was very focused about this one particular feature. So I went to talk to the programmer who fixed the Excel import feature. He said that his fix was only for the import feature and it will not affect any other features for creating user profiles. Well, that's what he said. There are six ways to create a user profile, and these were implemented about five years ago. The original programmers were long gone, and this programmer was assigned to fix this very simple bug for this very specific case. He didn't know about 'all' the other ways of creating user profiles, and said he did not review the code for 'all' the other implementations either.

Suddenly I felt an urgency to test every known way to create a user profile before the release. Clearly there wasn't any time left, but from the evidence I gathered, although there was little proof that there may be a problem there, there was just too much uncertainty and too much at stake. The problem with this change was that, if it went live, and the other user profile creation procedures had the same bug, those users will suddenly stop appearing in user lists used to send messages. That would affect a major portion of existing users, and since we had more than 50,000 users, that could result in a customer support nightmare.

So I quickly instructed another tester (who usually deals with the application's administration related features) to start testing all the five ways to create a user profile, while I continued the debrief (had some new dependencies to discuss). I briefly told her what to look for, i.e. the database table values she should monitor, and to check the values of the messaging preferences when creating user profiles from the UI. Note that she was not a geek, nor was she very technical. But checking database values is not that difficult. Since we run Oracle, we use Oracle SQL Developer to open tables and filter values. This is very simple to do. In this particular case there is no complicated SQL query writing capabilities required. However, if anyone did need specific SQL queries, then someone with more technical experience would write that for them.

After about ten to fifteen minutes, the results of one of the tests were in...

Tester: Are you sure there will be a new entry in the table?

Time for me to look into the results. We have two database servers running. We usually keep one server for ongoing development changes to the database tables. Then these changes are made into migration scripts by the programmers and database administrators. These migration scripts are then run on the other server that has a similar snapshot as the live database. This is done to verify if the migration scripts causes any side effects.

I wanted to make sure she was checking the right database. So we modified some values from the UI to check if that value changed in the corresponding table. It did. I wanted to make sure she was extracting the correct user ID from the browser HTML, and verifying that in the database. She was. I wanted to make sure if our test procedure was correct. So we did the Excel import test again, and checked if it entered the relevant values in the preference table. It did. Clearly there was a problem.

I informed the Team Lead of this and told him we are checking the other methods as well. He quickly went to investigate the code with his programmer. After a while the next test result came in. Same problem. Then the next. Same problem. All other methods of creating a user profile had the same problem. It was finally decided that the release will be postponed to another day, until these get fixed.

Picture reference: afrotc.com