Monday, July 4, 2011

Testing Mindset: Should we compete against each other?

I participated in a recent competition conducted by
There was a restriction of 40 testers and limited to 4 days contest. I saw few new testers join this contest. Whenever I join the contests, I do not jump in right from start. I wait for few hours and watch the bugs logged. I observe what areas are being tested, what testing oracles are being applied and so on. One of the advantages of starting late is that I get to know the thought process behind bug validation. What kind of bugs are termed as invalid and what bugs are accepted - these are useful information for me participating in the contest.

So, this time too I waited for sometime before starting my testing sessions. There were close to 20 bugs logged and some of them were validated too. The top three listed on the leaderboard were unknown to me. I love this challenge. New faces, new application and this application was a bit tough to understand. I felt the users had more of read-only access and no rights to create the data. It was different from the usual applications where the bugs danced right in front of you.

End of Day 01: Total bugs logged: ~200
Day 02: ~350
Day 03: ~400

Once the competition ended, a thought stuck me:
What made the testers log so many bugs even though the contest was a tough one in terms of easy bugs?
Is it the mindset of *find bugs at any cost* or *the five prizes* or *the competition mode* that brought out so many bugs out in the open?

In the daily projects, most of the times - there is a bug pattern: xx number of bugs. I have rarely seen so many bugs logged in such a less time. I do agree that the number of testers is different and might be the biggest factor in the number of bugs logged. There is obvious difference in having forty testers testing an application and say five testers testing an application.

It might not be feasible to add so many testers to every project. So, my question is: 
Should the testers test with a competition mindset once a while? 
It might be a good idea to engage a group of testers to test as if there were in a competition and there were prizes for the top testers. Maybe spend ten hours/week and see how it works...

I am going to try it out with my team in few days time :) Idea worth trying?

Image Credits:


Joe said...

"Should the testers test with a competition mindset once a while?"

Answer - very, very seldom.

It might be fun if presented as an occasional game, but not as a job.

I hope that most of the time your professional work doesn't simply involve finding "as many bugs as you can before others find them". Instead I hope strive to ship the best software you can within the constraints of the business.

If the former, then you rush to build your bug count numbers at the expense of everything else. You don't help others. You don't write documentation. You write short, often poor, bug reports. You don't verify any fixes. You don't show developers how to reproduce bugs. You don't seek the very hard-to-reproduce bugs. You never find ways to avoid bugs in the first place.

Competitions are inherently a measure-and-reward process. Be careful what you measure, and what you reward.

Kieron said...

I agree with Joe in that the aim is to ship the best software within given constraints.

What I liked is when the test manager offered a chocolate fish for the bug of the week. In NZ a chocolate fish is a confection made of marshmallow coated in chocolate, in the shape of a fish - in other words, a fun prize that costs not very much.

This set the scene for friendly competition, and people had fun earning their prizes!

Ajay Balamurugadas said...

Hi Joe & Kieron,

Thanks for the comments.
Yes, I totally agree with the comments.

This highlights two things:
1. Testers do many more things which are important to the overall project.
2. There are many activities which prevent testers from bug-hunting.

Maybe, there are some activities which could be minimized to increase bug-hunting.


Anuj Sharma said...

It depends. If the competition is to find the most number of bugs then NO. If it is to improve the quality of the system then YES.