July 23rd, 2009 | Published in Google Testing
There are two communities who regularly find bugs, the testers who are paid to find them and the users who stumble upon them quite by accident. Clearly the users aren’t doing so on purpose, but through the normal course of using the software to get work (or entertainment, socializing and so forth) done failures do occur. Often it is the magic combination of an application interacting with real user data on a real user’s computing environment that causes software to fail. Isn’t it obvious then that testers should endeavor to create such data and environmental conditions in the test lab in order to find these bugs before the software ships?
Actually, the test community has been diligently making an attempt to do just that for decades. I call this bringing the user into the test lab, either in body or in spirit. My own PhD dissertation was on the topic of statistical usage testing and I was nowhere near the first person to think of the idea as my multi-page bibliography will attest. But there is a natural limit to the success of such efforts. Testers simply cannot be users or simulate their actions in a realistic enough way to find all the important bugs. Unless you actually live in the software you will miss important issues.
It’s like homeownership. It doesn’t matter how well the house is built. It doesn’t matter how diligent the builder and the subcontractors are during the construction process. The house can be thoroughly inspected during every phase of construction by the contractor, the homeowner and the state building inspector. There are just some problems that will only be found once the house is occupied for some period of time. It needs to be used, dined in, slept in, showered in, cooked in, partied in, relaxed in and all the other things homeowners do in their houses. It’s not until the teenager takes an hour long shower while the sprinklers are running that the septic system is found deficient. It’s not until a car is parked in the garage overnight that we find out the rebar was left out of the concrete slab. The builder won't and often can't do these things.
And time matters as well. It takes a few months of blowing light bulbs at the rate of one every other week to discover the glitch in the wiring and a year has to pass before the nail heads begin protruding from the drywall. These are issues for the homeowner, not the builder. These are the software equivalents of memory leaks and data corruption, time is a necessary element in finding such problems.
These are some number of bugs that simply cannot be found until the house is lived in and software is no different. It needs to be in the hands of real users doing real work with real data in real environments. Those bugs are as inaccessible to testers as nail pops and missing rebar are to home builders.
Testers are homeless. We can do what we can do and nothing more. It’s good to understand our limitations and plan for the inevitable “punch lists” from our users. Pretending that once an application is released the project is over is simply wrong headed. There is a warranty period that we are overlooking and that period is still part of the testing phase.