There, but for the grace of testing, go I
July 17th, 2010 | Published in Google Testing
By James A. Whittaker
I've had more than a few emails about "antenna-gate" asking me to comment and suggesting clever, stabbing rebukes to a fallen competitor. I might aim a few of those at my own team in the future, some were genuinely funny, but none of them will appear here. Instead I offer first a word of caution and second a reflection that my Mom used to intone whenever disaster occurred around her. It's called "counting your blessings."
First, a caution that those of us who live in glass houses really should keep stones at arms length. The only way anyone can rebuke Apple, without risk of waking up one morning sucking on their own foot, is if they write no software or have no users. Apple does a lot of the former and they enjoy many of the latter. Bugs like this make me sick when they are mine and nervous when they aren't. If any tester in the industry isn't taking stock right now then they either aren't producing any software or aren't in possession of any users, at least ones they wish to keep.
Second, taking stock has made me realize that I enjoy some important blessings that make the infinite task of testing so much more manageable. Indeed, the three blessings I count here are really the reason that testing doesn't fail more often than it does.
The Blessing of Unit Testing
I am thankful for early cycle testing thinning out the bug herd. In late cycle testing major bugs are often masked by minor bugs and too many of the latter can hamper the search for the former. Every bug that requires a bug report means lost time. There is the time spent to find the bug; time spent to reproduce and report it; time to investigate its cause and ensure it is not a duplicate; time to fix it, or to argue about whether it should be fixed; time to build the new version and push it to the test lab; time to verify the fix; time to test that the fix introduced no additional bugs. Clearly the smaller the population to begin with, the easier the task becomes. Solid unit testing is a tester's best friend.
The Blessing of Rarity
I am thankful that the vast majority of bugs that affect entire user populations are generally nuisance-class issues. These are typically bugs concerning awkward UI elements or the occasional misfiring of some feature or another where workarounds and alternatives will suffice until a minor update can be made. Serious bugs tend to have a more localized effect. True recall class bugs, serious failures that affect large populations of users, are far less common. Testers can take advantage of the fact that not all bugs are equally damaging and prioritize their effort to find bugs in the order of their seriousness. The futility of finding every bug can be replaced by an investigation based on risk.
Risk analysis is so important that we've built an internal tool to help guide testers in performing it. Code-named "Testify" this tool streamlines the process of risk analysis, at least the way we do it at Google. We're working on open-sourcing an early prototype in time for GTAC 2010 (I can hear my team cringing now ... "you promised it when?").
The Blessing of Repetition
I am thankful that user behavior is highly repetitive. There are features that enjoy heavy usage across user populations and features that are far less popular. Mobile phones are a good example of this. The phone is constantly establishing connections to networks. Certain features like making and receiving calls, texting and so forth are used more often than taking pictures or searching maps. The popularity of user applications is a matter of hard data, not guesswork. Knowing what users do most often, less often and least often means testing resources can be applied with a commensurate amount of force and that testing itself can be patterned after actual usage profiles.
Testers can gain a great deal from taking the user’s point of view and weaving usage concerns into the software testing process. Focusing on the user ensures that high impact bugs are found early and software revisions that break key user scenarios are identified quickly and not allowed to persist.
Apple may be the company in the news today, who knows who it will be tomorrow. Every company that produces software people care about has either been there or will be there. The job is simply too big for perfection to be an option. But there are key advantages we have that make the job manageable.
Put down the stones and make sure that what few blessing we testers possess are being exploited for everything they are worth. Hopefully, your company will be spared and the next time a company suffers such a bug you won't be the one making excuses. Perhaps you'll be lucky enough to be the one saying, "there but for the grace of testing go I."
I've had more than a few emails about "antenna-gate" asking me to comment and suggesting clever, stabbing rebukes to a fallen competitor. I might aim a few of those at my own team in the future, some were genuinely funny, but none of them will appear here. Instead I offer first a word of caution and second a reflection that my Mom used to intone whenever disaster occurred around her. It's called "counting your blessings."
First, a caution that those of us who live in glass houses really should keep stones at arms length. The only way anyone can rebuke Apple, without risk of waking up one morning sucking on their own foot, is if they write no software or have no users. Apple does a lot of the former and they enjoy many of the latter. Bugs like this make me sick when they are mine and nervous when they aren't. If any tester in the industry isn't taking stock right now then they either aren't producing any software or aren't in possession of any users, at least ones they wish to keep.
Second, taking stock has made me realize that I enjoy some important blessings that make the infinite task of testing so much more manageable. Indeed, the three blessings I count here are really the reason that testing doesn't fail more often than it does.
The Blessing of Unit Testing
I am thankful for early cycle testing thinning out the bug herd. In late cycle testing major bugs are often masked by minor bugs and too many of the latter can hamper the search for the former. Every bug that requires a bug report means lost time. There is the time spent to find the bug; time spent to reproduce and report it; time to investigate its cause and ensure it is not a duplicate; time to fix it, or to argue about whether it should be fixed; time to build the new version and push it to the test lab; time to verify the fix; time to test that the fix introduced no additional bugs. Clearly the smaller the population to begin with, the easier the task becomes. Solid unit testing is a tester's best friend.
The Blessing of Rarity
I am thankful that the vast majority of bugs that affect entire user populations are generally nuisance-class issues. These are typically bugs concerning awkward UI elements or the occasional misfiring of some feature or another where workarounds and alternatives will suffice until a minor update can be made. Serious bugs tend to have a more localized effect. True recall class bugs, serious failures that affect large populations of users, are far less common. Testers can take advantage of the fact that not all bugs are equally damaging and prioritize their effort to find bugs in the order of their seriousness. The futility of finding every bug can be replaced by an investigation based on risk.
Risk analysis is so important that we've built an internal tool to help guide testers in performing it. Code-named "Testify" this tool streamlines the process of risk analysis, at least the way we do it at Google. We're working on open-sourcing an early prototype in time for GTAC 2010 (I can hear my team cringing now ... "you promised it when?").
The Blessing of Repetition
I am thankful that user behavior is highly repetitive. There are features that enjoy heavy usage across user populations and features that are far less popular. Mobile phones are a good example of this. The phone is constantly establishing connections to networks. Certain features like making and receiving calls, texting and so forth are used more often than taking pictures or searching maps. The popularity of user applications is a matter of hard data, not guesswork. Knowing what users do most often, less often and least often means testing resources can be applied with a commensurate amount of force and that testing itself can be patterned after actual usage profiles.
Testers can gain a great deal from taking the user’s point of view and weaving usage concerns into the software testing process. Focusing on the user ensures that high impact bugs are found early and software revisions that break key user scenarios are identified quickly and not allowed to persist.
Apple may be the company in the news today, who knows who it will be tomorrow. Every company that produces software people care about has either been there or will be there. The job is simply too big for perfection to be an option. But there are key advantages we have that make the job manageable.
Put down the stones and make sure that what few blessing we testers possess are being exploited for everything they are worth. Hopefully, your company will be spared and the next time a company suffers such a bug you won't be the one making excuses. Perhaps you'll be lucky enough to be the one saying, "there but for the grace of testing go I."