How Google Tests Software – Part Six
May 2nd, 2011 | Published in Uncategorized, Google Testing
By James Whittaker
The Life of an SET
SETs are Software Engineers in Test. They are software engineers who happen to write testing functionality. First and foremost, SETs are developers and the role is touted as a 100% coding role in our recruiting literature and internal job promotion ladders. When SET candidates are interviewed, the “coding bar” is nearly identical to the SWE role with more emphasis that SETs know how to test the code they create. In other words, both SWEs and SETs answer coding questions. SETs are expected to nail a set of testing questions as well.
As you might imagine, it is a difficult role to fill and it is entirely possible that the low numbers of SETs isn’t because Google has created a magic formula for productivity but more of a result of adapting our engineering practice around the reality that the SET skill set is really hard to find. We optimize on this very important task and build processes around the people who do it.
It is usually the case that SETs are not involved early in the design phase. Their exclusion is not so much purposeful as it is a by-product of how a lot of Google projects are born. A common scenario for new project creation is that some informal 20% effort takes a life of its own as an actual Google branded product. Gmail and Chrome OS are both projects that started out as ideas that were not formally mandated by Google but over time grew into shipping products with teams of developers and testers working on them. In such cases early development is not about quality, it is about proving out a concept and working on things like scale and performance that must be right before quality could even be an issue. If you can't build a web service that scales, testing is not your biggest problem!
Once it is clear that a product can and will be built and shipped, that's when the development team seeks out test involvement.
You can imagine a process like this: someone has an idea, they think about it, write experimental code, seek out opinions of others, write some more code, get others involved, write even more code, realize they are onto something important, write more code to mold the idea into something that they can present to others to get feedback ... somewhere in all this an actual project is created in Google's project database and the project becomes real. Testers don't get involved until it becomes real.
Do all real projects get testers? Not by default. Smaller projects and those meant for limited users often get tested exclusively by the people who build it. Others that are riskier to our users or the enterprise (much more about risk later) get testing attention.
The onus is on the development teams to solicit help from testers and convince them that their project is exciting and full of potential. Dev Directors explain their project, progress and ship schedule to Test Directors who then discuss how the testing burden is to be shared and agree on things like SWE involvement in testing, expected unit testing levels and how the duties of the release process are going to be shared. SETs may not be involved at project inception, but once the project becomes real we have vast influence over how it is to be executed.
And when I say "testing" I don't just mean exercising code paths. Testers might not be involved from the beginning ... but testing is. In fact, an SET's impact is felt even before a developer manages to check code into the build. Stay tuned to understand what I am talking about.
The Life of an SET
SETs are Software Engineers in Test. They are software engineers who happen to write testing functionality. First and foremost, SETs are developers and the role is touted as a 100% coding role in our recruiting literature and internal job promotion ladders. When SET candidates are interviewed, the “coding bar” is nearly identical to the SWE role with more emphasis that SETs know how to test the code they create. In other words, both SWEs and SETs answer coding questions. SETs are expected to nail a set of testing questions as well.
As you might imagine, it is a difficult role to fill and it is entirely possible that the low numbers of SETs isn’t because Google has created a magic formula for productivity but more of a result of adapting our engineering practice around the reality that the SET skill set is really hard to find. We optimize on this very important task and build processes around the people who do it.
It is usually the case that SETs are not involved early in the design phase. Their exclusion is not so much purposeful as it is a by-product of how a lot of Google projects are born. A common scenario for new project creation is that some informal 20% effort takes a life of its own as an actual Google branded product. Gmail and Chrome OS are both projects that started out as ideas that were not formally mandated by Google but over time grew into shipping products with teams of developers and testers working on them. In such cases early development is not about quality, it is about proving out a concept and working on things like scale and performance that must be right before quality could even be an issue. If you can't build a web service that scales, testing is not your biggest problem!
Once it is clear that a product can and will be built and shipped, that's when the development team seeks out test involvement.
You can imagine a process like this: someone has an idea, they think about it, write experimental code, seek out opinions of others, write some more code, get others involved, write even more code, realize they are onto something important, write more code to mold the idea into something that they can present to others to get feedback ... somewhere in all this an actual project is created in Google's project database and the project becomes real. Testers don't get involved until it becomes real.
Do all real projects get testers? Not by default. Smaller projects and those meant for limited users often get tested exclusively by the people who build it. Others that are riskier to our users or the enterprise (much more about risk later) get testing attention.
The onus is on the development teams to solicit help from testers and convince them that their project is exciting and full of potential. Dev Directors explain their project, progress and ship schedule to Test Directors who then discuss how the testing burden is to be shared and agree on things like SWE involvement in testing, expected unit testing levels and how the duties of the release process are going to be shared. SETs may not be involved at project inception, but once the project becomes real we have vast influence over how it is to be executed.
And when I say "testing" I don't just mean exercising code paths. Testers might not be involved from the beginning ... but testing is. In fact, an SET's impact is felt even before a developer manages to check code into the build. Stay tuned to understand what I am talking about.