Motivation:

Software systems are getting larger, more complex and more configurable all the time. While beneficial in many ways, these changes dramatically increase testing obligations.

For example, today's systems are increasingly built to run on multiple OS, compiler and library platforms. They are composed from multiple components, each with multiple versions. They are configured by manipulating numerous compile- and run-time options. Additionally, distributed applications add an allocation dimension in which runtime topologies can vary widely.

This situation is further complicated by agile and flexible development practices in which systems are evolved incrementally over short, but varying update cycles and in which development teams may be geographically distributed.

Basically, each new configuration dimension increases the number of potential runtime configurations combinatorially. Since each of these configurations might behave differently or host different bugs, each of these configurations, at least in theory, must be tested. Furthermore, this increased amount of testing must be done in shorter and shorter time frames because the systems themselves are changing faster than ever.

The Skoll project:

Our research aims to create better, faster and more powerful techniques for testing these kinds of systems. Our vision is to redesign traditional testing processes so that they can be executed around-the-world, around-the-clock. These processes are logically divided into multiple tasks that are distributed intelligently to client machines around the world and then executed by them. The results from these distributed tasks are returned to central collection sites where they are merged and analyzed to complete the overall QA process.

At the second Google Test Automation Conference, I and my colleague Atif Memon presented an infrastructure and approach called Skoll that was created to support this vision. The video is embedded below.

Skoll in action:

To give you a better sense of how this works we're going to walk you through how we set up a Skoll server to run a simplified version of the continuous build, test and integration process we've developed to run the MySQL Build Farm Initiative. This process runs on computing resources that are temporarily volunteered by members of the MySQL developer and user communities. After reading the rest of this post, we hope you too will be ready and willing to volunteer as well!

MySQL, as you probably know, is a very popular open source database comprising over 2 million lines of code. It is developed by a world wide community and runs on many different platforms, has over 150 configuration options, and allows substantial static and runtime end-user customization. For instance, you can select different database front- and back-ends, and you can run the system with different runtime topologies to support database replication. In short, it's exactly the kind of system we had in mind when we created Skoll.

We first started speaking with MySQL developers after the first Google Test Automation Conference. They were interested in running a continuous build, integration and test (CBIT) process. They had several goals, such as: testing a broader variety of configurations, not just a handful of popular ones; giving developers greater visibility into the quality and stability of the system; improving bug fix turnaround time; and managing test data to enable long term statistical analysis and improved decision making.

Since they couldn't seem to find anything off-the-shelf that was sufficiently targeted to their needs, we worked with them to develop a Skoll-based CBIT process. This process has several parts: defining a configuration model, implementing a sampling strategy, executing the tests, and analyzing and visualizing the results.

We will discuss each of these below. Readers who just want to run a client can jump straight to the Section marked Test execution.

Some more details:

Configuration Model: We are starting out by looking at 23 options. There are some inter-option constraints as well. For example, a configuration can compile in support for either the libedit library (--with-libedit) or the readline library (--with-readline), but not both. Here's a ling to the current configuration model . We will expand this model as we gain more experience with this process and more insight into the key issues concerning MySQL.

Sampling strategy: There are over 48 million unique configurations in this test space. Since testing 1 configuration can take up to 2 hours and because new releases come out more or less daily, exhaustive testing of each check-in is clearly impossible. Therefore, we only test specially-chosen subsets of configurations in which all t-way (2 <= t <= 4) combinations of option settings are tested at least once. Our particular algorithm for selecting these configurations, which we invented with Myra Cohen of the University of Nebraska, works incrementally. First it tests 3 sets of ~23 configurations, each set of which covers all combinations of settings between every pair of options (i.e., 2-way combinations). We then move up to testing 3 sets of ~84 configurations that cover all settings of every triple of options and then move up again to cover all quadruples (~272 configurations), also 3 times each. This allows us to test low level interactions quickly, even if new releases come in before all levels of interactions can be tested. As we gain experience with this process, we will evaluate whether and how much we may need to increase t.

Test execution: To participate in this process, users can go to our MySQL 5.1 project page. On that page you can find links to download a client along with instructions on how to install and run it. This client is a simple Perl script that connects to a Skoll server and asks for a test job. The server examines its internal databases, selects an outstanding job and returns it to the client who then executes it. Currently, testing a configurations involves compiling MySQL in a specific configuration and then running about 750 MySQL-supplied tests on that configuration. The test scripts determine which test cases can be run in the current configuration and runs them. After completing the tests, the client uploads the results to the server.

Feedback, analysis and visualization: For this process, we are interested in understanding where configuration-related bugs might be hiding. To figure this out, we periodically analyze test results by building a classification tree for each test that has failed in a minimum number of configurations. These trees model test failures in terms of the configuration option and settings that were set when the tests failed. Users can look at a web page to see the current test results . This page shows summary statistics for each build ID. Clicking on a build ID takes you to a detail page listing each test that failed a minimum number of times. Each test presents classification information, both in a raw form and as an interactive treemap.

Current status:

We have just started running this process on a continuous basis using an a small number of MySQL developer community machines. We hope to bring many more test machines online in the coming weeks. Please check out the results web page to watch our progress. And don't let everyone else have all the fun -- download your client today! Thanks!

All the presentations from the conference will be posted on YouTube Google Channel within a New York minute after the speaker leaves the podium (well, maybe 72 New York minutes). We look forward to seeing you at the conference.

At last year's conference, we found that the informal hallway discussions were tremendously valuable to everyone who came, and we would like to continue that custom. Therefore, the main point in the application process is to tell us how you can contribute to the conversation and what you would like to learn from the conference.

To get you started, here is what our speakers will be talking about:

• Apple Chow and Santiago Etchebehere -- Building a flexible and extensible automation framework around Selenium
• Ali-Akber Saifee -- muvee Framework for Autonomous Testing
• Olivier Warin -- Spirent's WorkSuite Manager
• Douglas Sellers -- CustomInk Domain Specific Language for automating an AJAX based application
• Julian Harty -- Mobile Wireless Test Automation
• Matt Heusser and Sean McMillan -- Examining Interaction-Based testing
• Risto Kumpulainen -- Automated testing for F-Secure's Linux/UNIX Anti-Virus products
• Sergio Pinon -- User Interface Functional Testing with AFTER (Automated Functional Testing Engine in Ruby)
• Vivek Prahlad -- Functional Testing Swing applications with Frankenstein
• Simon Stewart -- Web Driver
• Adam Porter and Atif Memon -- Skoll distributed continuous quality assurance system
• Cedric Beust -- TestNG
• Hadar Ziv -- Specification-based Testing

I had played with UCBLogo for two weeks and hadn’t made it crash once. Brian brought the whole thing down in three commands. The most telling part is that when I tried to reproduce the defect a week later I couldn’t. I issued rt with a ton of 9s and just couldn’t get it to break. As it turns, it only crashes when you omit the space, which of course I didn’t think of doing. It took me more time to reproduce the defect than it took Brian to discover it.

As suites of tests grow, it's common to see duplication emerge. Like any code, tests should ideally be kept in a state that's easy to understand and maintain. So, you'll want to refactor your tests, too.

However, refactoring tests can be hard because you don't have tests for the tests.

How do you know that your refactoring of the tests was safe and you didn't accidentally remove one of the assertions?

If you intentionally break the code under test, the failing test can show you that your assertions are still working. For example, if you were refactoring methods in CombineHarvesterTest, you would alter CombineHarvester, making it return the wrong results.

Check that the reason the tests are failing is because the assertions are failing as you'd expect them to. You can then (carefully) refactor the failing tests. If at any step they start passing, it immediately lets you know that the test is broken – undo! When you're done, remember to fix the code under test and make sure the tests pass again.
(revert is your friend, but don't revert the tests!)

Let's repeat that important point:

When you're done...remember to fix the code under test!

Summary

• Refactor production code with the tests passing. This helps you determine that the production code still does what it is meant to.
• Refactor test code with the tests failing. This helps you determine that the test code still does what it is meant to.

Remember to download this episode of Testing on the Toilet and post it in your office.

Zurich, Switzerland is the location of one of Google's largest engineering offices in Europe. We like to say it is no longer necessary to live in Silicon Valley to develop great software for Google, and the Zurich office has over 100 software engineers working on Google products and infrastructure. As a result the Zurich Test Engineering team is kept very busy.

Producing great software is the driving motivation for each member of the Test Engineering team, which is relatively small by industry standards. But we're growing quickly and are passionate about software development, software quality, and testing.

We work on several projects: some are customer-facing and some are infrastructure. We currently work on Google Maps, which has testers and developers in several of our offices around the world. Our team builds tools to check the consistency of the data feeds that support the local search features in Maps. Another project mainly developed in Zurich is Google Transit, which provides public transportation itineraries and schedule information for commuters who take trains, buses, and trams. On this project, we build tools to verify the proper alignment of the transportation layer of the map with the actual location coordinates of transit stops. We also focus on many projects related to Google’s infrastructure. For example, with Google Base API, we work with the Software Engineering team to measure response time and to track bottlenecks during large-scale bulk updates.

Our aim is to assign local test engineers to most projects developed in this office, so hiring for this team is always a priority. Candidates are from all over the world, and many different nationalities are represented in our office. Adapting to Zurich is quite easy, because it is already an international place: many companies have their headquarters here, and Zurich has been named the best city in the world for its quality of life in 2005, 2006, and 2007.

• Global - Google has engineering offices (and, of course, customers) all over the world. So not only do we have engineers from everywhere, every engineer gets the chance to make software that will be as great in Singapore as it is in Finland.
• Comfortable - Engineers need to be comfortable to be effective...and Google gives us the equipment to be comfortable. And if our muscles get tight from sitting, we can get an on-site professional massage.
• Flexible - Getting work done is more important than what time we work.
• Unhampered - We're a company designed by engineers for engineers, so functional merit tends to outweigh other considerations at decision-making time.
• Entertained - We have lots of fun activities. Every winter, there's a company-wide ski trip. Every summer, there's an all-engineer picnic on the Santa Cruz beach boardwalk. Google has bought out theaters for opening days of films like Star Wars and Lord of the Rings.
• Listened to - Google has a very open environment and truly values each employee's opinions. Every Friday, there's a company-wide meeting where Larry and Sergey (our founders) report on significant events and have an open-to-all Q&A session.
• Stuffed - In most offices, Google provides free gourmet-quality breakfast, lunch, and dinner. If you come in too late to get breakfast (or you get hungry between meals), you can always head to the nearby mini-kitchen and grab some fruit, cereal, and tea (or crisps, chocolate, and espresso for the less healthily inclined).
• Cutting-edge - We have the audacious goal of organizing the world's information -- and to meet it, we've built the largest distributed computer system in the world and written some of the world's most widely-used software.
• High-impact - People in every country and every language use our products.
• Dependable - We try to keep a high bar for hiring, so you can depend on the ability of your teammates to handle most any engineering problem.

By substituting custom objects for some of your module's dependencies, you can thoroughly test your code, increase your coverage, and still run in less than a second. You can even simulate rare scenarios like database failures and test your error handling code.

A variety of different terms are used to refer to these “custom objects”. In an effort to clarify the vocabulary, Gerard Meszaros provides the following definitions:

• Test Double is a generic term for any test object that replaces a production object.
• Dummy objects are passed around but not actually used. They are usually fillers for parameter lists.
• Fakes have working implementations, but take some shortcut (e.g., InMemoryDatabase).
• Stubs provide canned answers to calls made during a test.
• Mocks have expectations which form a specification of the calls they do and do not receive.

For example, to test a simple method like getIdPrefix() in the IdGetter class:

public class IdGetter {  // Constructor omitted.
  public String getIdPrefix() {
    try {
      String s = db.selectString("select id from foo");
      return s.substring(0, 5);
    } catch (SQLException e) { return ""; }
  }
}

You could write:

  db.execute("create table foo (id varchar(40))");  // db created in setUp().
  db.execute("insert into foo (id) values ('hello world!')");
  IdGetter getter = new IdGetter(db);
  assertEquals("hello", getter.getIdPrefix());

The test above works but takes a relatively long time to run (network access), can be unreliable (db machine might be down), and makes it hard to test for errors. You can avoid these pitfalls by using stubs:

  public class StubDbThatReturnsId extends Database {
    public String selectString(String query) { return "hello world"; }
  }
  public class StubDbThatFails extends Database {
    public String selectString(String query) throws SQLException {
      throw new SQLException("Fake DB failure");
    }
  }
  public void testReturnsFirstFiveCharsOfId() throws Exception {
    IdGetter getter = new IdGetter(new StubDbThatReturnsId());
    assertEquals("hello", getter.getIdPrefix());
  }
  public void testReturnsEmptyStringIfIdNotFound() throws Exception {
    IdGetter getter = new IdGetter(new StubDbThatFails());
    assertEquals("", getter.getIdPrefix());
  }

Remember to download this episode of Testing on the Toilet and post it in your office.
 

function showProgress(status) {
  status.message = "Loading";
  for (var time = 1000; time <= 3000; time += 1000) {
    // Append a '.' to the message every second for 3 secs.
    setTimeout(function() {
      status.message += ".";
    }, time);
  }
  setTimeout(function() {
    // Special case for the 4th second.
    status.message = "Done";
  }, 4000);
}

function testUpdatesStatusMessageOverFourSeconds() {
  Clock.reset(); // Clear any existing timeout functions on the event queue.
  var  status = {};
  showProgress(status); // Call our function.
  assertEquals("Loading", status.message);
  Clock.tick(2000); // Call any functions on the event queue that have
                    // been scheduled for the first two seconds.
  assertEquals("Loading..",  status.message);
  Clock.tick(2000); // Same thing again, for the next two seconds.
  assertEquals("Done", status.message);
}

Software testing is tough. It can be exhausting, and there is rarely enough time to find all the important bugs. Wouldn't it be nice to have a staff of tireless servants working day and night to make you look good? Well, those days are here. On Thursday, March 22, I'll give a lunchtime presentation titled "How to Build Your Own Robot Army" for the Quality Assurance SIG of the Software Association of Oregon.

Two decades ago, machine time was expensive, so test suites had to run as quickly and efficiently as possible. Today, CPUs are cheap, so it becomes reasonable to move test creation to the shoulders of a test machine army. But we're not talking about the run-of-the-mill automated scripts that only do what you explicitly told them. We're talking about programs that create and execute tests you never thought of to find bugs you never dreamed of. From Orcs to Zergs to Droids to Cyborgs, this presentation will show how to create a robot test army using tools lying around on the Web. Most importantly, it will cover how to take appropriate credit for your army's work!

• Al Snow - Form Letter Generator Technique

• Chris McMahon – Emulating User Actions in Random and Deterministic Modes

• Dave Liebreich – Test Mozilla

• David Martinez – Tk-Acceptance

• Dave W. Smith – System Effects of Slow Tests

• Harry Robinson – Exploratory Automation

• Jason Reid – Not Trusting Your Developers

• Jeff Brown – MBUnit

• Jeff Fry – Generating Methods on the Fly

• Keith Ray – ckr_spec

• Kurman Karabukaev – Whitebox testing using Watir

• Mark Striebeck – How to Get Developers and Tester to Work Closer Together

• Sergio Pinon – UI testing + Cruise Control

is a spectrum that has "Tester" at one end and "Developer" at the other. Where would you put yourself, and why?

The idea for Testapalooza came out of discussions about how to build a vibrant testing community here at Google. Many diverse groups work daily on quality-related activities, but each group uses different tools and has different ideas for testing an application, so it can be difficult to find out what others are doing. So we decided to put on a conference!

We asked engineers from Testing, Development, User Experience, and other groups to submit conference sessions: tool presentations, tutorials, workshops, panels, and experience reports. We hoped to get 30-40 submissions from which we could select about 20. In typical conference mode, the day before the submission deadline, we had 12 submissions. The day after the deadline, we had more than 130! It was very impressive and fun to read what our engineers submitted. We had some of our most involved engineers on the reviewing committee, and even they were surprised about the breadth and depth of the proposed sessions. It was extremely hard to pick just 41 of these proposals, but we couldn't fit any more into a one-day conference.

We ran 11 tracks: Agile, Automation, Developer, Internationalization, Perf, QA, Release Engineering, Security, Reliability, SysOps, and User Experience. Registrations for the event filled up quickly and proved that there is indeed a great desire for more cross-specialty collaboration: software developers signed up to attend QA sessions, operations engineers learned more about unit testing, and QA engineers were everywhere.

The conference was a great success. We had sessions going the whole day, and people were discussing testing in the hallways. New ideas were generated and debated in every corner of the Googleplex. People appreciated the variety of topics from agile testing to micro-level unit testing to testing tools to usability testing. We also had a poster session, where internal groups could show other Googlers what they were doing, our equivalent of the conference expo.

Of course, this wouldn't be a true Google event without some great food, and we were fortunate to have enthusiastic participation from our chefs: Taste-a-Palooza!

We finished the day with an hour of lightning talks. At the end of the day everybody was exhausted, but with new and interesting ideas to think about.

All Testapalooza sessions were video recorded (many were videoconferenced to other offices). We want to publish as many of these videos as possible, and will review them over the coming weeks to publish sessions which did not contain any confidential information. Watch this space for more information on the videos.

In software as in life there are things we notice that help confirm whether something is satisfactory or unsatisfactory. Let's call these things that affect our judgment of the results "factors." Some factors provide stronger indications than others. When using these factors to rate results, we will assign higher scores (or "weightings") to the stronger indicators. By assigning higher weightings to the stronger indicators, we enable these to have a stronger influence on the overall outcome.

Several readers have commented that our current blog slogan, "Life is too short for manual testing," implies that we don't value manual and exploratory testing. In fact, we are big fans of exploratory testing, and we intended the message to be liberating, not insulting.

Manual testing can find bugs quickly and with little overhead in the short run. But it can be expensive and exhausting in a long project. And manual testing is gated by how fast and long humans can work. Running millions of test sequences and combinations by hand would take longer than most people's lifetimes - life is literally too short for manual testing to reach all the bugs worth reaching.

We originally featured the "Life is too short ..." slogan on T-shirts at the Google London Test Automation Conference. One theme of that conference was that it makes sense to get machines to do some of the heavy lifting in software testing, leaving human testers free to do the kinds of testing that people do well.

If you'd like to find out more about computer-assisted testing, check out the LTAC videos as well as Cem Kaner's excellent STAR 2004 presentation on High Volume Test Automation . And if you can wait a bit, I will be doing a talk on "The Bionic Exploratory Tester" at CAST 2007 in Bellevue, Washington, in July.

I asked Jon Bach 's opinion on the slogan, and he suggested that what we are really trying to say is:

Life's too short to only use an approach for testing that relies solely on a human's ability to execute a series of mouse clicks and keystrokes when the processing power that makes computers so useful can be leveraged to execute these tests, freeing testers from especially mundane or repetitive testing so that their brains can be used for higher order tests that computers can't do yet.

I agree, but it would've been a heck of a T-shirt. :-)

Future slogans:

Testing is about being willing to try different approaches and entertain different perspectives, so a single slogan can't do it justice. We are planning to feature different slogans on a regular basis, and already have a few of our favorites lined up. If you've got a slogan to share, we'd love to hear it. Post it in the comments below or email us.

On Saturday, February 24, the Mountain View Googleplex will offer hospitality, support and free munchies for the first-ever "Developer-Tester/Tester-Developer Summit" -- a peer-driven gathering of developer-testers and tester-developers from around the industry to share knowledge and code.

The Summit is being organized by Elisabeth Hendrickson of Quality Tree Software and Chris McMahon of Lyris Technologies, who describe the day this way:

Our emphasis will be on good coding practices for testing, and good testing practices for automation. That might include topics like: test code and patterns; refactoring test code; creating abstract layers; programmatically analyzing/verifying large amounts of data; achieving repeatability with random tests; OO model-based tests; creating domain specific languages; writing test fixtures or harnesses; and/or automatically generating large amounts of test data.