Google Data

At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.

Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.

We were wrong, but in a good way!

We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it…”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?

During that first week of trying out RPF, Wensi found:

• 77% of the features in Webstore were testable by RPF
• Generating regression test scripts via this early version of RPF was about 8X faster than building them via Selenium/WebDriver
• The RPF scripts caught 6 functional regressions and many more intermittent server failures.
• Common setup routines like login should be saved as modules for reuse (a crude version of this was working soon after)
• RPF worked on Chrome OS, where Selenium by definition could never run as it required client-side binaries. RPF worked because it was a pure cloud solution, running entirely within the browser, communicating with a backend on the web.
• Bugs filed via bite, provided a simple link, which would install BITE on the developers machine and re-execute the repros on their side. No need for manually crafted repro steps. This was cool.
• Wensi wished RPF was cross browser. It only worked in Chrome, but people did occasionally visit the site with a non-Chrome browser.

So, we knew we were onto something interesting and continued development. In the near term though, chrome web store testing went back to using Selenium because that final 23% of features required some local Java code to handle file upload and secure checkout scenarios. In hindsight, a little testability work on the server could have solved this with some AJAX calls from the client.

We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.

Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven’t done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.

RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.

Please join us in maintaining BITE/RPF, and be nice to Po Hu and Joel Hynoski who are driving this work forward within Google.

By Jim Reardon

The test plan is dead!

Well, hopefully.  At a STAR West session this past week, James Whittaker asked a group of test professionals about test plans.  His first question: “How many people here write test plans?”  About 80 hands shot up instantly, a vast majority of the room.  “How many of you get value or refer to them again after a week?”  Exactly three people raised their hands.

That’s a lot of time being spent writing documents that are often long-winded, full of paragraphs of details on a project everyone already knows to get abandoned so quickly.

A group of us at Google set about creating a methodology that can replace a test plan — it needed to be comprehensive, quick, actionable, and have sustained value to a project.  In the past few weeks, James has posted a few blogs about this methodology, which we’ve called ACC.  It’s a tool to break down a software product into its constituent parts, and the method by which we created “10 Minute Test Plans” (that only take 30 minutes!)

Comprehensive
The ACC methodology creates a matrix that describes your project completely; several projects that have used it internally at Google have found coverage areas that were missing in their conventional test plans.

Quick
The ACC methodology is fast; we’ve created ACC breakdowns for complex projects in under half an hour.  Far faster than writing a conventional test plan.

Actionable
As part of your ACC breakdown, risk is assessed to the capabilities of your appliciation.  Using these values, you get a heat map of your project, showing the areas with the highest risk — great places to spend some quality time testing.

Sustained Value
We’ve built in some experimental features that bring your ACC test plan to life by importing data signals like bugs and test coverage that quantify the risk across your project.

Today, I’m happy to announce we’re open sourcing Test Analytics, a tool built at Google to make generating an ACC simple — and which brings some experimental ideas we had around the field of risk-based testing that work hand-in-hand with the ACC breakdown.

Test Analytics has two main parts: first and foremost, it’s a step-by-step tool to create an ACC matrix that’s faster and much simpler than the Google Spreadsheets we used before the tool existed.  It also provides visualizations of the matrix and risks associated with your ACC Capabilities that were difficult or impossible to do in a simple spreadsheet.

The second part is taking the ACC plan and making it a living, automatic-updating risk matrix.  Test Analytics does this by importing quality signals from your project: Bugs, Test Cases, Test Results, and Code Changes.  By importing these data, Test Analytics lets you visualize risk that isn’t just estimated or guessed, but based on quantitative values.  If a Component or Capability in your project has had a lot of code change or many bugs are still open or not verified as working, the risk in that area is higher.  Test Results can provide a mitigation to those risks — if you run tests and import passing results, the risk in an area gets lower as you test.

This part’s still experimental; we’re playing around with how we calculate risk based on these signals to best determine risk.  However, we wanted to release this functionality early so we can get feedback from the testing community on how well it works for teams so we can iterate and make the tool even more useful.  It’d also be great to import even more quality signals: code complexity, static code analysis, code coverage, external user feedback and more are all ideas we’ve had that could add an even higher level of dynamic data to your test plan.

You can check out a live hosted version, browse or check out the code along with documentation, and of course if you have any feedback let us know – there’s a Google Group set up for discussion, where we’ll be active in responding to questions and sharing our experiences with Test Analytics so far.

Long live the test plan!

By Aaron Jacobs

Google JS Test is a JavaScript unit testing framework that runs on the V8 JavaScript Engine, the same open source project that is responsible for Google Chrome’s super-fast JS execution speed. Google JS Test is used internally by several Google projects, and we’re pleased to announce that it has been released as an open source project.

Features of Google JS Test include: