Early successes

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa 
READ of size 2 at 0x615000000ffa thread T0
SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)
   #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31

Contributions and feedback are welcome

• Tight coupling and insufficient abstraction made unit testing very hard, and as a consequence, a lot of end-to-end tests served as functional tests of that code.
• The infrastructure used for the end-to-end tests had no good way to create and inject fakes or mocks for these services. As a result, the tests had to run the large number of servers for all these external dependencies. This led to very large and brittle tests that our existing test execution infrastructure was not able to handle reliably.

• The new tests are as effective in finding bugs as the end-to-end tests are.
• The new tests run in about 3 minutes instead of 30 minutes for the end-to-end tests.
• The client side tests are 0% flaky. The verification tests are usually less flaky than the end-to-end tests, and never more.

Build Systems Speed

• Build and test times longer than a handful of seconds cause many developers’ minds to wander, taking them out of the zone.
• Excessive build or release times* makes tinkering and refactoring much harder. All developers have a threshold when they start hedging (e.g. “I’d like to remove this branch, but I don’t know if I’ll break the iOS build”) which causes refactoring to not happen.

Figure 2: How a Chromium patch is tested.

Figure 3. Test value.

Figure 5: A VP8-in-WebRTC regression (bug) and its fix displayed in a Catapult dashboard.

Running on Localhost

b_serv --port=1234 --db=/tmp/fakedb
c_serv --port=1235
a_serv --b_spec=localhost:1234 --c_spec=localhost:1235

Debugging Mobile Apps

When Debugging gets Tricky

chrome --renderer-cmd-prefix='xterm -title renderer -e gdb --args'

Proper Logging

Monitoring and Statistics

System Under Test (SUT) Size

Sources

• For each important use case, there should be one corresponding end-to-end test. This should include one test for each important class of error. The goal is the keep your total end-to-end count low.
• Be prepared to allocate at least one week a quarter per test to keep your end-to-end tests stable in the face of issues like slow and flaky dependencies or minor UI changes.
• Focus your efforts on verifying overall system behavior instead of specific implementation details; for example, when testing login behavior, verify that the process succeeds independent of the exact messages or visual layouts, which may change frequently.
• Make your end-to-end test easy to debug by providing an overview-level log file, documenting common test failure modes, and preserving all relevant system state information (e.g.: screenshots, database snapshots, etc.).

• System components that are owned by other teams may change unexpectedly, and break your tests. This increases overall maintenance cost, but can highlight incompatible changes
• It may be more difficult to make an end-to-end test fully hermetic; leftover test data may alter future tests and/or production systems. Where possible keep your test data ephemeral.
• An end-to-end test often necessitates multiple test doubles (fakes or stubs) for underlying dependencies; they can, however, have a high maintenance burden as they drift from the real implementations over time.

• Automate a manual verification process for product release candidates so developers have more time to respond to potential release-blocking issues.
• Design and implement an automated way to track and surface Android battery usage to developers, so that they know immediately when a new feature will cause users drained batteries.
• Quantify if a regenerated data set used by a product, which contains a billion entities, is better quality than the data set currently live in production.
• Write an automated test suite that validates if content presented to a user is of an acceptable quality level based on their interests.
• Read an engineering design proposal for a new feature and provide suggestions about how and where to build in testability.
• Investigate correlated stack traces submitted by users through our feedback tracking system, and search the code base to find the correct owner for escalation.
• Collaborate on determining the root cause of a production outage, then pinpoint tests that need to be added to prevent similar outages in the future.
• Organize a task force to advise teams across the company about best practices when testing for accessibility.

Introduction

In Practice

• Developing is easy
• Fast build
• Good, fast tests
• Clean code
• Easy running + debugging
• One-click rollbacks

• Broken HEAD (tip-of-tree)
• Slow presubmit (i.e. checks running before submit)
• Builds take hours
• Incremental build/link > 30s
• Flakytests
• Can’t attach debugger
• Logs full of uninteresting information

The Three Pillars of Hackability

Pillar 1: Code Health

Tests

Readable Code and Code Review

Presubmit Testing and Lint

$ git cl upload
Error: the media/audio directory requires formatting. Please run 
    git cl format media/audio.

Single Branch And Reducing Risk

Loose Coupling and Testability

Aggressively Reduce Technical Debt

• A tool that monitors the flakiness of tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the critical path and files a bug for developers to reduce the flakiness. This prevents it from becoming a problem for developers, but could easily mask a real race condition or some other bug in the code being tested.
• Another tool detects changes in the flakiness level of tests and works to identify the change that caused the test to change the level of flakiness.

• We have a new team dedicated to providing accurate and timely information about test flakiness to help developers and build monitors so that they know whether they are being harmed by test flakiness.
• As we analyze the data from flaky test executions, we are seeing promising correlations with features that should enable us to identify a flaky result accurately without re-running the test.

• Be 18 years of age or older.
• Be from a traditionally underrepresented group in technology.
• Work or study in Computer Science, Computer Engineering, Information Technology, or a technical field related to software testing.
• Be able to attend core dates of GTAC, November 15th - 16th 2016 in Sunnyvale, CA.

• Test Engineers (TEs) --  tested new products and systems integration
• Release Engineers (REs) --  pushed bits into production
• Site Reliability Engineers (SREs) --  managed systems and data centers 24x7.

• Test Engineers (TEs) -- With their deep product knowledge and test/quality domain expertise, TEs focused on what should be tested.
• Software Engineers in Test (SETs) -- Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.

• Automated tests became more efficient and deterministic (e.g. by improving runtimes, eliminating sources of flakiness, etc.) 
• Metrics driven engineering proliferated (e.g. improving code and feature coverage led to higher quality products).

• Extending IDEs to make writing and reviewing code easier, shortening the “write code” cycle
• Automating release verification, shortening the “release code” cycle.
• Automating real time production system log verification and anomaly detection, helping automate production monitoring.
• Automating measurement of developer productivity, helping understand what’s working and what isn’t.

• Powerful built-in synchronization : Tests will automatically wait for events such as animations, network requests, etc. before interacting with the UI. This will result in tests that are easier to write (no sleeps or waits) and simple to maintain (straight up procedural description of test steps). 
• Visibility checking : All interactions occur on elements that users can see. For example, attempting to tap a button that is behind an image will lead to test failure immediately. 
• Flexible design : The components that determine element selection, interaction, assertion and synchronization have been designed to be extensible. 

• Our 82 survey respondents were mostly (81%) test focused professionals with a wide range of 1 to 40 years of experience. 
• Another 76% of respondents rated the conference as a whole as above average, with marked satisfaction for the venue, the food (those Diwali treats!), and the breadth and coverage of the talks themselves.

• The Uber Challenge of Cross-Application/Cross-Device Testing (Apple Chow and Bian Jiang) 
• Your Tests Aren't Flaky (Alister Scott) 
• Statistical Data Sampling (Celal Ziftci and Ben Greenberg) 
• Coverage is Not Strongly Correlated with Test Suite Effectiveness (Laura Inozemtseva) 
• Chrome OS Test Automation Lab (Simran Basi and Chris Sosa).

What is Automatic Gain Control? 

Testing the AGC

• Add file support to Chrome’s fake audio input device, so we can play a known file. The original audio test avoided this by using WebAudio, but AGC doesn’t run in the WebAudio path, just the microphone capture path, so that won’t work.
• Instead of running PESQ, run an analysis that compares the gain between input and output.

Adding Fake File Support

int FileSource::OnMoreData(AudioBus* audio_bus, uint32 total_bytes_delay) {
  // Load the file if we haven't already. This load needs to happen on the
  // audio thread, otherwise we'll run on the UI thread on Mac for instance.
  // This will massively delay the first OnMoreData, but we'll catch up.
  if (!wav_audio_handler_)
    LoadWavFile(path_to_wav_file_);
  if (load_failed_)
    return 0;

  DCHECK(wav_audio_handler_.get());

  // Stop playing if we've played out the whole file.
  if (wav_audio_handler_->AtEnd(wav_file_read_pos_))
    return 0;

  // This pulls data from ProvideInput.
  file_audio_converter_->Convert(audio_bus);
  return audio_bus->frames();
}

chrome --use-fake-device-for-media-stream 
       --use-file-for-fake-audio-capture=/tmp/file.wav

The Analysis Stage

  // ...
  size_t bytes_written;
  wav_audio_handler->CopyTo(audio_bus.get(), 0, &bytes_written);
  CHECK_EQ(bytes_written, wav_audio_handler->data().size())
      << "Expected to write entire file into bus.";

  // Set the filter coefficient to the whole file's duration; this will make
  // the power monitor take the entire file into account.
  media::AudioPowerMonitor power_monitor(wav_audio_handler->sample_rate(),
                                         file_duration);
  power_monitor.Scan(*audio_bus, audio_bus->frames());
  // ...
  return power_monitor.ReadCurrentPowerAndClip().first;

What Went Wrong?

Clock Drift and Packet Loss

Silence Splitting to the Rescue!

Result

  base::FilePath reference_file = 
      test::GetReferenceFilesDir().Append(reference_filename);
  base::FilePath recording = CreateTemporaryWaveFile();

  ASSERT_NO_FATAL_FAILURE(SetupAndRecordAudioCall(
      reference_file, recording, constraints,
      base::TimeDelta::FromSeconds(30)));

  base::ScopedTempDir split_ref_files;
  ASSERT_TRUE(split_ref_files.CreateUniqueTempDir());
  ASSERT_NO_FATAL_FAILURE(
      SplitFileOnSilenceIntoDir(reference_file, split_ref_files.path()));
  std::vector<base::FilePath> ref_segments =
      ListWavFilesInDir(split_ref_files.path());

  base::ScopedTempDir split_actual_files;
  ASSERT_TRUE(split_actual_files.CreateUniqueTempDir());
  ASSERT_NO_FATAL_FAILURE(
      SplitFileOnSilenceIntoDir(recording, split_actual_files.path()));

  // Keep the recording and split files if the analysis fails.
  base::FilePath actual_files_dir = split_actual_files.Take();
  std::vector<base::FilePath> actual_segments =
      ListWavFilesInDir(actual_files_dir);

  AnalyzeSegmentsAndPrintResult(
      ref_segments, actual_segments, reference_file, perf_modifier);

  DeleteFileUnlessTestFailed(recording, false);
  DeleteFileUnlessTestFailed(actual_files_dir, true);

void AnalyzeSegmentsAndPrintResult(
    const std::vector<base::FilePath>& ref_segments,
    const std::vector<base::FilePath>& actual_segments,
    const base::FilePath& reference_file,
    const std::string& perf_modifier) {
  ASSERT_GT(ref_segments.size(), 0u)
      << "Failed to split reference file on silence; sox is likely broken.";
  ASSERT_EQ(ref_segments.size(), actual_segments.size())
      << "The recording did not result in the same number of audio segments "
      << "after on splitting on silence; WebRTC must have deformed the audio "
      << "too much.";

  for (size_t i = 0; i < ref_segments.size(); i++) {
    float difference_in_decibel = AnalyzeOneSegment(ref_segments[i],
                                                    actual_segments[i],
                                                    i);
    std::string trace_name = MakeTraceName(reference_file, i);
    perf_test::PrintResult("agc_energy_diff", perf_modifier, trace_name,
                           difference_in_decibel, "dB", false);
  }
}

Updating Dependencies is Hard

• Update the library to its latest release. If it’s been a long time since you did this, it can be quite risky and you may have to spend significant testing resources to ensure all the accumulated changes don’t break your application. You may have to catch up to interface changes in the library as well. 
• Cherry-pick the feature/bug fix you want into your copy of the library. This is even riskier because your cherry-picked patches may depend on other changes in the library in subtle ways. Also, you still are not up to date with the latest version. 
• Find some way to make do without the feature or bug fix.

How “Rolling” Works 

{
  # ... 
  'src/third_party/webrtc':
      'https://chromium.googlesource.com/' +
      'external/webrtc/trunk/webrtc.git' + 
      '@' + '5727038f572c517204e1642b8bc69b25381c4e9f',
}

Bots That Can See the Future: “FYI Bots” 

• If all the FYI bots were green, we knew a roll most likely would go smoothly. 
• If the bots didn’t compile, we knew we would have to adapt Chrome to an interface change in the next roll patch. 
• If the bots were red, we knew we either had a bug in WebRTC or that Chrome would have to be adapted to some semantic change in WebRTC.

Making Gradual Interface Changes 

class WebRtcAmplifier {
  ...
  int SetOutputVolume(float volume);
}

class WebRtcAmplifier {
  ...
  int SetOutputVolume(float volume, bool allow_eleven1);
}

class WebRtcAmplifier {
  ...
  int SetOutputVolume(float volume);
  int SetOutputVolume2(float volume, bool allow_eleven);
}

1. Roll into Chrome 
2. Make Chrome use SetOutputVolume2 
3. Update SetOutputVolume’s signature 
4. Roll again and make Chrome use SetOutputVolume 
5. Delete SetOutputVolume2

Results

Future Work: Optimistic Auto-rolling

References

Footnotes

1. We actually did have a hilarious bug in WebRTC where it was possible to set the volume to 1.1, but only 0.0-1.0 was supposed to be allowed. No, really. Thus, our WebRTC implementation must be louder than the others since everybody knows 1.1 must be louder than 1.0.

End-to-End Tests in Theory 

• Developers like it because it offloads most, if not all, of the testing to others. 
• Managers and decision-makers like it because tests that simulate real user scenarios can help them easily determine how a failing test would impact the user. 
• Testers like it because they often worry about missing a bug or writing a test that does not verify real-world behavior; writing tests from the user's perspective often avoids both problems and gives the tester a greater sense of accomplishment. 

End-to-End Tests in Practice 

1. The latest version of the service is built. 
2. This version is then deployed to the team's testing environment. 
3. All end-to-end tests then run against this testing environment. 
4. An email report summarizing the test results is sent to the team.

Analysis 

What Went Well 

• Customer-impacting bugs were identified and fixed before they reached the customer.

What Went Wrong 

• The team completed their coding milestone a week late (and worked a lot of overtime). 
• Finding the root cause for a failing end-to-end test is painful and can take a long time. 
• Partner failures and lab failures ruined the test results on multiple days. 
• Many smaller bugs were hidden behind bigger bugs. 
• End-to-end tests were flaky at times. 
• Developers had to wait until the following day to know if a fix worked or not. 

The True Value of Tests 

Building the Right Feedback Loop

• It's fast. No developer wants to wait hours or days to find out if their change works. Sometimes the change does not work - nobody is perfect - and the feedback loop needs to run multiple times. A faster feedback loop leads to faster fixes. If the loop is fast enough, developers may even run tests before checking in a change. 
• It's reliable. No developer wants to spend hours debugging a test, only to find out it was a flaky test. Flaky tests reduce the developer's trust in the test, and as a result flaky tests are often ignored, even when they find real product issues. 
• It isolates failures. To fix a bug, developers need to find the specific lines of code causing the bug. When a product contains millions of lines of codes, and the bug could be anywhere, it's like trying to find a needle in a haystack. 

Think Smaller, Not Larger

Unit Tests

• Unit tests are fast. We only need to build a small unit to test it, and the tests also tend to be rather small. In fact, one tenth of a second is considered slow for unit tests. 
• Unit tests are reliable. Simple systems and small units in general tend to suffer much less from flakiness. Furthermore, best practices for unit testing - in particular practices related to hermetic tests - will remove flakiness entirely. 
• Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you only need to search that small unit under test to find the bug. 

Unit Tests vs. End-to-End Tests

	Unit	End-toEnd
Fast
Reliable
Isolates Failures
Simulates a Real User

Integration Tests

Testing Pyramid

• Inverted pyramid/ice cream cone. The team relies primarily on end-to-end tests, using few integration tests and even fewer unit tests. 
• Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle. 

• Very large and slow tests. 
• High flakiness rate due to timeouts and memory issues. 
• Hard to debug/investigate failures. 
• Authentication issues (ex: authentication from automated tests is very tricky).

1. Don’t write E2E tests instead of UI tests. Instead write unit tests and integration tests beside the UI tests. 
2. Hermetic tests are the way to go. 
3. Use dependency injection while designing your app. 
4. Build your application into small libraries/modules, and test each one in isolation. You can then have a few integration tests to verify integration between components is correct . 
5. Componentized UI tests have proven to be much faster than E2E and 99%+ stable. Fast and stable tests have proven to drastically improve developer productivity.

// Production code:
def abs(i: Int)
  return (i < 0) ? i * -1 : i

// Test code:
for (line: String in File(prod_source).read_lines())
  switch (line.number)
    1: assert line.content equals def abs(i: Int)
    2: assert line.content equals   return (i < 0) ? i * -1 : i

// Production code:
def process(w: Work)
  firstPart.process(w)
  secondPart.process(w)

// Test code:
part1 = mock(FirstPart)
part2 = mock(SecondPart)
w = Work()
Processor(part1, part2).process(w)
verify_in_order
  was_called part1.process(w)
  was_called part2.process(w)

class UserInfoValidator {
  public void validate(UserInfo info) {
    if (info.getDateOfBirth().isInFuture()) { throw new ValidationException()); }
  }
}

public class UserInfoService {
  private UserInfoValidator validator;
  public void save(UserInfo info) {
    validator.validate(info); // Throw an exception if the value is invalid.
    writeToDatabase(info);   
  }
}

assertEquals("March", monthMap.get(3));          // JUnit
assertThat(monthMap).containsEntry(3, "March");  // Truth

ImmutableSet<String> colors = ImmutableSet.of("red", "green", "blue", "yellow");
assertTrue(colors.contains("orange"));  // JUnit
assertThat(colors).contains("orange");  // Truth

• You don’t need to add waits or sleeps to your test. Protractor can communicate with your AngularJS app automatically and execute the next step in your test the moment the webpage finishes pending tasks, so you don’t have to worry about waiting for your test and webpage to sync. 
• It supports Angular-specific locator strategies (e.g., binding, model, repeater) as well as native WebDriver locator strategies (e.g., ID, CSS selector, XPath). This allows you to test Angular-specific elements without any setup effort on your part. 
• It is easy to set up page objects. Protractor does not execute WebDriver commands until an action is needed (e.g., get, sendKeys, click). This way you can set up page objects so tests can manipulate page elements without touching the HTML. 
• It uses Jasmine, the framework you use to write AngularJS unit tests, and Javascript, the same language you use to write AngularJS apps.

npm install -g protractor

webdriver-manager update & webdriver-manager start

// It is a good idea to use page objects to modularize your testing logic
var angularHomepage = {
  nameInput : element(by.model('yourName')),
  greeting : element(by.binding('yourName')),
  get : function() {
    browser.get('index.html');
  },
  setName : function(name) {
    this.nameInput.sendKeys(name);
  }
};

// Here we are using the Jasmine test framework 
// See http://jasmine.github.io/2.0/introduction.html for more details
describe('angularjs homepage', function() {
  it('should greet the named user', function(){
    angularHomepage.get();
    angularHomepage.setName('Julie');
    expect(angularHomepage.greeting.getText()).
        toEqual('Hello Julie!');
  });
});

exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  
  specs: ['testFolder/*'],

  multiCapabilities: [{
    'browserName': 'chrome',
    // browser-specific tests
    specs: 'chromeTests/*' 
  }, {
    'browserName': 'firefox',
    // run tests in parallel
    shardTestFiles: true 
  }],

  baseUrl: 'http://www.angularjs.org',
};

protractor conf.js

1 test, 1 assertions, 0 failures

@Test public void isUserLockedOut_invalidLogin() {
  authenticator.authenticate(username, invalidPassword);
  assertFalse(authenticator.isUserLockedOut(username));

  authenticator.authenticate(username, invalidPassword);
  assertFalse(authenticator.isUserLockedOut(username));

  authenticator.authenticate(username, invalidPassword);
  assertTrue(authenticator.isUserLockedOut(username));
}

Verifying the Call

bool WaitForCallToComeUp(content::WebContents* tab_contents) {
  // Apprtc will set remoteVideo.style.opacity to 1 when the call comes up.
  std::string javascript =
      "window.domAutomationController.send(remoteVideo.style.opacity)";
  return test::PollingWaitUntil(javascript, "1", tab_contents);
}

Verifying Video is Playing

bool DetectRemoteVideoPlaying(content::WebContents* tab_contents) {
  if (!EvalInJavascriptFile(tab_contents, GetSourceDir().Append(
      FILE_PATH_LITERAL(
          "chrome/test/data/webrtc/test_functions.js"))))
    return false;
  if (!EvalInJavascriptFile(tab_contents, GetSourceDir().Append(
      FILE_PATH_LITERAL(
          "chrome/test/data/webrtc/video_detector.js"))))
    return false;

  // The remote video tag is called remoteVideo in the AppRTC code.
  StartDetectingVideo(tab_contents, "remoteVideo");
  WaitForVideoToPlay(tab_contents);
  return true;
}

What to Send in the Call

if (!HasWebcamOnSystem())
  return;

Downloading all Code and Components 

• The AppEngine SDK in order to bring up the local AppRTC instance, 
• The AppRTC code itself, 
• Chrome (already present in the checkout), and 
• Firefox nightly.

Putting it Together

• Also run on Windows/Mac. 
• Also test Opera. 
• Interop between Chrome/Firefox mobile and desktop browsers. 
• Also ensure audio is playing. 
• Measure bandwidth stats, video quality, etc.

Calling in WebRTC

Test Design

Bringing up AppRTC on localhost

google_appengine/dev_appserver.py apprtc_code/

bool LaunchApprtcInstanceOnLocalhost() 
  // ... Figure out locations of SDK and apprtc code ...
  CommandLine command_line(CommandLine::NO_PROGRAM);
  EXPECT_TRUE(GetPythonCommand(&command_line));

  command_line.AppendArgPath(appengine_dev_appserver);
  command_line.AppendArgPath(apprtc_dir);
  command_line.AppendArg("--port=9999");
  command_line.AppendArg("--admin_port=9998");
  command_line.AppendArg("--skip_sdk_update_check");

  VLOG(1) << "Running " << command_line.GetCommandLineString();
  return base::LaunchProcess(command_line, base::LaunchOptions(),
                             &dev_appserver_);
}

Figuring out Whether the Local Server is Up 

bool LocalApprtcInstanceIsUp() {
  // Load the admin page and see if we manage to load it right.
  ui_test_utils::NavigateToURL(browser(), GURL("localhost:9998"));
  content::WebContents* tab_contents =
      browser()->tab_strip_model()->GetActiveWebContents();
  std::string javascript =
      "window.domAutomationController.send(document.title)";
  std::string result;
  if (!content::ExecuteScriptAndExtractString(tab_contents, 
                                              javascript,
                                              &result))
    return false;

  return result == kTitlePageOfAppEngineAdminPage;
}

while (!LocalApprtcInstanceIsUp())
  VLOG(1) << "Waiting for AppRTC to come up...";

Launching the Browsers 

from mozprofile import profile
from mozrunner import runner

WEBRTC_PREFERENCES = {
    'media.navigator.permission.disabled': True,
}

def main():
  # Set up flags, handle SIGTERM, etc
  # ...
  firefox_profile = 
      profile.FirefoxProfile(preferences=WEBRTC_PREFERENCES)
  firefox_runner = runner.FirefoxRunner(
      profile=firefox_profile, binary=options.binary, 
      cmdargs=[options.webpage])

  firefox_runner.start()

GURL room_url = 
    GURL(base::StringPrintf("http://localhost:9999?r=room_%d",
                            base::RandInt(0, 65536)));
content::WebContents* chrome_tab = 
    OpenPageAndAcceptUserMedia(room_url);
ASSERT_TRUE(LaunchFirefoxWithUrl(room_url));

run_firefox_webrtc.py --binary /path/to/firefox --webpage http://localhost::9999?r=my_room

<div class="button">Save</div>

<div class="button">Edit</div>

div.button

//div[@class='button' and text()='Save']

contact-form.save-button
contact-form.edit-button

@Override protected void onEnsureDebugId(String baseId) {
  super.onEnsureDebugId(baseId);
  saveButton.ensureDebugId(baseId + ".save-button");
  editButton.ensureDebugId(baseId + ".edit-button");
}

<tr id="feedback-{{$index}}" class="feedback" ng-repeat="feedback in ctrl.feedbacks" >

@UiField FlexTable table;
UIObject.ensureDebugId(table.getCellFormatter().getElement(rowIndex, columnIndex),
    baseID + colIndex + "-" + rowIndex);

@Test public void shouldNavigateToPhotosPage() {
  String baseUrl = "http://plus.google.com/";
  Navigator nav = new Navigator(baseUrl);
  nav.goToPhotosPage();
  assertEquals(baseUrl + "/u/0/photos", nav.getCurrentUrl());
}

@Test public void shouldNavigateToPhotosPage() {
  Navigator nav = new Navigator("http://plus.google.com/");
  nav.goToPhotosPage();
  assertEquals("http://plus.google.com//u/0/photos", nav.getCurrentUrl()); // Oops!
}

Introduction

How we measured

Results

Future work

@Test public void testProcessTransaction() {
  User user = newUserWithBalance(LOW_BALANCE_THRESHOLD.plus(dollars(2));
  transactionProcessor.processTransaction(
      user,
      new Transaction("Pile of Beanie Babies", dollars(3)));
  assertContains("You bought a Pile of Beanie Babies", ui.getText());
  assertEquals(1, user.getEmails().size());
  assertEquals("Your balance is low", user.getEmails().get(0).getSubject());
}

@Test public void testProcessTransaction_displaysNotification() {
  transactionProcessor.processTransaction(
      new User(), new Transaction("Pile of Beanie Babies"));
  assertContains("You bought a Pile of Beanie Babies", ui.getText());
}
@Test public void testProcessTransaction_sendsEmailWhenBalanceIsLow() {
  User user = newUserWithBalance(LOW_BALANCE_THRESHOLD.plus(dollars(2));
  transactionProcessor.processTransaction(
      user,
      new Transaction(dollars(3)));
  assertEquals(1, user.getEmails().size());
  assertEquals("Your balance is low", user.getEmails().get(0).getSubject());
}

class LinkGeneratorTest(googletest.TestCase):

  def setUp(self):
    self.generator = link_generator.LinkGenerator()

  def testGetLinkFromIDs(self):
    expected = ('https://frontend.google.com/advancedSearchResults?'
                's.op=ALL&s.r0.field=ID&s.r0.val=1288585+1310696+1346270+')
    actual = self.generator.GetLinkFromIDs(set((1346270, 1310696, 1288585)))
    self.assertEqual(expected, actual)

import urllib

class LinkGenerator(object):

  _URL = (
      'https://frontend.google.com/advancedSearchResults?'
      's.op=ALL&s.r0.field=ID&s.r0.val=')

  def GetLinkFromIDs(self, ids):
    result = []
    for id in sorted(ids):
      result.append('%s ' % id)
    return self._URL + urllib.quote_plus(''.join(result))

@Test public void shouldPerformAddition() {
  Calculator calculator = new Calculator(new RoundingStrategy(), 
      "unused", ENABLE_COSIN_FEATURE, 0.01, calculusEngine, false);
  int result = calculator.doComputation(makeTestComputation());
  assertEquals(5, result); // Where did this number come from?
}

@Test public void shouldPerformAddition() {
  Calculator calculator = newCalculator();
  int result = calculator.doComputation(makeAdditionComputation(2, 3));
  assertEquals(5, result);
}

interface Service {
  Data get();
}

when(service.get()).thenReturn(cannedData);

interface Service {
  void get(Callback callback);
}

doAnswer(new Answer<Void>() {
    public Void answer(InvocationOnMock invocation) {
       Callback callback = (Callback) invocation.getArguments()[0];
       callback.onSuccess(cannedData);
       return null;
    }
}).when(service).get(any(Callback.class));

interface Translator {
  String translate(String msg);
}

when(translator.translate(any(String.class))).thenAnswer(reverseMsg())
...
// extracted a method to put a descriptive name
private static Answer<String> reverseMsg() { 
  return new Answer<String>() {
    public String answer(InvocationOnMock invocation) {
       return reverseString((String) invocation.getArguments()[0]));
    }
  }
}

• Simplify your synchronization logic. If it’s too hard to understand, it will be difficult to reproduce and debug complex concurrency problems.
• Always obtain locks in the same order. This is a tried-and-true guideline to avoid deadlocks, but I still see code that breaks it periodically. Define an order for obtaining multiple locks and never change that order.
• Don’t optimize by creating many fine-grained locks, unless you have verified that they are needed. Extra locks increase concurrency complexity.
• Avoid shared memory, unless you truly need it. Shared memory access is very easy to get wrong, and the bugs may be quite difficult to reproduce.

• Stress test your system regularly. You don't want to be surprised by unexpected failures when your system is under heavy load.
• Test timeouts. Create tests that mock/fake dependencies to test timeout code. If your timeout code does something bad, it may cause a bug that only occurs under certain system conditions.
• Test with debug and optimized builds. You may find that a well behaved debug build works fine, but the system fails in strange ways once optimized.
• Test under constrained resources. Try reducing the number of data centers, machines, processes, threads, available disk space, or available memory. Also try simulating reduced network bandwidth.
• Test for longevity. Some bugs require a long period of time to reveal themselves. For example, persistent data may become corrupt over time.
• Use dynamic analysis tools like memory debuggers, ASan, TSan, and MSan regularly. They can help identify many categories of unreproducible memory/threading issues.

void ScheduleEvent(int timeDurationMilliseconds) {
  if (timeDurationMilliseconds <= 0) {
    timeDurationMilliseconds = 1;
  }
  ...
}

• Enforce preconditions in your functions unless you have a good reason not to.

double GetMonthlyLoanPayment() {
  double rate = GetTodaysInterestRateFromExternalSystem();
  if (rate < 0.001 || rate > 0.5) {
    throw BadInterestRate(rate);
  }
  ...
}

• When possible, use defensive programming to verify the work of your dependencies with known risks of failure like user-provided data, I/O operations, and RPC calls.

• Use fuzz testing to test your systems hardiness when enduring bad data.

• Only hide errors from the user when you are certain that there is no impact to system state or the user.
• Any error with impact to the user should be reported to the user with instructions for how to proceed. The information shown to the user, combined with data available to an engineer, should be enough to determine what went wrong.

• Always test your error handling code. This is usually best accomplished by mocking or faking the component triggering the error.
• It’s also a good practice to examine your log quality for all types of error handling.

• Try to guarantee uniqueness of all keys.
• When not possible to guarantee unique keys, check if the recently generated key is already in use before using it.
• Watch out for potential race conditions here and avoid them with synchronization.

• Always test for concurrent data access if it’s a feature of the system. Actually, even if it’s not a feature, verify that the system rejects it. Testing concurrency can be challenging. An approach that usually works for me is to create many worker threads that simultaneously attempt access and a master thread that monitors and verifies that some number of attempts were indeed concurrent, blocked or allowed as expected, and all were successful. Programmatic post-analysis of all attempts and changing system state may also be necessary to ensure that the system behaved well.

• Understand when the APIs and operations you use might have undefined behavior and prevent those conditions.
• Do not depend on data structure iteration order unless it is guaranteed. It is a common mistake to depend on the ordering of sets or associative arrays.

• Follow good logging practices, especially in your error handling code.
• If logs are stored on a user’s machine, create an easy way for them to provide you the logs.

• Save your test logs for potential analysis later.