While working with a search quality development team, I was asked to collect information from their result pages across all the supported languages. The aim was to quickly get an overview, and then manually look through them for irregularities. One option would have been to grab the pages using tools like Selenium or WebDriver. However, this would have been complex and expensive. Instead, I opted for a much simpler solution: Display each language variation in a separate IFrame within the same HTML page.

An example of how this looks, can be seen here, where the Google Search page is shown in 43 different languages. There are also some links for other Google sites, or you can type in your own link, and the query attribute for language, "hl=", will be appended at the end.

(Warning: Do not try this with YouTube, as 43 Flash windows on the same pages will crash your browser. Also, Firefox 2 is known to be slow, while Firefox 3 works fine.)

The JavaScript Code

Creating the IFrames is easy using JavaScript, as can be seen in the example below. I assume that the array of languages to iterate over is retrieved by the function getLanguages(). Then a simple loop uses document.write(...) to dynamically add the IFrames. It is worth mentioning that this method seemed to be the best way of dynamically creating them; using the document.createElement(...) resulted in some complex race condition issues when adding the IFrames and their content at the same time.

var languages = getLanguages();
for (var lang, i = 0; lang = languages[i]; i++) {
  document.write('<hr><a name="' + lang + '"/>' +
                 '<h2>' + lang + '</h2><center>' +
                 '<iframe src="' + url + lang +
                 '" width="' + queryMap['width'] +
                 '" height="' + queryMap['height'] +
                 '"></iframe>' +
                 </center><br/><br/>');
}

The rest of the source code, can be seen in this example. Nothing else is needed to get the overview.

Conclusion

The example in this article shows that a very simple and inexpensive solution can be useful for exploratory testing of web pages; especially when quickly looking over the same pages in multiple languages. The small amount of code required, makes it easy to customize for any project or page, and the fact that the requests are done dynamically, gives a view which is always up to date.

Of course, this type of overview lends itself best to very simple stateless pages, which do not require complex navigation. It would for example be more difficult to get the same list of IFrames for Gmail, or other complex applications. Also, as the IFrames are loaded dynamically, no history is kept, so tracking when a potential bug was introduced in the page under test might prove more tedious.

Furthermore, it should be noted that the overview only simplifies a manual process of looking at the pages. In some situations, this might be very beneficial, and enough for the developer, while in other projects more automated tests might be designed. E.g., it could be difficult to automate tests for aesthetic issues, but easy to spot them manually, while it may prove more beneficial to automate checks for English terms in other languages.

Finally, a word on the issue of translations and language skills. The overview in this example, quickly highlights issues like incorrect line wrapping, missing strings, etc. in all variations of the page. Also, it was easy to spot strings not already translated in some of the languages, like Japanese, and in fact I reported a bug against the Search front page for this. However, for other issues, more language specific skills are necessary to spot and file bugs: E.g. should the Arabic page show Eastern or Western Arabic numerals? And have the Danes picked the English term for "Blogs", while the Norwegians and Swedish prefer a localized term? I don't know.

interface Duck {
Point getLocation();
void quack();
void swimTo(Point p);
}

class ForwardingDuck implements Duck {
private final Duck d;
ForwardingDuck(Duck delegate) {
this.d = delegate;
}
public Point getLocation() {
return d.getLocation();
}
public void quack() {
d.quack();
}
public void swimTo(Point p) {
d.swimTo(p);
}
}

public void testDuckCrossesPoolAndQuacks() {
final Duck mock = EasyMock.createStrictMock(Duck.class);
mock.swimTo(FAR_SIDE);
mock.quack(); // quack after the race
EasyMock.replay(mock);
Duck duck = OlympicDuck.createInstance();
Duck partialDuck = new ForwardingDuck(duck) {
@Override public void quack() {
mock.quack();
}
@Override public void swimTo(Point p) {
mock.swimTo(p);
super.swimTo(p);
}
// no need to @Override “Point getLocation()”
}

OlympicSwimmingEvent.createEventForDucks()
.withDistance(ONE_LENGTH)
.sponsoredBy(QUACKERS_CRACKERS)
.addParticipant(partialDuck)
.doRace();
MatcherAssert.assertThat(duck.getLocation(), is(FAR_SIDE));
EasyMock.verify(mock);

• quack() calls the mock object. It verifies that the duck doesn't promote the sponsor (by quacking) until after the race. (We skip the real quack() method so that our continuous build doesn't drive us crazy.)


• getLocation() calls the real object. It allows us to use the OlympicDuck's location logic instead of rewriting/simulating the logic from that implementation.


• swimTo(point) calls both objects. It allows us to verify the call to the real duck before executing it.

MVC

MVP

public CongressionalHearingView() {
testimonyWidget.addModifyListener(
new ModifyListener() {
public void modifyText(ModifyEvent e) {
presenter.onModifyTestimony(); // presenter decides action to take
}});
}

public class CongressionalHearingPresenter {
public void onModifyTestimony() {
model.parseTestimony(view.getTestimonyText()); // manipulate model
}
public void setWitness(Witness w) {
view.setTestimonyText(w.getTestimony()); // update view
}
}

public void testSetWitness() {
spyView = new SpyCongressionalHearingView();
presenter = new CongressionalHearingPresenter(spyView);
presenter.setWitness(new Witness(“Mark McGwire”, “I didn't do it”));
assertEquals( “I didn't do it”, spyView.getTestimonyText());
}

TEST_F(BabyCondorTest, EatsCarrion) {
  FakeCondor mother;
  scoped_ptr carrion;
  BabyCondor* pchick = &chick_;
  mother.Imprint(vector(&pchick, &pchick + 1)); // just one chick
  while(!chick_.HasFood()) {
    mother.Eat(); // disposes of any food the mother kept for herself
    mother.Scavenge(carrion.reset(new FakeCarrion)); // finds new food
    mother.RandomlyDistributeFoodAmongYoungAndSelf(); // feeds baby or mom
  }
  chick_.Eat();
  EXPECT_TRUE(carrion->WasEaten());
}

class CondorMotherhoodRoleInterface {
 public:
  virtual Carrion* GiveFood() = 0;
  virtual SomeReturnTypes* OtherMomBehaviors() = 0;
};

class CondorFeedingPuppet: public CondorMotherhoodRoleInterface {
 public:
  virtual Carrion* GiveFood() { return test_carrion_; }
  virtual SomeReturnTypes* OtherMomBehaviors() { return NULL; }
  Carrion* test_carrion_; // public var is tolerable in a one-off object
};

TEST_F(BabyCondorTest, EatsCarrion) {
  CondorFeedingPuppet mother; FakeCarrion test_carrion;
  mother.test_carrion_ = &test_carrion;
  chick_.ReceiveFood(&mother);
  chick_.Eat();
  EXPECT_TRUE(test_carrion.WasEaten());
}

public interface Atm {
  boolean enterAccount(String accountNumber);
  boolean enterPin(String pin);
  boolean enterWithdrawalAmount(int dollars);
}

public void testAtmLogin() {
  Atm mockAtm = createMock(Atm.class); // 1
  EasyMock.expect(mockAtm.enterAccount("MyAccount")).andReturn(true); // 2
  EasyMock.expect(mockAtm.enterPin("1234")).andReturn(true); // 3
  EasyMock.replay(mockAtm); // 4
  Account account = new Account();
  account.login(mockAtm); // 5
  assertTrue(account.isLoggedIn());
  EasyMock.verify(mockAtm); // 6
}

 4 int bottles_of_beer = 0;
 5 void Intern1() { bottles_of_beer++; } // Intern1 forgot to use Mutex.
 6 void Intern2() { bottles_of_beer++; } // Intern2 copied from Intern1.
 7 int main() {
 8 // Folks, bring me one bottle of beer each, please.
 9 ClosureThread intern1(NewPermanentCallback(Intern1)),
10 intern2(NewPermanentCallback(Intern2));
11 intern1.SetJoinable(true); intern2.SetJoinable(true);
12 intern1.Start(); intern2.Start();
13 intern1.Join(); intern2.Join();
14 CHECK_EQ(2, bottles_of_beer) << "Who didn't bring me my beer!?";
15 }

$ helgrind path/to/your/program
Possible data race during read of size 4 at 0x5429C8
at 0x400523: Intern2() tott.cc:6
  by 0x400913: _FunctionResultCallback_0_0::Run() ...
  by 0x4026BB: ClosureThread::Run() ...
  ...
  Location 0x5429C8 has never been protected by any lock
  Location 0x5429C8 is 0 bytes inside global var "bottles_of_beer"
  declared at tott.cc:4

// Deletes files that are no longer reachable via our storage system's
// metadata.
class FileCleaner {
public:
  class Env {
  public:
    virtual bool MatchFiles(const char* pattern, vector* filenames) = 0;
    virtual bool BulkDelete(const vector& filenames) = 0;
    virtual MetadataReader* NewMetadataReader() = 0;
    virtual ~Env();
  };
  // Constructs a FileCleaner. Uses “env” to access files and metadata.
  FileCleaner(Env* env, QuotaManager* qm);
  // Deletes files that are not reachable via metadata.
  // Returns true on success.
  bool CleanOnce();
};

class NoFileSystemEnv : public FileCleaner::Env {
  virtual bool MatchFiles(const char* pattern, vector* filenames) {
  match_files_called_ = true;
  return false;
  }
  ...
};

TEST(FileCleanerTest, FileCleaningFailsWhenFileSystemFails) {
  NoFileSystemEnv* env = new NoFileSystemEnv();
  FileCleaner cleaner(env, new MockQuotaManager());
  ASSERT_FALSE(cleaner.CleanOnce());
  ASSERT_TRUE(env->match_files_called_);
}

ASSERT_FLOAT_EQ(expected, actual);
ASSERT_DOUBLE_EQ(expected, actual);
EXPECT_FLOAT_EQ(expected, actual);
EXPECT_DOUBLE_EQ(expected, actual);

assertEquals(float expected, float actual, float delta);
assertEquals(double expected, double actual, double delta);

TEST(SquareRootTest, CorrectResultForPositiveNumbers) {
  EXPECT_FLOAT_EQ(2.0f, FloatSquareRoot(4.0f));
  EXPECT_FLOAT_EQ(23.3333f, FloatSquareRoot(544.44444f));
  EXPECT_DOUBLE_EQ(2.0, DoubleSquareRoot(4.0));
  EXPECT_DOUBLE_EQ(23.33333333333333, DoubleSquareRoot(544.44444444444444));
    // the above succeeds
  EXPECT_EQ(2.0, DoubleSquareRoot(4.0));
    // the above fails
  EXPECT_EQ(23.33333333333333, DoubleSquareRoot(544.44444444444444));
}

function showProgress(status) {
  status.message = "Loading";
  for (var time = 1000; time <= 3000; time+= 1000) {
    // Append a '.' to the message every second for 3 secs.
    setTimeout(function() {
      status.message += ".";
    }, time);
  }
  setTimeout(function() {
    // Special case for the 4th second.
    status.message = "Done";
  }, 4000);
}

function testUpdatesStatusMessageOverFourSeconds() {
  Clock.reset(); // Clear any existing timeout functions on the event queue.
  var status = {};
  showProgress(status); // Call our function.

  assertEquals("Loading", status.message);

  Clock.tick(2000); // Call any functions on the event queue that have been
                    // scheduled for the first two seconds.
  assertEquals("Loading..", status.message);

  Clock.tick(2000); // Same thing again, for the next two seconds.
  assertEquals("Done", status.message);
}

GIMME lol_io LIKE LOLIO

SO IM LIKE PROCESSIN WIT DATAZ OK?
  GIMME EACH BUCKET IN UR DATAZ OK?
    BUCKET OWN FUBARRED?
      N CAN HAS NONE
    NOPE?
      N CAN HAS 1
  KTHXBYE N

IZ __name__ KINDA LIKE “__main__”?
  UR PROCESSIN WIT LOLIO OWN GET_SOME_DATAZ
  BTW, GET_SOME_DATAZ USES UR INTERNETS LOL

GIMME mock_lol_io LIKE LOLIO

BTW, GIMME THING TO TEST
BTW, TEST THE THING NOW KTHX

BTW, SOMETIMES THEY BE CALLIN DIS DEPENDENCY INJECTION ROFL

BTW, YOU CAN UZE MOCKZ N STUF FER DIS LOOK:

IN MAI library GIMME mock_filesystem LIKE LOL_FAKE_FILEYSTEM

BTW, NOW U CAN USE LOL_FAKE_FILESYSTEM TO MAKE FAKE FILEZ IN MEMORY N STUFF
BTW, IS FASTER THAN OPENIN FILEZ ON TEST SERVAR

const struct {const char* word; bool is_word;} test_data[] = {
  {"milk", true}, {"centre", false}, {"jklm", false},
};

TEST(IsWordTest, TestEverything) {
  for (int i = 0; i < ARRAYSIZE(test_data); i++)
    EXPECT_EQ(test_data[i].is_word, IsWord(test_data[i].word));
}

Locale LOCALES[] = { Word::US, Word::UK, Word::France, ... };
const struct {const char* word; bool is_word[NUM_LOCALES];} data[] = {
  {"milk", {true, true, false, ...}}, // one bool per language
  {"centre", {false, true, true, ...}},
  {"jklm", {false, false, false, ...}}
};

TEST(IsWordTest, TestEverything) {
  for (int i = 0; i < ARRAYSIZE(data); i++)
    for (int j = 0; j < ARRAYSIZE(LOCALES); j++)
      EXPECT_EQ(data[i].is_word[j], IsWord(data[i].word, LOCALES[i]));
}

TEST(IsWordTest, IsWordInMultipleLocales) {
  EXPECT_TRUE(IsWord("milk", Word::UK));
  EXPECT_TRUE(IsWord("milk", Word::US));
  EXPECT_FALSE(IsWord("milk", Word::France));
}

TEST(IsWordTest, IsWordWithNonExistentWord) { // 'jklm' test is not repeated
  EXPECT_FALSE(IsWord("jklm", Word::US)); // as it uses the same code path
}

def testInternMakesCoffee(self):

  self.caffeinated = False
  def DrinkCoffee(): self.caffeinated = True

  DelegateToIntern(work=Intern().MakeCoffee, callback=DrinkCoffee)

  self.assertFalse(self.caffeinated, "I watch YouTubework; intern brews")
  time.sleep(60) # 1min should be long enough to make coffee, right?
  self.assertTrue(self.caffeinated, "Where's mah coffee?!?")

def testInternMakesCoffee(self):

  is_started, can_finish, is_done = Event(), Event(), Event()

  def FakeCoffeeMaker():
    is_started.set() # Allow is_started.wait() to return.
    # Wait up to 1min for can_finish.set() to be called. The timeout
    # prevents failures from hanging, but doesn't delay a passing test.
    can_finish.wait(timeout=60) # .await() in Java

  DelegateToIntern(work=FakeCoffeeMaker, callback=lambda:is_done.set())

  is_started.wait(timeout=60)
  self.assertTrue(is_started.isSet(), "FakeCoffeeMaker should have started")
  self.assertFalse(is_done.isSet(), "Don't bug me before coffee's made")

  can_finish.set() # Now let FakeCoffeeMaker return.
  is_done.wait(timeout=60)
  self.assertTrue(is_done.isSet(), "Intern should ping when coffee's ready")

• In hundreds of stalls in 30 Google offices
• With episodes covering many programming languages and application domains
• Written by volunteer authors from offices worldwide.
• TotT is also published to fans outside our walls on Google's public Testing Blog.

Speaking of black-and-white, take decoding of two-dimensional barcodes, like QR Codes. From a blurry, skewed, rotated image of a barcode, software has to pick it out, transform it, and decode it:

Even the best software can't always find that barcode. What should tests for such software do?

We have some answers, from experience testing such software. We have two groups of black-box tests that verify that images decode correctly: must-have and nice-to-have. Tests verify that the must-have set – the easy images – definitely decode correctly. This is what traditional tests would include, which typically demand a 100% pass rate. But we also see how well we do on the more difficult nice-to-have set. We might verify that 50% of them decode, and fail otherwise.

The advantage? We can include tougher test cases in unit tests, instead of avoiding them. We can observe small changes – improvements as well as degradations – in decode accuracy over time. It doubles as a crude quality evaluation framework.

Where can this progressive thinking be applied? Maybe when your code...

Only needs to be correct in most cases. As here, write tests to verify easy cases work, but also that some hard cases pass too.

Needs to be fast. You write unit tests that verify it runs "fast enough" on simple input. How about writing tests that make sure it runs "fast enough" on most of some larger inputs too?

Is heuristic. You write unit tests that verify that the answer is “really close” to optimal on simple input, but also that it's “kind of close” on difficult input.

By the way, did we mention project ZXing, Google's open-source decoder project? Or that Print Ads is already helping clients place these two-dimensional barcodes in the New York Times? Or that there are other formats like Data Matrix? or that you can put more than just a URL in these barcodes? This is a technology going global, so, time to read up on it.

Remember to download this episode of Testing on the Toilet and post it in your office.

public interface PlanetaryDeathRay {
  public void aim(double xPosition, double yPosition);
  public boolean fire(); /* call this if she says the rebel
                            base is on Dantooine */
}

public class BlueLaserPlanetaryDeathRay
    implements PlanetaryDeathRay { /* implementation here */ }
public class GreenLaserPlanetaryDeathRay
    implements PlanetaryDeathRay { /* implementation here */ }

public abstract class PlanetaryDeathRayTestCase
    extends TestCase {
  protected PlanetaryDeathRay deathRay;
  @Override protected void setUp() {
    deathRay = createDeathRay();
  }
  @Override protected void tearDown() {
    deathRay = null;
  }
  protected abstract PlanetaryDeathRay createDeathRay();
      /* create the PlanetaryDeathRay to test */

  public void testAim() {
    /* write implementation-independent tests here against
       deathRay.aim() */
  }
  public void testFire() {
    /* write implementation-independent tests here against
       deathRay.fire() */
  }
}

public class BlueLaserPlanetaryDeathRayTest
    extends PlanetaryDeathRayTestCase {
  protected PlanetaryDeathRay createDeathRay() {
    return new BlueLaserPlanetaryDeathRay();
  }
}

Google C++ Testing Framework supports two families of assertions with the same interface:

• ASSERT: Fails fast, aborting the current function.
• EXPECT: Continues after the failure.

As Moka the code monkey has shown Uni the testing robot, EXPECT is often more appropriate because it: reveals more failures in a single run, and allows more to be fixed in a single edit/compile/run-tests cycle. Example:

TEST(WordStemmerTest, StemsPluralSuffixes) {
  EXPECT_STREQ("jump", stemmer->Stem("jumps"));
  EXPECT_STREQ("pixi", stemmer->Stem("pixies"));
  EXPECT_STREQ("prioriti", stemmer->Stem("priorities"));
  // etc ...
}

Use ASSERT when it doesn't make sense to continue. Example:

TEST(FileGeneratorTest, CreatesFileContainingSequenceOfChars) {
  ASSERT_TRUE(fp = fopen(path, "r"))
        << "Failed to open " << path;
  ASSERT_EQ(10, fread(buffer, 1, 10, fp))
        << "File " << path << " is too small";
  buffer[10] = '\0';
  EXPECT_STREQ("123456789", buffer)
        << "File " << path << " is corrupted";
}

Remember to download this episode of Testing on the Toilet and post it in your office.

public class MyObject {
  public int doSomething(int id) {
    return TheirEntity.selectById(id).getSomething();
  }
}

interface TheirEntityRepository {
  TheirEntity selectById(int id);
  // other static methods on TheirEntity can be
  // represented here too
}
public class TheirEntityStaticRepository
        implements TheirEntityRepository {
  public TheirEntity selectById(int id) { // non-static
    return TheirEntity.selectById(id); // calls static method
  }
}

public class MyObject {
  private final TheirEntityRepository repository;
  public MyEntity(TheirEntityRepository arg) {
    this.repository = arg;
  }
  public int doSomething(int id) {
    return repository.selectById(id).getSomething();
  }
}

Item item = new Item(ITEM_NAME);
ShoppingCart cart = new ShoppingCart();
cart.add(item, QUANTITY);
assertEquals(QUANTITY, cart.getItem(ITEM_NAME));

A Stub overrides the real object and returns hard-coded values. Testing with stubs only is state-based testing; you exercise the system and then assert that the system is in an expected state.

ItemPricer pricer = new ItemPricer() {
  public BigDecimal getPrice(String name){ return PRICE; }
};
ShoppingCart cart = new ShoppingCart(pricer);
cart.add(dummyItem, QUANTITY);
assertEquals(QUANTITY*PRICE, cart.getCost(ITEM_NAME));

A Mock can return values, but it also cares about the way its methods are called (“strict mocks” care about the order of method calls, whereas “lenient mocks” do not.) Testing with mocks is interaction-based testing; you set expectations on the mock, and the mock verifies the expectations as it is exercised. This example uses JMock to generate the mock (EasyMock is similar):

Mockery context = new Mockery();
final ItemPricer pricer = context.mock(ItemPricer.class);
context.checking(new Expectations() {{
  one(pricer).getPrice(ITEM_NAME);
  will(returnValue(PRICE));
}});
ShoppingCart cart = new ShoppingCart(pricer);
cart.add(dummyItem, QUANTITY);
cart.getCost(ITEM_NAME);
context.assertIsSatisfied();

A Spy serves the same purpose as a mock: returning values and recording calls to its methods. However, tests with spies are state-based rather than interaction-based, so the tests look more like stub style tests.

TransactionLog log = new TransactionLogSpy();
ShoppingCart cart = new ShoppingCart(log);
cart.add(dummyItem, QUANTITY);
assertEquals(1, logSpy.getNumberOfTransactionsLogged());
assertEquals(QUANTITY*PRICE, log.getTransactionSubTotal(1));

A Fake swaps out a real implementation with a simpler, fake implementation. The classic example is implementing an in-memory database.

Repository repo = new InMemoryRepository();
ShoppingCart cart = new ShoppingCart(repo);
cart.add(dummyItem, QUANTITY);
assertEquals(1, repo.getTransactions(cart).Count);
assertEquals(QUANTITY, repo.getById(cart.id()).getQuantity(ITEM_NAME));

While this episode used Java for its examples, all of the above “friends” certainly exist in C++, Python, JavaScript, and probably in YOUR favorite language as well.

Remember to download this episode of Testing on the Toilet and post it in your office.

Let's consider a case where branch coverage and statement coverage aren't the same. Suppose we test the following snippet. We can get complete statement coverage with a single test by using a berserk EvilOverLord:

bool DeathRay::ShouldFire(EvilOverLord& o, Target& t) {
  double accumulated_rage = 0.0;

  if (o.IsBerserk())
    accumulated_rage += kEvilOverlordBerserkRage;

  accumulated_rage += o.RageFeltTowards(t);
  return (accumulated_rage > kDeathRayRageThreshold);
}

But what if DeathRay should fire at this Target even with a non-berserk Overlord? Well, we need another test for that. What should the test be? Let's rewrite the code a little bit. We would never see code like this in the real world, but it'll help us clarify an important point.

bool DeathRay::ShouldFire(EvilOverLord& o, Target& t) {
  double accumulated_rage = 0.0;

  if (o.IsBerserk()) {
    accumulated_rage += kEvilOverlordBerserkRage;
  } else {
  }

  accumulated_rage += o.RageFeltTowards(t);
  return (accumulated_rage > kDeathRayRageThreshold);
}

Why do we add an else clause if it doesn't actually do anything? If you were to draw a flowchart of both snippets (left as an exercise – and we recommend against using the paper provided), the flowcharts would be identical. The fact that the else isn't there in the first snippet is simply a convenience for us as coders – we generally don't want to write code to do nothing special – but the branch still exists... put another way, every if has an else. Some of them just happen to be invisible.

When you're testing, then, it isn't enough to cover all the statements – you should cover all the the edges in the control flow graph – which can be even more complicated with loops and nested ifs. In fact, part of the art of large-scale white-box testing is finding the minimum number of tests to cover the maximum number of paths. So the lesson here is, just because you can't see a branch doesn't mean it isn't there – or that you shouldn't test it.

Remember to download this episode of Testing on the Toilet and post it in your office.

public class Client {
  public int process(Params params) {
    Server server = Server.getInstance();
    Data data = server.retrieveData(params);
    ...
  }
}

public class Client {
  private final Server server;

  public Client(Server server) {
    this.server = server;
  }

  public int process(Params params){
    Data data = this.server.retrieveData(params);
    ...
  }
}

public void testProcess() {
  Server mockServer = createMock(Server.class);
  Client c = new Client(mockServer);
  assertEquals(5, c.process(params));
}

bool SomeCollection::GetObjects(vector* objects) const {
  objects->clear();
  typedef vector::const_iterator Iterator;
  for (Iterator i = collection_.begin();
        i != collection_.end();
        ++i) {
    if ((*i)->IsFubarred()) return false;
    objects->push_back(*i);
  }
  return true;
}

bool SomeCollection::GetObjects(vector* objects) const {
  vector known_good_objects;
  typedef vector::const_iterator Iterator;
  for (Iterator i = collection_.begin();
        i != collection_.end();
        ++i) {
    if ((*i)->IsFubarred()) return false;
    known_good_objects->push_back(*i);
  }
  objects->swap(known_good_objects);
  return true;
}

1. http://www.gotw.ca/publications
2. Scott Meyers, Effective C++
3. http://www.boost.org/more/generic_exception_safety.html

def CreateGapLease(self):
  data_file = open('/leases/gap', 'w+')
  data_file.write(contract_data)
  data_file.close()

def testCreateGapLease(self):
  contract_writer.CreateGapLease()
  self.assertEqual(ReadFileContents('/leases/gap'),
         contract_data)

def testCreateGapLease(self):
  if os.path.exists(lease_file):
    RemoveFile(lease_file)
  ...

def CreateGapLease(self, lease_path=None):
  if lease_path is None:
    lease_path = '/leases/gap'
  ...

def testCreateGapLease(self):
  lease_file = os.path.join(FLAGS.test_tmpdir, 'gap')
  contract_writer.CreateGapLease(lease_path=lease_file)
  self.assertEqual(ReadFileContents(lease_file),
         contract_data)

/** Return a date object representing the start of the next minute from now */
public Date nextMinuteFromNow() {
  long nowAsMillis = System.currentTimeMillis();
  Date then = new Date(nowAsMillis + 60000);
  then.setSeconds(0);
  then.setMilliseconds(0);
  return then;
}

1. There is no easy way to test corner cases; you're at the mercy of the system clock to supply input conditions.


2. When nextMinuteFromNow() returns, the time has changed. This means the test will not be an assertion, it will be a guess, and may generate low-frequency, hard-to-reproduce failures... flakiness! Class loading and garbage collection pauses, for example, can influence this.

public Date minuteAfter(Date now) {
  Date then = new Date(now.getTime() + 60000);
  then.setSeconds(0);
  then.setMilliseconds(0);
  return then;
}

// Retain original functionality
@Deprecated public Date nextMinuteFromNow() {
  return minuteAfter(new Date(System.currentTimeMillis()));
}

public void testMinuteAfter () {
  Date now = stringToDate("2012-12-22 11:59:59.999PM");
  Date then = minuteAfter(now);
  assertEquals("2012-12-23 12:00:00.000AM", dateToString(then));
}

public class PirateShipTest {
  @Test(dataProvider = "cannons")
  public void testFireCannonDepletesAmmunition(int ballsToLoad,
         int ballsToFire,
         int expectedRemaining) {
    PirateShip ship = new PirateShip("The Black Pearl");
    ship.loadCannons(ballsToLoad);
    for (int i = 0; i < ballsToFire; i++) {
      ship.fireCannon();
    }
    assertEquals(ship.getBallsRemaining(), expectedRemaining);
  }
  @DataProvider(name = "cannons")
  public Object[][] getShipSidesAndAmmunition() {
    // Each 1-D array represents a single execution of a @Test that
    // refers to this provider. The elements in the array represent
    // parameters to the test call.
    return new Object[] {
      {5, 1, 4}, {5, 5, 0}, {5, 0, 5}
    };
  }
}

<suite name="PirateShip suite" parallel="methods" thread-count="2">

@Test(expectedExceptions = { NoAmmunitionException.class })
public void testFireCannonEmptyThrowsNoAmmunitionException() {
  PirateShip ship = new PirateShip("The Black Pearl");
  ship.fireCannon();
}

Remember to download this episode of Testing on the Toilet and post it in your office.

• It does not take into account all the possible data inputs. Consider this code:
  
  

  int a = b / c;
  

  
  This could be covered with b = 18 and c = 6, but never tested where c = 0.


• Some tools do not provide fractional coverage. For instance, in the following code, when condition a is true, the code already has 100% coverage. Condition b is not evaluated.
  
  

  if (a || b) {
    // do something
  }
  



• Coverage analysis can only tell you how the code that exists has been exercised. It cannot tell you how code that ought to exist would have been executed. Consider the following:
  
  

  error_code = FunctionCall();
  // returns kFatalError, kRecoverableError, or kSuccess
  if (error_code == kFatalError) {
    // handle fatal error, exit
  } else {
    // assume call succeeded
  }
  

  
  
  This code is only handling two out of the three possible return values (a bug!). It is missing code to do error recovery when kRecoverableError is returned. With tests that generate only the values kFatalError and kSuccess, you will see 100% coverage. The test case for kRecoverableError does not increase coverage, and appears “redundant” for coverage purposes, but it exposes the bug!
  

1. Make your tests as comprehensive as you can, without coverage in mind. This means writing as many test case as are necessary, not just the minimum set of test cases to achieve maximum coverage.


2. Check coverage results from your tests. Find code that's missed in your testing. Also look for unexpected coverage patterns, which usually indicate bugs.


3. Add additional test cases to address the missed cases you found in step 2.


4. Repeat step 2-3 until it's no longer cost effective. If it is too difficult to test some of the corner cases, you may want to consider refactoring to improve testability.

public void decide(int a, int b, int c, int d,
      int e, int f) {
  if (a > b || c > d || e > f) {
    DoOneThing();
  } else {
    DoAnother();
  } // One-letter variable names are used here only
        because of limited space.
} // You should use better names. Do as I say, not
      as I do. :-)

public void decide(int a, int b, int c, int d,
      int e, int f) {
  if (tallerThan(a, b)
      || harderThan(c, d)
      || heavierThan(e, f)) {
    DoOneThing();
  } else {
    DoAnother();
  }
}
boolean tallerThan(int a, int b) { return a > b; }
      // Note “package scope”
boolean harderThan(int c, int d) { return c > d; }
      // rather than public; JUnit
boolean heavierThan(int e, int f) { return e > f; }
      // tests can access these.

1. ...read all 25 words out loud: RED, GREEN, BLUE, ... (Try it now!)


2. ...say all 25 colors out loud: GREEN, YELLOW, WHITE... (Try it now!)

public void testProtanopiaColorMatcherIsDistinguishable() {
  ColorMatcher colorMatcher = new ColorMatcher(PROTANOPIA);
  assertFalse(“BLUE and VIOLET are indistinguishable”,
    colorMatcher.isDistinguishable(Color.BLUE, Color.VIOLET));
}

Failure: testProtanopiaColorMatcherIsDistinguishable:
Message: BLUE and VIOLET are indistinguishable

Failure: testProtanopiaColorMatcherCannotDistinguishBetweenCertainPairsOfColors
Message: BLUE and VIOLET should be indistinguishable

• When someone breaks your test – will your test name and message be useful to them?


• Opinionated test names like testMethodDoesSomething can be more helpful than testMethod.


• Great test messages not only identify the actual behavior,but also the expected behavior.


• Should is a handy word to use in messages – it clarifies what expected behavior didn't actually happen.