Cross-posted from the Android Developers Google+ PageEarlier this year, we presented Espresso at GTAC as a solution to the UI testing problem. Today we are announcing the launch of the developer preview for Espresso!The compelling thing about developin…
by Eduardo Bravo Ortiz
“Mobile first” is the motto these days for many companies. However, being able to test a mobile app in a meaningful way is very challenging. On the Google+ team we have had our share of trial and error that has led us to successful strategies for testing mobile applications on both iOS and Android.
General
Android Testing
Unit Tests
Separating UI code from code logic is especially hard in Android. For example, an Activity is expected to act as a controller and view at the same time; make sure you keep this in mind when writing unit tests. Another useful recommendation is to decouple unit tests from the Android emulator, this will remove the need to build an APK and install it and your tests will run much faster. Robolectric is a perfect tool for this; it stubs the implementation of the Android platform while running tests.
Hermetic UI Tests
A hermetic UI test typically runs as a test without network calls or external dependencies. Once the tests can run in a hermetic environment, a white box testing framework like Espresso can simulate user actions on the UI and is tightly coupled to the app code. Espresso will also synchronize your tests actions with events on the UI thread, reducing flakiness. More information on Espresso is coming in a future Google Testing Blog article.
Monkey Tests
Monkey tests look for crashes and ANRs by stressing your Android application. They exercise pseudo-random events like clicks or gestures on the app under test. Monkey test results are reproducible to a certain extent; timing and latency are not completely under your control and can cause a test failure. Re-running the same monkey test against the same configuration will often reproduce these failures, though. If you run them daily against different SDKs, they are very effective at catching bugs earlier in the development cycle of a new release.
iOS Testing
Unit Tests
Unit test frameworks like OCUnit, which comes bundled with Xcode, or GTMSenTestcase are both good choices.
Hermetic UI Tests
KIF has proven to be a powerful solution for writing Objective-C UI tests. It runs in-process which allows tests to be more tightly coupled with the app under test, making the tests inherently more stable. KIF allows iOS developers to write tests using the same language as their application.
Following the same paradigm as Android UI tests, you want Objective-C tests to be hermetic. A good approach is to mock the server with pre-canned responses. Since KIF tests run in-process, responses can be built programmatically, making tests easier to maintain and more stable.
Monkey Tests
iOS has no equivalent native tool for writing monkey tests as Android does, however this type of test still adds value in iOS (e.g. we found 16 crashes in one of our recent Google+ releases). The Google+ team developed their own custom monkey testing framework, but there are also many third-party options available.
Backend Testing
A mobile testing strategy is not complete without testing the integration between server backends and mobile clients. This is especially true when the release cycles of the mobile clients and backends are very different. A replay test strategy can be very effective at preventing backends from breaking mobile clients. The theory behind this strategy is to simulate mobile clients by having a set of golden request and response files that are known to be correct. The replay test suite should then send golden requests to the backend server and assert that the response returned by the server matches the expected golden response. Since client/server responses are often not completely deterministic, you will need to utilize a diffing tool that can ignore expected differences.
To make this strategy successful you need a way to seed a repeatable data set on the backend and make all dependencies that are not relevant to your backend hermetic. Using in-memory servers with fake data or an RPC replay to external dependencies are good ways of achieving repeatable data sets and hermetic environments. Google+ mobile backend uses Guice for dependency injection, which allows us to easily swap out dependencies with fake implementations during testing and seed data fixtures.
Conclusion
Mobile app testing can be very challenging, but building a comprehensive test strategy that understands the nature of different platforms and tools is the key to success. Providing a reliable and hermetic test environment is as important as the tests you write.
Finally, make sure you prioritize your automation efforts according to your team needs. This is how we prioritize on the Google+ team:
Happy mobile testing from the Google+ team.
By Andrew Trenk This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office. Your trusty Calculator class is one of your most popular open sour…
By Andrew Trenk
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.
A test double is an object that can stand in for a real object in a test, similar to how a stunt double stands in for an actor in a movie. These are sometimes all commonly referred to as “mocks”, but it’s important to distinguish between the different types of test doubles since they all have different uses. The most common types of test doubles are stubs, mocks, and fakes.
A stub has no logic, and only returns what you tell it to return. Stubs can be used when you need an object to return specific values in order to get your code under test into a certain state. While it’s usually easy to write stubs by hand, using a mocking framework is often a convenient way to reduce boilerplate.
// Pass in a stub that was created by a mocking framework.
AccessManager accessManager = new AccessManager(stubAuthenticationService);
// The user shouldn't have access when the authentication service returns false.
when(stubAuthenticationService.isAuthenticated(USER_ID)).thenReturn(false);
assertFalse(accessManager.userHasAccess(USER_ID));
// The user should have access when the authentication service returns true.
when(stubAuthenticationService.isAuthenticated(USER_ID)).thenReturn(true);
assertTrue(accessManager.userHasAccess(USER_ID));
A mock has expectations about the way it should be called, and a test should fail if it’s not called that way. Mocks are used to test interactions between objects, and are useful in cases where there are no other visible state changes or return results that you can verify (e.g. if your code reads from disk and you want to ensure that it doesn’t do more than one disk read, you can use a mock to verify that the method that does the read is only called once).
// Pass in a mock that was created by a mocking framework.
AccessManager accessManager = new AccessManager(mockAuthenticationService);
accessManager.userHasAccess(USER_ID);
// The test should fail if accessManager.userHasAccess(USER_ID) didn't call
// mockAuthenticationService.isAuthenticated(USER_ID) or if it called it more than once.
verify(mockAuthenticationService).isAuthenticated(USER_ID);
A fake doesn’t use a mocking framework: it’s a lightweight implementation of an API that behaves like the real implementation, but isn’t suitable for production (e.g. an in-memory database). Fakes can be used when you can’t use a real implementation in your test (e.g. if the real implementation is too slow or it talks over the network). You shouldn’t need to write your own fakes often since fakes should usually be created and maintained by the person or team that owns the real implementation.
// Creating the fake is fast and easy.
AuthenticationService fakeAuthenticationService = new FakeAuthenticationService();
AccessManager accessManager = new AccessManager(fakeAuthenticationService);
// The user shouldn't have access since the authentication service doesn't
// know about the user.
assertFalse(accessManager.userHasAccess(USER_ID));
// The user should have access after it's added to the authentication service.
fakeAuthenticationService.addAuthenticatedUser(USER_ID);
assertTrue(accessManager.userHasAccess(USER_ID));
The term “test double” was coined by Gerard Meszaros in the book xUnit Test Patterns. You can find more information about test doubles in the book, or on the book’s website. You can also find a discussion about the different types of test doubles in this article by Martin Fowler.
By Jonathan Rockway and Andrew Trenk
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.
After many years of blogging, you decide to try out your blog platform’s API. You start playing around with it, but then you realize: how can you tell that your code works without having your test talk to a remote blog server?
public void deletePostsWithTag(Tag tag) {
for (Post post : blogService.getAllPosts()) {
if (post.getTags().contains(tag)) { blogService.deletePost(post.getId()); }
}
}
Fakes to the rescue! A fake is a lightweight implementation of an API that behaves like the real implementation, but isn’t suitable for production. In the case of the blog service, all you care about is the ability to retrieve and delete posts. While a real blog service would need a database and support multiple frontend servers, you don’t need any of that to test your code, all you need is any implementation of the blog service API. You can achieve this with a simple in-memory implementation:
public class FakeBlogService implements BlogService {
private final Setposts = new HashSet (); // Store posts in memory
public void addPost(Post post) { posts.add(post); }
public void deletePost(int id) {
for (Post post : posts) {
if (post.getId() == id) { posts.remove(post); return; }
}
throw new PostNotFoundException("No post with ID " + id);
}
public SetgetAllPosts() { return posts; }
}
Now your tests can swap out the real blog service with the fake and the code under test won’t even know the difference.
Fakes are useful for when you can’t use the real implementation in a test, such as if the real implementation is too slow (e.g. it takes several minutes to start up) or if it’s non-deterministic (e.g. it talks to an external machine that may not be available when your test runs).
You shouldn’t need to write your own fakes often since each fake should be created and maintained by the person or team that owns the real implementation. If you’re using an API that doesn’t provide a fake, it’s often easy to create one yourself: write a wrapper around the part of the code that you can’t use in your tests, and create a fake for that wrapper. Remember to create the fake at the lowest level possible (e.g. if you can’t use a database in your tests, fake out the database instead of faking out all of your classes that talk to the database), that way you’ll have fewer fakes to maintain, and your tests will be executing more real code for important parts of your system.
Fakes should have their own tests to ensure that they behave like the real implementation (e.g. if the real implementation throws an exception when given certain input, the fake implementation should also throw an exception when given the same input). One way to do this is to write tests against the API’s public interface, and run those tests against both the real and fake implementations.
If you still don’t fully trust that your code will work in production if all your tests use a fake, you can write a small number of integration tests to ensure that your code will work with the real implementation.
by Anthony Vallone
How long does it take to find the root cause of a failure in your system? Five minutes? Five days? If you answered close to five minutes, it’s very likely that your production system and tests have great logging. All too often, seemingly unessential features like logging, exception handling, and (dare I say it) testing are an implementation afterthought. Like exception handling and testing, you really need to have a strategy for logging in both your systems and your tests. Never underestimate the power of logging. With optimal logging, you can even eliminate the necessity for debuggers. Below are some guidelines that have been useful to me over the years.
Channeling Goldilocks
Never log too much. Massive, disk-quota burning logs are a clear indicator that little thought was put in to logging. If you log too much, you’ll need to devise complex approaches to minimize disk access, maintain log history, archive large quantities of data, and query these large sets of data. More importantly, you’ll make it very difficult to find valuable information in all the chatter.
The only thing worse than logging too much is logging too little. There are normally two main goals of logging: help with bug investigation and event confirmation. If your log can’t explain the cause of a bug or whether a certain transaction took place, you are logging too little.
Good things to log:
Bad things to log:
There is More Than One Level
Don’t log everything at the same log level. Most logging libraries offer several log levels, and you can enable certain levels at system startup. This provides a convenient control for log verbosity.
The classic levels are:
Practically speaking, only two log configurations are needed:
Test Logs Are Important Too
Log quality is equally important in test and production code. When a test fails, the log should clearly show whether the failure was a problem with the test or production system. If it doesn’t, then test logging is broken.
Test logs should always contain:
Conditional Verbosity With Temporary Log Queues
When errors occur, the log should contain a lot of detail. Unfortunately, detail that led to an error is often unavailable once the error is encountered. Also, if you’ve followed advice about not logging too much, your log records prior to the error record may not provide adequate detail. A good way to solve this problem is to create temporary, in-memory log queues. Throughout processing of a transaction, append verbose details about each step to the queue. If the transaction completes successfully, discard the queue and log a summary. If an error is encountered, log the content of the entire queue and the error. This technique is especially useful for test logging of system interactions.
Failures and Flakiness Are Opportunities
When production problems occur, you’ll obviously be focused on finding and correcting the problem, but you should also think about the logs. If you have a hard time determining the cause of an error, it’s a great opportunity to improve your logging. Before fixing the problem, fix your logging so that the logs clearly show the cause. If this problem ever happens again, it’ll be much easier to identify.
If you cannot reproduce the problem, or you have a flaky test, enhance the logs so that the problem can be tracked down when it happens again.
Using failures to improve logging should be used throughout the development process. While writing new code, try to refrain from using debuggers and only use the logs. Do the logs describe what is going on? If not, the logging is insufficient.
Might As Well Log Performance Data
Logged timing data can help debug performance issues. For example, it can be very difficult to determine the cause of a timeout in a large system, unless you can trace the time spent on every significant processing step. This can be easily accomplished by logging the start and finish times of calls that can take measurable time:
Following the Trail Through Many Threads and Processes
You should create unique identifiers for transactions that involve processing across many threads and/or processes. The initiator of the transaction should create the ID, and it should be passed to every component that performs work for the transaction. This ID should be logged by each component when logging information about the transaction. This makes it much easier to trace a specific transaction when many transactions are being processed concurrently.
Monitoring and Logging Complement Each Other
A production service should have both logging and monitoring. Monitoring provides a real-time statistical summary of the system state. It can alert you if a percentage of certain request types are failing, it is experiencing unusual traffic patterns, performance is degrading, or other anomalies occur. In some cases, this information alone will clue you to the cause of a problem. However, in most cases, a monitoring alert is simply a trigger for you to start an investigation. Monitoring shows the symptoms of problems. Logs provide details and state on individual transactions, so you can fully understand the cause of problems.
By Andrew Trenk
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.
When writing tests for your code, it can seem easy to ignore your code’s dependencies by mocking them out.
public void testCreditCardIsCharged() {
paymentProcessor = new PaymentProcessor(mockCreditCardServer);
when(mockCreditCardServer.isServerAvailable()).thenReturn(true);
when(mockCreditCardServer.beginTransaction()).thenReturn(mockTransactionManager);
when(mockTransactionManager.getTransaction()).thenReturn(transaction);
when(mockCreditCardServer.pay(transaction, creditCard, 500)).thenReturn(mockPayment);
when(mockPayment.isOverMaxBalance()).thenReturn(false);
paymentProcessor.processPayment(creditCard, Money.dollars(500));
verify(mockCreditCardServer).pay(transaction, creditCard, 500);
}
However, not using mocks can sometimes result in tests that are simpler and more useful.
public void testCreditCardIsCharged() {
paymentProcessor = new PaymentProcessor(creditCardServer);
paymentProcessor.processPayment(creditCard, Money.dollars(500));
assertEquals(500, creditCardServer.getMostRecentCharge(creditCard));
}
Overusing mocks can cause several problems:
- Tests can be harder to understand. Instead of just a straightforward usage of your code (e.g. pass in some values to the method under test and check the return result), you need to include extra code to tell the mocks how to behave. Having this extra code detracts from the actual intent of what you’re trying to test, and very often this code is hard to understand if you’re not familiar with the implementation of the production code.
- Tests can be harder to maintain. When you tell a mock how to behave, you’re leaking implementation details of your code into your test. When implementation details in your production code change, you’ll need to update your tests to reflect these changes. Tests should typically know little about the code’s implementation, and should focus on testing the code’s public interface.
- Tests can provide less assurance that your code is working properly. When you tell a mock how to behave, the only assurance you get with your tests is that your code will work if your mocks behave exactly like your real implementations. This can be very hard to guarantee, and the problem gets worse as your code changes over time, as the behavior of the real implementations is likely to get out of sync with your mocks.
Some signs that you’re overusing mocks are if you’re mocking out more than one or two classes, or if one of your mocks specifies how more than one or two methods should behave. If you’re trying to read a test that uses mocks and find yourself mentally stepping through the code being tested in order to understand the test, then you’re probably overusing mocks.
Sometimes you can’t use a real dependency in a test (e.g. if it’s too slow or talks over the network), but there may better options than using mocks, such as a hermetic local server (e.g. a credit card server that you start up on your machine specifically for the test) or a fake implementation (e.g. an in-memory credit card server).
For more information about using hermetic servers, see http://googletesting.blogspot.com/2012/10/hermetic-servers.html. Stay tuned for a future Testing on the Toilet episode about using fake implementations.
by The GTAC Committee
The Google Test Automation Conference (GTAC) was held last week in NYC on April 23rd & 24th. The theme for this year’s conference was focused on Mobile and Media. We were fortunate to have a cross section of attendees and presenters from industry and academia. This year’s talks focused on trends we are seeing in industry combined with compelling talks on tools and infrastructure that can have a direct impact on our products. We believe we achieved a conference that was focused for engineers by engineers. GTAC 2013 demonstrated that there is a strong trend toward the emergence of test engineering as a computer science discipline across companies and academia alike.
All of the slides, video recordings, and photos are now available on the GTAC site. Thank you to all the speakers and attendees who made this event spectacular. We are already looking forward to the next GTAC. If you have suggestions for next year’s location or theme, please comment on this post. To receive GTAC updates, subscribe to the Google Testing Blog.
Here are some responses to GTAC 2013:
“My first GTAC, and one of the best conferences of any kind I’ve ever been to. The talks were consistently great and the chance to interact with so many experts from all over the map was priceless.” – Gareth Bowles, Netflix
“Adding my own thanks as a speaker (and consumer of the material, I learned a lot from the other speakers) — this was amazingly well run, and had facilities that I’ve seen many larger conferences not provide. I got everything I wanted from attending and more!” – James Waldrop, Twitter
“This was a wonderful conference. I learned so much in two days and met some great people. Can’t wait to get back to Denver and use all this newly acquired knowledge!” – Crystal Preston-Watson, Ping Identity
“GTAC is hands down the smoothest conference/event I’ve attended. Well done to Google and all involved.” – Alister Scott, ThoughtWorks
“Thanks and compliments for an amazingly brain activity spurring event. I returned very inspired. First day back at work and the first thing I am doing is looking into improving our build automation and speed (1 min is too long. We are not building that much, groovy is dynamic).” – Irina Muchnik, Zynx Health
by The GTAC Committee
GTAC is just around the corner, and we’re all very busy and excited. I know we say this every year, but this is going to be the best GTAC ever! We have updated the GTAC site with important details:
If you are on the attendance list, we’ll see you on April 23rd. If not, check out the Live Stream page where you can watch the conference live and can get involved in Q&A after each talk. Perhaps your team can gather in a conference room and attend remotely.
by Anthony Vallone
We have two excellent, new videos to share about testing at Google. If you are curious about the work that our Test Engineers (TEs) and Software Engineers in Test (SETs) do, you’ll find both of these videos very interesting.
The Life at Google team produced a video series called Do Cool Things That Matter. This series includes a video from an SET and TE on the Maps team (Sean Jordan and Yvette Nameth) discussing their work on the Google Maps team.
Meet Yvette and Sean from the Google Maps Test Team
The Google Students team hosted a Hangouts On Air event with several Google SETs (Diego Salas, Karin Lundberg, Jonathan Velasquez, Chaitali Narla, and Dave Chen) discussing the SET role.
Software Engineers in Test at Google – Covering your (Code)Bases
Interested in joining the ranks of TEs or SETs at Google? Search for Google test jobs.
By Andrew Trenk
This article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.
There are typically two ways a unit test can verify that the code under test is working properly: by testing state or by testing interactions. What’s the difference between these?
Testing state means you’re verifying that the code under test returns the right results.
public void testSortNumbers() {
NumberSorter numberSorter = new NumberSorter(quicksort, bubbleSort);
// Verify that the returned list is sorted. It doesn't matter which sorting
// algorithm is used, as long as the right result is returned.
assertEquals(
new ArrayList(1, 2, 3),
numberSorter.sortNumbers(new ArrayList(3, 1, 2)));
}
Testing interactions means you’re verifying that the code under test calls certain methods properly.
public void testSortNumbers_quicksortIsUsed() {
// Pass in mocks to the class and call the method under test.
NumberSorter numberSorter = new NumberSorter(mockQuicksort, mockBubbleSort);
numberSorter.sortNumbers(new ArrayList(3, 1, 2));
// Verify that numberSorter.sortNumbers() used quicksort. The test should
// fail if mockQuicksort.sort() is never called or if it's called with the
// wrong arguments (e.g. if mockBubbleSort is used to sort the numbers).
verify(mockQuicksort).sort(new ArrayList(3, 1, 2));
}
The second test may result in good code coverage, but it doesn’t tell you whether sorting works properly, only that quicksort.sort() was called. Just because a test that uses interactions is passing doesn’t mean the code is working properly. This is why in most cases, you want to test state, not interactions.
In general, interactions should be tested when correctness doesn’t just depend on what the code’s output is, but also how the output is determined. In the above example, you would only want to test interactions in addition to testing state if it’s important that quicksort is used (e.g. the method would run too slowly with a different sorting algorithm), otherwise the test using interactions is unnecessary.
What are some other examples of cases where you want to test interactions?
- The code under test calls a method where differences in the number or order of calls would cause undesired behavior, such as side effects (e.g. you only want one email to be sent), latency (e.g. you only want a certain number of disk reads to occur) or multithreading issues (e.g. your code will deadlock if it calls some methods in the wrong order). Testing interactions ensures that your tests will fail if these methods aren’t called properly.
- You’re testing a UI where the rendering details of the UI are abstracted away from the UI logic (e.g. using MVC or MVP). In tests for your controller/presenter, you only care that a certain method of the view was called, not what was actually rendered, so you can test interactions with the view. Similarly, when testing the view, you can test interactions with the controller/presenter.
by The GTAC Committee
We have completed selection and confirmation of all speakers for GTAC 2013. You can find the detailed agenda at:
developers.google.com/gtac/2013/schedule
Thank you to all who submitted proposals! It was very hard to make selections from so many fantastic submissions.
If you were not extended an invitation, don’t forget that you can join us via YouTube live streaming. We’ll be setting up Google Moderator, so remote attendees can get involved in Q&A after each talk. Information about live streaming, Moderator, and other details will be posted on the GTAC site soon and announced here.