Taming the Beast (a.k.a. how to test AJAX applications) : Part 2
August 27th, 2008 | Published in Google Testing
Posted by John Thomas and Markus Clermont
This is the second of a two part blog series titled 'Taming the Beast : How to test AJAX applications'. In part one we discussed some philosophies around web application testing. In this part we walk through a real example of designing a test strategy for an AJAX application by going 'beyond the GUI'.
Application under test
The sample application we want to test is a simple inventory management system that allows users to increase or decrease the number of parts at various store locations. The application is built using GWT (Google Web Toolkit) but the testing methodology described here could be applied to any AJAX application.
To quickly recap from part one, here's our recipe for testing goodness:
- Explore the system's functionality
- Identify the system's architecture
- Identify the interfaces between components
- Identify dependencies and fault conditions
- For each function
- Identify the participating components
- Identify potential problems
- Test in isolation for problems
- Create a 'happy path' test
1. Explore the system's functionality
2. Identify the architecture
Learning about the system architecture is the next critical step. At this point think of the system as a set of components and figure out how they talk to each other. Design documents and architecture diagrams are helpful in this step. In our example we have the following components:
- GWT client: Java code compiled into JavaScript that lives in the users browser. Communicates with the server via HTTP-RPC
- Servlet: standard Apache Tomcat servlet that serves the "frontend.html" (main page) with the injected JavaScript and also serves RPCs to communicate with the client-side JavaScript.
- Server-side implementation of the RPC-Stubs: The servlet dispatches the RPC over HTTP calls to this implementation. The RPCImpl communicates with the RPC-Backend via protocol-buffers over RPC
- RPC backend: deals with the business logic and data storage.
- Bigtable: for storing data
In our sample application, the RPC-Implementation is called "StoreService" and the other RPC-Backend is called "OfficeBackend".
3. Identify the interfaces between components
Some obvious ones are:
- gwt_module target in Ant build file
- "service" servlet of Apache Tomcat
- definition of the RPC-Interface
- Protocol buffers
- Bigtable
- UI (it is an interface, after all!)
4. Identify dependencies and fault conditions
With the interfaces correctly identified, we need to identify dependencies and figure out input values that are needed to simulate error conditions in the system.
In our case the UI talks to the servlet which in turn talks to StoreService (RPCImpl). We should verify what happens when the StoreService:
- returns null
- returns empty lists
- returns huge lists
- returns lists with malformed content (wrongly encoded, null or long strings)
- times out
- gets two concurrent calls
- returns malformed content
- times out
- sends two concurrent requests
- throws exceptions
To get a better overview, we will first look at individual use-cases, and see how the components interact. An example would be the filter-function in the UI (only those items under a 'checked' in a checked-location in the NavPane will be displayed in the table).
Analyze the NavPane filter
- Client
- Gets all offices from RPC
- On select, fetch items with RPC. On completion, update table.
- On deselect, clear items from table.
- RPCImpl
- Gets all offices from RPC-Backend
- Fetches all stock for an office from RPC-Backend
- RPC-Backend
- Scan bigtable for all offices
- Query stock for a given office from bigtable.
Test client-side behavior
Make sure that de-selecting an item removes it. For that, we need to be sure what items will be in the list. A fake RPCImpl could do just that - independent of other tests that might use the same datasource.
The task is to make the Servlet talk to the "MockStoreService" as RPCImpl. We have different possibilities to achieve that:
- Introduce a flag to switch
- Use the proxy-pattern
- Switch it at run time
- Add a different constructor to the servlet
- Introduce a different build-target that links to the fake implementation
- Use dependency injection to swap out real for fake implementations
There are various frameworks to allow this form of dependency injection. We want to briefly introduce GuiceBerry as one of them.
In our example we need to annotate the RPCImpl object with "@Inject" in the servlet test class and create an alternate implementation called MockStoreService to swap in at run time. Here's a code snippet that shows how:
@GuiceBerryEnv(StoreGuiceBerryEnvs.NORMAL)
public class StorePortalTest extends TestCase {
@Inject
StoreServiceImpl storeService;
public void testStorePortal() {}
...
storeService.doSomething();
...
}
In the code snippet above, note the lines marked in bold. That's Guiceberry magic that allows us to inject a StoreServiceImpl object into the StorePortalTest class. The construction of the StoreServiceImpl is done inside a Guiceberry environment class called NormalStoreGuiceBerryEnvs (and linked to StorePortal via the StoreGuiceBerryEnvs class). To inject a mock RPCImpl into StorePortalTest we would need to create a MockStoreGuiceBerryEnvs (which would instantiate a mock StoreService) and swap that for NormalStoreGuiceBerryEnvs at run time. All we need to do is to specify the following JVM args for the test ...
JVM_ARGS="-DNormalStoreGuiceBerryEnvs=MockStoreGuiceBerryEnvs"
This is just a quick peek at how Guiceberry works. Go to the official Guiceberry website to learn more.
This will be enough to decouple the client from the rest of the system. GwtTestCase does the rest of the job on the client side. You find more details here. Don't forget to inject all possible failure scenarios through the MockStoreService.
Let's see what we found out so far:
- We know that
-
- UI callbacks work correctly
- Interaction UI - Frontend works fine
- Expected errors are handled adequately by the UI
- We don't know whether
- things are rendered correctly
- things we expect to be on a page are really there
This is where some more traditional techniques, namely automated UI tests, enter the stage.
- Add JavaScript hooks into the page, that return the elements (JSNI is the way to go here)
- Use Selenium for UI tests (using both the hooks and the MockStoreService). All we need to do is check whether
-
- the elements exist
- all the buttons (which need to be clicked on) are clickable
- scrollbars are added when needed
One problem we have often had with Selenium tests in the past was that people relied overly on XPath queries to retrieve the elements from webpages. Of course, when the DOM changed it caused many tests to break. One way to work around that is to introduce JavaScript hooks. They are only added when the application runs with a special "testing" flag and they directly return the elements needed.
You might wonder why this approach is any better? Well for one, we can catch problems earlier, and fix them without even looking at the tests that use them. A small and fast JsUnit test can be used to determine whether a hook is broken. If so, it is only a line of code to fix the problem.
Let's review what we have found out so far:
- We know that
-
- UI callbacks work correctly
- Interaction UI - Frontend works fine
- Expected errors are handled adequately by the UI
- Things are rendered appropriately
- DOM is correct
- We don't know whether
-
- Other (non-UI) components work as expected
Test the StoreService (RPCImpl)
The methods in StoreService (RPCImpl) need a lot of good unit testing. If we write a good amount of unit tests, we probably already have a MockOfficeAdministration (RPC-Backend) that we can use for our further testing efforts.
The main value we can add here is to verify that (1) each interface method in the StoreService behaves correctly, even in the face of communication errors with the RPC-Backend and (2) each method behaves as expected. By using a MockOfficeAdministration as RPC-Backend, we don't have to worry about setting up the data (plus injecting faults is easy!)
Besides testing the basic functionality, e.g.
- Are all the records that we expect retrieved
- Are no records that shouldn't be retrieved passed on to the caller
- Does the application behave correctly, even if no records are found
- Malformed or Unexpected data
- Too much data
- Empty replies
- Exceptions
- Time-outs
- Concurrency problems
Some example test cases at this level are:
- Retrieve the list of all offices Let the mock-RPC-Backend
-
- return no office
- return 100 offices, 1 malformed encoded
- return 100 offices, 1 null
- ...
- throw an exception
- time out
- Retrieve product / stock for an office Let the mock-RPC-Backend stubby return
-
- ...
- Retrieve a product for an office Let the mock-RPC-Backend block, and
-
- issue a second query for the same product at the same time (and to make it more interesting, play with the results that the mock could return!).
- ....
- the UI works in isolation as expected
- the StoreService (RPCImpl) appropriately invokes the RPC-Backend-Service
- the StoreService (RPCImpl) properly handles any error-conditions
- A little bit about the app's behavior under concurrency
We don't know whether
Now let us verify the interaction between OfficeAdministration (RPC-Backend) and StoreService (RPCImpl). This is an essential task, and is not really that difficult. The following points should make testing this quick and easy:
- the RPC-Backend-Service really expects the behavior the StoreServiceImpl exposes.
- Backend correctly reads from Bigtable
- Business logic in the backend works correctly
- Backend knows how to handle error-conditions
- Backend knows how to handle missing data
- Backend is used correctly, i.e. in the way it is intended to be used
Now let us verify the interaction between OfficeAdministration (RPC-Backend) and StoreService (RPCImpl). This is an essential task, and is not really that difficult. The following points should make testing this quick and easy:
- Easy to test (through Java API)
- Easy to understand
- Ideally contains all the business logic
- Available early
- Executes fast (MockBigtable is an option here)
- Maintenance burden is low (because of stable interfaces)
- Potentially subset of tests as for StoreService (RPCImpl) alone
Let's see what we have found out so far: We know that
- the UI works in isolation as expected
- the OfficeAdministration (RPC-Backend) and the StoreService (RPCImpl) work together as expected
- The results find their way to the user
Last but not the least ... system test!
Now we need to plug all the components together and do the 'big' system test. In our case, a typical set up would be:
- Manipulate the "real" Bigtable and populate with "good" data for our test
-
- 5 offices, each with 5 products and each with a stock of 5
- Use Selenium (with the hooks) to
-
- Navigate via the Navbar
- Exclude an item
- Add an item
- ...
We now know that all components plugged together can handle one typical use case. We should repeat this test for each function that we can invoke through the UI.
The biggest advantage, however, is that we just need to look for communication issues between all 3 building blocks. We don't need to verify boundary cases, inject network errors, or other things (because we have already verified that earlier!)
Conclusion
Our approach requires that we
The biggest advantage, however, is that we just need to look for communication issues between all 3 building blocks. We don't need to verify boundary cases, inject network errors, or other things (because we have already verified that earlier!)
Conclusion
Our approach requires that we
- Understand the system
- Understand the platform
- Understand what can go wrong (and where)
- Start early with our tests
- Invest in infrastructure to run our tests (mocks, fakes, ...)
What we get in return is
- Faster test execution
- Less maintenance for the tests
-
- shared ownership
- early execution > early breakage > easy fix
- Shorter feedback loops
- Easier debugging / better localization of bugs due to fewer false negatives.