Google Test Automation Conference (GTAC) 2010

Call for Attendance & Proposals

• Testing Applications on Mobile Devices (Doron Reuveni, uTest)
• JsTestDriver (Jeremie Lenfang-Engelmann, Misko Hevery, Google)
• Fighting Layout Bugs (Michael Tamm, optivo GmbH)
• Even better than the real thing - Lessons learned from testing GWT applications (Nicolas Wettstein, Google)
• Selenium: to 2.0 and Beyond! (Simon Stewart, Google)
• Automating Performance Test Data Collection and Reporting (David Burns, David Henderson, smartFOCUS DIGITAL)
• Achieving Web Test Automation with a Mixed-Skills Team (Mark Micallef, BBC Future Media and Technology)
• Score One for Quality! (Joshua Williams and Ross Smith, Microsoft)
• Automatic workarounds for web applications (Antonio Carzaniga, Alessandra Gorla, Nicolò Perino, Mauro Pezzè, University of Lugano )
• Precondition Satisfaction by Smart Object Selection in Random Testing (Yi Wei, Serge Gebhardt, ETH Zurich)

Testing for the Web is the theme of the 4th Google Test Automation Conference (GTAC), to be held in Zurich, October 21-22.

We are happy to announce that we are now accepting applications for attendance. The success of the conference depends on active participation of the attendees. Because the available spaces for the conference are limited, we ask each person to apply for attendance. Since we aim for a balanced audience of seasoned practitioners, students and academics, we ask the applicants to provide a brief background statement.


How to apply
Please visit http://www.gtac.biz/call-for-attendance to apply for a attendance.


Deadline
Please submit your application until August 28, 2009 at the latest.


Registration Fees
There are no registration fees. We will send out detailed registration instructions to each invited applicant. We will provide breakfast and lunch. There will be a reception on the evening of October 21.


Cancellation
If you applied but can no longer attend the conference please notify us
immediately by sending an email to gtac-2009-cfa@google.com so
someone from the waiting list can get the opportunity instead.


Further information
General website: http://www.gtac.biz/
Call for proposals: http://www.gtac.biz/call-for-proposals
Call for attendance: http://www.gtac.biz/call-for-attendance
Accommodations: http://www.gtac.biz/accomodations

We get questions once in a while about our readership. Here's a brief summary of the last 30 days...

Page view count: 34,140
Time on each page: 2:52
Most popular day to read: Tuesday's.
Most traffic (top 5 in order): US, India, UK, Germany, Canada
Number of countries with readers: 131

Most popular posts...

1. /2009/07/why-are-we-embarrassed-to-admit-that-we (2,227)
2. /2009/06/7-plagues-of-software-testing (2,151)
3. /2009/07/software-testing-categorization
4. /2009/07/advantages-of-unit-testing
5. /2009/06/plague-of-repetitiveness
6. /2009/07/old-habits-die-hard
7. /2009/07/separation-anxiety
8. /2009/06/gtac-call-for-proposals
9. /2009/07/plague-of-amnesia

GTAC 2009: Testing for the web

The 4th Google Test Automation Conference brings together a selected set of industry practitioners around the topic of software testing and automation. This annual conference provides a forum for presentations and connects professionals with each other. To increase outreach, presentations are published online for everybody to see.

This years theme is Testing for the Web, topics may include:

• Testing the UI of modern web applications (HTML5, Ajax)
• Testing applications on mobile devices
• Testing in the cloud
• Web testing tools (Selenium, Webdriver and co)
• Testing distributed asynchronous applications
• Testing for web browser compatibility
• Testing large storage systems
• Load and performance testing
• Finding and reproducing failures that matter
• It seemed like a good idea (things you expected to work, but that didn't)
  

Presentations are targeted at experienced engineers actively working on problems of quality, test automation and techniques, but also include students and academics. We encourage innovative ideas, controversial experiences, problems, and solutions that further the discussion of software engineering and testing. Presentations are 45 min in length and speakers should be prepared for an active question and answer session following their presentation. While ideas are good, ideas refined by experience are even more interesting to participants at GTAC.

The conference is a two day event comprised of a single track of talks. Our philosophy is to engage a small set of active participants who all experience the same topics carrying the discussions into lightning talks, speaker Q&A, and topical discussion groups. Each year we have worked to identify a location that has a unique profile of technology professionals. This year the conference will be held at the Google office in Zurich, Switzerland on October 21 and 22, 2009.


Submission of Proposals
Please email a detailed and extended abstract (one page at most) to gtac-2009-cfp@google.com. Your submission must include the name of topic, author(s), affiliation, and an outline of the proposed talk. We strongly recommend you to also submit one or two highlight slides of the talk. Submit your proposal before August 1, 2009. We will acknowledge reception within one business day. Where employer or disclosure authorization is needed, authors need to obtain it prior to submitting. The program committee will evaluate proposals based on quality and relevance. All submissions will be held confidentially prior to contacting the selected presenters.

Notification of Acceptance
Notification of acceptance will be sent out on or before August 8, 2009. Authors of accepted proposals will present at the conference and their talk will be made available to the public on YouTube.

Copyright
GTAC requires authors to present at the conference and permit their presentation to be made available on YouTube.

Attendees
To ensure active participation and provide a variety of technical perspectives, we select applying attendees. Further information will be published via a call for participation at a later time.

Important Dates
August 1 - Deadline for presentation proposals
August 8 - Notification of acceptance
October 21+22 - GTAC conference (Zurich, Switzerland)

Questions
If you have questions regarding the submission process or potential topics please email us at: gtac-2009@google.com

We will add more information to the Google Testing Blog as we get closer to the dates.

• Is this email spam?
• Is this search result relevant?
• What product category does that query belong to?
• What is the ad that users are most likely to click on for the query “flowers”?
• Is this click fraudulent?
• Is this ad likely to result in a purchase (not merely a click)?
• Is this image pornographic?
• Does this page contain malware? –Should this query bring up a maps onebox?

z = x0w0 + x1w1 + x2w2 + ... + xkwk

The logistic function has nice characteristics since it can take any input, and basically squash it to 0 or 1. Ideal for predicting probabilities on events that are dependent on multiple factors(X_i) each with different importance weights(W_i). The Stochastic Gradient Descent provides fast convergence to find the optimal minimums of the error(E) that the function is making on the prediction as well as if there are multiple local minimums the algorithms guarantees converging to the global minimum of the prediction error. So let’s go back now into the real online world where we want to give answers (predictions) to our users in milliseconds and ask the question how are we going to design automated tests for the Stochastic Gradient Descent Algorithm embedded into a live online prediction system. The environment is pretty agile and dynamic, the code is being changed every hour, you want your tests to run on 24/7 basis, you want to detect errors upstream in the development process, but you don’t want to block the development process with tests that are running days, on the other side you want to release new features fast, but the release process has to be error prone(imagine the world with google being down for 5 mins, that is a global catastrophe, isn’t it?!

So let’s look at some of the test strategies:

Should we try to train the model(set of the importance factors) and test the model with the subset of the training data? What if this takes far more than hours, maybe days to do that? Should we try to reduce the set of importance factors (X_i) and get the convergence(E->0) on the reduced model?

Should we try to reduce the training data set(the variety of set of values for X as an input to the algorithm) and keep the original model and get the convergence by any price? Should we be happy with reducing both the model size and the training set? Are we going to worry for over-fitting in the test environment? Given the original data is online data and evolves fast, are we going to be satisfied with fixed data test set or change the input test data frequently? What are the triggers that will make you do so? What else should we do?

Drop us a note, all ideas are more than welcome.

• Deadline for applying as a participant: August 15
• Notification of acceptance: August 29
• Wait list callbacks: October 3
• GTAC 2008 Conference: October 23 and 24

• Atom Publishing Protocol, Testing a Server Implementation by David Calavera
• JInjector: a Coverage and End-To-End Testing Framework for J2ME and RIM by Julian Harty, Olivier Gaillard, and Michael Sama
• Advances in Automated Software Testing Technologies by Elfriede Dustin
• Taming the Beast: How to test an AJAX Application by Markus Clermont and John Thomas
• Automated Model-Based Testing of Web Applications by Oluwaseun Akinmade and Prof. Atif M Memon
• Boosting Your Testing Productivity with Groovy by Andres Almiray
• Practicing Testability in the Real World by Vishal Chowdhary
• The New Genomics: Software Development at Petabyte Scale by Matt Wood
• The Value of Small Tests by Christopher Semturs
• Deployment and Test Automation: Extending the Notion of 'Done' to 'System Tested on a Production Deployment' by Marc-Elian Begin

• Unit Testing as the name implies asks you to test a Class (Unit) in isolation.


• If your code mixes Object Construction with Logic you will never be able to achieve isolation.


• In order to unit-test you need to separate object graph construction from the application logic into two different classes


• The end goal is to have either: classes with logic OR classes with "new" operators.

The “Gamer Generation” is here. According to the Washington Post, in 2006 at least 40% of Americans play video games on a computer or a console. The largest demographic is 18-34 year-olds, many of whom are working as our friends and peers as software developers and testers. There is a great book, Got Game, which describes the impact of games on the workforce of today and in the future.

Many businesses are wondering these days how to use simple concepts popular in video games and apply them to work.

It’s happening already. From a construction site with a sign posted with “days since last accident” to the car dealer with salespersons leader board scrawled out in chalk. Simple competition helps work get done in all kinds of industries. The success of sales contests is a great example of how well competition works.

“In every job that must be done, there is an element of fun. You find the fun and—SNAP—the job’s a game!” —Mary Poppins

A company called Seriosity has made interesting use of video game concepts at work with their application of virtual currency to email. To help with information overload from an overflowing inbox, email senders can “spend” virtual currency (Serios) to help prioritize their messages. Spending to denote the importance of a message is not a whole lot different than buying new golf clubs with virtual money earned in a Tiger Woods game, buying new tires in a car racing game – or using gamer points to upgrade your armor in an RPG (Role-playing Game). Wired Magazine had a great article on the use of WOW as a leadership training game. Playing games at work is more than just a quick game of online Hearts with one finger on the “boss key”. Games at work can have real impact.

With other industries using productivity games, what about software development? How can productivity games help testers improve the quality of the products they test? One of the more famous test techniques is the bug bash. Teams pick a certain date, and everyone bangs on a certain area of the product to find bugs. At the end, prizes are awarded while everyone enjoys pizza and drinks. That sounds a lot like game, doesn't it? It’s also a perfect example of a simple productivity game.

As the world of software gets more complex and the test matrix grows, there is no shortage of opportunities for testers to do more testing. Quality is always “improved”, it’s never completed, fixed or finished. Therefore, testers must choose from a variety of techniques and activities to pursue tasks with the highest return and value. In 1990, Harry Markowitz won the Nobel Prize for his work on Portfolio Selection (that he began in the 1950’s). The theory, familiar in financial markets and mutual funds, is that a portfolio of diverse assets with varying risk and returns actually reduces risk in time uncertain conditions. Testers can benefit from the work on portfolio selection by varying the techniques they use to find defects. However, some techniques are more expensive, more difficult, or less attractive than others. Some teams, in some situations, can rely on the altruism and volunteer efforts – organizational citizenship behaviors – of their members to take on the more difficult tasks. However, that doesn't always work, and other incentives may be required. That’s where productivity games can help.

When a project needs a bit of a push towards a certain behavior, building a game around it can help. For example, if a test team wants to improve the usage of unit tests by developers, encourage the team to “eat their own dogfood” or run automated tests on personal machines overnight, then creating a game will help people’s motivation can motivate people to participate in those activities.

Productivity games differ from traditional games in a few distinct ways. The goal of a productivity game is to attract players. These games don’t need winners as much as they need players. In many cases, productivity games don’t even need prizes. Productivity games can be simple. There’s no need to invest in expensive graphics or 3-D modeling – people play the game to compete around a work activity, not because of the appeal of the user interface. A simple leader board might be all that’s needed.

Using games at work is a powerful and effective method to influence change in organizational behavior, and therefore requires care in the design and use. It is possible to overdo the use of games to a point where they are counter-productive or ineffective. Successful game design is best achieved through experience and experimentation, and the goal should be to keep things interesting enough to always attract players. A game where one player leaps out to an insurmountable lead is not effective, because that will discourage others from playing. When a team wants to encourage testers to invest in fault injection techniques, setting up a leader board with bug totals found via fault injection techniques will attract attention of other testers, often just by word-of-mouth. You should encourage team-wide competition, smack talk, and public embarrassment as a means to draw attention. Games should be short in duration. Many testers remember the Dilbert “I’m gonna write me a new minivan” from 11/13/95. This is what results from a poorly designed productivity game.

Productivity games are great at motivating people to do things and change behavior. And as Uncle Ben said to Spiderman, “With great power comes great responsibility”. It’s critical to have specific goals for the games, and to understand the impact before deploying a game – it’s pretty likely that people will play, at least initially, and the impact may not be what you intended. Keep the duration of games short in order to be able to adjust. And finally, keep the games focused on volunteer or “extra” activities. A game designed around an individual’s job or portions of everyone’s regular job can introduce unusual feelings when it comes to rewards and performance evaluation. It’s easier to steer clear of that than to try to figure out the right balance.

Shigeru Miyamoto, who designed Super Mario Bros., talks about games as “miniature gardens”, metaphorically representing the cultural values, humanity, and challenges of everyday life the way a miniature garden might represent a real one.

The whole idea, for me, is about the exploration of a hakoniwa, a miniature garden. It's like a garden in a box, where if you look at it from different angles, you can see different plants and arrangements. And you have that sense of surprise and exploration. There's always things you can find. [2]

Games are a lot like testing. Software quality assurance and testing efforts represent a proxy for real users in much the same way that games are a proxy for the real world. The challenges that Mario faces with fire-breathing dragons and knife throwing turtles are balanced by the allure of coins and princesses. A testing organization has a good idea of the tools available in their portfolio to improve quality and eliminate those “fire-breathing dragons” for their customers. Productivity games allow teams to offer coins and position rewards around their equivalents of Markowitz’s “diverse assets” at their disposal, in the form of defect detection and removal techniques, to improve quality.

For the players, productivity games offer people the ability to learn, solve problems, and earn a reward. They offer a way to challenge players to compete, to explore and discover, and to establish a surrogate identity or status via a leader board. There are many fundamental motivations to play a game. “If you build it, they will come [3]." A good productivity game designer can build a game around “work that needs to be done that no one wants to do” – and like the lone teenager who plays Halo for weeks on end, people will take on that work. Experiment and explore – the results can be surprising.

I’d be interested in any comments about Productivity Games that you might have.

Thanks,

Ross Smith

http://www.defectprevention.org/

Interesting Links:

When Work becomes a Game

Serious Games Initiative

Got Game - How the Gamer Generation is Reshaping Business Forever

Chapter 5 – Productivity Games - The Practical Guide to Defect Prevention

2008 Serious Games Summit presentation

Miniature Gardens and Magic Crayons: Games, Spaces, & Worlds

HBR Leadership’s Online Labs

NPR: Software lets users assign value to email

Game Tycoon blog (How Video Games are transforming the Business World)

Productivity Games Blog

Special thanks to Joshua Williams for his help with this post.

1. Explore the system's functionality
2. Identify the system's architecture
3. Identify the interfaces between components
4. Identify dependencies and fault conditions
5. For each function:

• Identify the participating components
• Identify potential problems
• Test in isolation for problems
• Create a 'happy path' test

• Unit test: is the method fulfilling its contract?
• Small integration test: Can two classes interact with each other?
• Medium integration test: Is a class interacting properly with all its dependencies? Does it anticipate and handle errors correctly? Are the needed functions exposed on an API/GUI?
• Sub-system test: Can two sub-systems interact with each other? Does one of them anticipate all errors of the other and does it deal with them appropriately?
  
• System test: Does the entire system behave as expected?
  

Testing Google Talk is challenging -- we have multiple client implementations, between the Google Talk client, the Google Talk Gadget, and Gmail chat, while also managing new features and development. We rely heavily on automation. Yet there's still a need to do manual testing before the release of the product to the public.

We've found that one of the best ways to unearth interesting bugs in the product is to use Exploratory Testing (http://www.satisfice.com/articles/et-article.pdf) The trouble with ET is that while there appears to be a genetic disposition to be naturally good at exploring the product effectively, it's very easy to miss great swathes of the product when one follows their intuition through the product rather than focusing on coverage metrics. And speaking of coverage, how do we measure how well a team is doing finding bugs and getting coverage over the functional use cases for the product? All of the things that we rely on to measure the quality of the product, boundary and edge cases being covered? Plus, if not everyone is proficient at ET, how do we solve the overhead of having an experienced team member looking over people's shoulders to make sure they are executing well?

To do this, we start with the definition of a Test Strategy. This is where we outline the approach we are taking to the testing of the product as a whole. It's not super-detailed -- instead it mentions the overarching areas that need to be tested, whether automation can be used to test the area, and what role manual testing needs to play. This information lets developers and PMs know what we think we need to test for the product, and allows them to add unit tests etc to cover more ground.

Some basic test case definition go into the Test Plan. The aim of the test plan (and any test artifacts generated) is not to specify a set of actions to be followed in a rote manner, but instead a rough guide that encourages creative exploration. The test plan also acts as the virtual test expert, providing some framework under which exploratory testing can be executed effectively by the team. The plan decomposes the application into different areas of responsibility, that are doled out to members of the team in sessions that are one-working-day or less duration. By guiding people's thinking, we can cover the basics, fuzzy cases, and avoids a free-for-all, duplication, and missed areas.

Finally we get a status report from the testers every day, that describes the testing that was performed that day, any bugs raised, and blocking issues identified. The reports acts as an execution of the "contract" and gives traceability, and the ability to tweak exploratory testing that has gone off track from where we've determined we need to concentrate efforts. We can use these status reports along with bug statistics to gauge the effectiveness of the test sessions.

This is approach is fairly simple, but sometimes simple works best. Using this method has allowed us to make the best use of test engineers and maximized the effectiveness of each test pass. It's proven itself to be a fruitful approach and balances the need for reporting and accountability with the agility of exploratory testing.

There are side benefits to well-tested code as well such as enforcing good design practices such as decomposition, encapsulation, etc. but these are secondary to avoiding bugs.

Short examples of highly-beneficial tests are hard to come by, however counter-examples abound. The following examples have been anonymized but were found at various time in our code tree. Take this test:

def testMyModule(self):
mymodule.main()

Although it probably creates a lot of coverage in mymodule, this test will only fail if main throws an exception. Certainly this is a useful condition to detect but it is wasteful to consume the time of a full run of mymodule without verifying its output. Let's look at another low-value test:

def testFooInitialization(self):
try:
foo = Foo()
self.assertEquals(foo.name, 'foo')
self.assertEquals(foo.bar, 'bak')
except:
pass

This one probably hits the bottom of the value scale for a test. Although it exercises Foo's constructor, catching all exceptions means that the test will never fail. It creates coverage but never catches any bugs.

A fellow bottom-dweller of the value scale is the test that doesn't get fixed. This can be a broken test in a continuous build or a manual test that generates a bug that stays open: either way it's a waste of time. If it's an automated test, it is worth deleting and was probably not worth writing in the first place. If it's a manual test, it's probably a sign that QA is not testing what the PM cares about. Some even apply the broken-window principle to these tests saying that tests that don't get fixed give the impression that testing is not valuable.

Our final specimen is slightly higher value but still not very high. Consider the function Bar:

def Bar():
SlowBarHelper1()
SlowBarHelper2()
SlowBarHelper3()

We could employ stubs or mocks to write a quick unit test of Bar but all we could assert is that the three helpers got called in the right order. Hardly a very insightful test. In non-compiled languages, this kind of test does serve as a substitute syntax-checker but provides little value beyond that.

Let's now turn our attention to the dark side of testing: its cost. The budget of a whole testing team is easy to understand but what about the cost of an individual test?

The first such cost is the one-time cost creating the test. Whether it is the time it takes to write down the steps to reproduce a manual test or the time it takes to code an automated test it is mostly dependent on the testability of the system or the code. Keeping this cost down is an essential part of test-driven development: think about your tests before you start coding.

While the creation of a test has a significant cost, it can be dwarfed by the incremental cost of running it. This is the most common objection to manual testing since the salary of the tester must be paid with every run of the test but it applies to automated tests too. An automated test uses a machine while it's running, that machine and it's maintenance both have a cost. If a test requires specialized hardware to run, those costs go up. Similarly, adding a test that takes 20 minutes to run will consume 20 minutes of the time of each engineer that tries to run it, every time s/he tries to run it! If it's a test that's run before each check-in the cost of that test will go up rapidly. It could be worth the engineering time to reduce its run time to a more reasonable level.

There is one more incremental cost to a test: the cost of its failure. Whenever a test fails, time is spent to diagnose the failure. The reduction of this cost is the reason behind two key principles of good testing:
- don't write flaky tests: flaky tests waste time by making us investigate failures that are not really there
- write self-diagnosing tests: a test should make it clear what went wrong when it fails to allow us to rapidly move towards a fix

The 'economics' of testing can be used to analyze various testing methodologies. For example, true unit tests (small, isolated tests) take one approach to this problem: they minimize the repeated costs (by being cheap to run and easy to diagnose) while incurring a slightly higher creation cost (have to mock/stub, refactor, ...) and slightly lesser benefits (confidence about a small pieces of the system as opposed to the overall system). By contrast, regression tests tend to incur a greater cost (since most regression tests are large tests) but attempt to maximize their benefits by targeting areas of previous failures under the assumption that those are most likely to have bugs in the future.

So think about both the benefits and the costs of each test that you write. Weigh the one-time costs against the repeated costs that you and your team will incur and make sure that you get the benefits that you want at the least possible cost.

For those German-speaking folks among our readers of this English Google Testing Blog we have exciting news: We have just launched the German Testing Blog!

This is a tribute to the fact that the German-speaking software test community is one of the biggest non-english audiences of this blog. The german blog will contain a mix of German versions of postings from this blog as well as unique postings about regional-specific issues from Europe. Some of our biggest engineering offices outside of US are in Europe, e.g. in Switzerland and the UK.

So German speakers around the world check out this new resource for interesting discussions about software test, performance test, load test, and how to create rock-solid software.

It took eighteen months for Julian Harty to overcome the various challenges of testing mobile wireless applications. In turn, he has learned some valuable lessons that he wants to share with you in this week's column.

In a previous episode, we extracted methods to simplify testing in Python. But if these extracted methods make the most sense as private class members, how can you write your production code so it doesn't depend on your test code? In Python this is easy; but in C++, testing private members requires more friend contortions than a game of Twister®.

// my_package/dashboard.h
class Dashboard {
 private:
  scoped_ptr<Database> database_;  // instantiated in constructor

  // Declaration of functions GetResults(), GetResultsFromCache(),
  // GetResultsFromDatabase(), CountPassFail()

  friend class DashboardTest; // one friend declaration per test
                              // fixture
};

You can apply the Extract Class and Extract Interface refactorings to create a new helper class containing the implementation. Forward declare the new interface in the .h of the original class, and have the original class hold a pointer to the interface. (This is similar to the Pimpl idiom.) You can distinguish between the public API and the implementation details by separating the headers into different subdirectories (/my_package/public/ and /my_package/ in this example):

// my_package/public/dashboard.h
class ResultsLog;  // extracted helper interface
class Dashboard {
 public:
  explicit Dashboard(ResultsLog* results) : results_(results) { }
 private:
  scoped_ptr<ResultsLog> results_;
};

// my_package/results_log.h
class ResultsLog {
 public:
  // Declaration of functions GetResults(),
  // GetResultsFromCache(),
  // GetResultsFromDatabase(), CountPassFail()
};

// my_package/live_results_log.h
class LiveResultsLog : public ResultsLog {
 public:
  explicit LiveResultsLog(Database* database)
      : database_(database) { }
};

Now you can test LiveResultsLog without resorting to friend declarations. This also enables you to inject MockResultsLog instance when testing the Dashboard class. The functionality is still private to the original class, and the use of a helper class results in smaller classes with better-defined responsibilities.

Remember to download this episode of Testing on the Toilet and post it in your office.

I’d like to offer the readers a chance to post their ideas on this blog.

To keep this simple I only have 3 rules:

1. no commercial postings or links to commercial sites,
   
2. post need to be interesting and practical ideas relating to testing, and
   
3. posts need to be proof read and reviewed on your end before submitting.
   

I’ll read the potential posts and make a decision within 1-2 weeks about posting it. Remember that this is a popular blog, the content is public, and your name will be on the posts. Make sure you don’t reveal proprietary information. If you are interested please post a comment to this entry. If enough people are interested, I’ll post later with an email address for the submissions.

• Scripting your manual tests this way takes far longer than just executing them manually.
• The UI is one of the least stable interfaces of any system, so we can start automating quite late in the development phase.
• Maintenance of the tests takes a significant amount of time.
• Execution is slow, and sometimes cumbersome.
• Tests become flaky.
• Tests break for the wrong reasons.

• It takes long to automate a test—Well, let's automate only tests that are important, and will be executed again and again in regression testing.
• Execution might be slow, but it is still faster than manual testing.
• Tests cannot break for the wrong reason—When they break we found a bug.

• Is there a database? If so, the verification should probably not be performed against the UI but against the database.
• Do we need to interface with a supplier? If so, how should this interaction look?
• Is the same functionality available via an API? If so, it should be tested through the API, and the UI should just be checked to interact with the API correctly.

• write many more tests through the API, e.g., to cover many boundary conditions,
• execute multiple threads of tests on the same machine, giving us a chance to spot race-conditions,
• start earlier with testing the system, as we can test each interface when it becomes 'quasi-stable',
• makes maintenance of tests and debugging easier, as the tests break closer to the source of the problem,
• require fewer machine resources, and still execute in reasonable time.

• start early in the development cycle,
• take the internal structure of the system into account,
• have a feedback loop to developers to influence the system-design.

• removing any spatial separation between the test engineers and the development engineers. Sitting on the next desk is probably the best way to promote information exchange,
• using the same tools and methods as the developers,
• getting involved into daily stand-ups and design-discussions.

• to take the internal details and exposed interface of the system under test into account,
• to have many fast tests for each interface (including the UI),
• to verify the functionality at the lowest possible level,
  
• to have a set of end-to-end tests,
• to start at the same time as development,
• to overcome traditional boundaries between development and testing (spatial, organizational and process boundaries), and
  
• to use the same tools as the development team.

• Speed - does the system respond quickly enough
• Capacity - is the infrastructure sized adequately
• Scalability - can the system grow to handle future volumes
• Stability - does the system behave correctly under load

So, I do performance testing of a service when risk analysis indicates that failing in any of the above categories would be more costly to the company than performing the tests. (Which, if your name is Google and you care about your brand, happens with any service you launch.) Note that I am talking about services - I work almost exclusively with servers and spend no time worrying about client-side rendering/processing issues. While those are becoming increasingly more important, and have always been more complex than my work, I consider those to be a part of functionality tests, and they are designed, created and executed by functional testing teams.

Another interesting thing about performance testing is that you will never be able to be 100% "right" or 100% "done. Accept it, deal with it, and move on. Any system in existence today will depend on thousands of different parameters, and if I spent the time analyzing each one of them, understanding the relationships between each two or each three, graphing their impact curves, trying to non-dimensionalize them, I would still be testing my first service two years later. The thought of doing anything less filled me with horror (They cannot seriously expect me to provide meaningful performance results in less than a year, can they?) but I have since learned that I can provide at least 90% of meaningful information to my customers by applying only 10% of my total effort and time. And, 90% is more than enough for vast majority of problems.

So, here is what I really do - I create benchmarks. If I am lucky and have fantastic information about current usage patterns of a particular product (which I usually do), I will make sure this benchmark covers most operations that are top resource hogs (either per single use or cumulative). I'll run this benchmark with different loads (number of virtual users) against a loosely controlled system (it would be nice to have 100 machines all to myself for every service we have, which I can use once a day or once a week, but that would be expensive and unrealistic) and investigate its behavior. Which transactions are taking the most time? Which transactions seem to get progressively worse with increasing load? Which transactions seem unstable (I cannot explain their behavior)? I call this exploratory performance testing, and I'll repeat my tests until I am convinced I am observing real system behavior. While I am doing this, I make sure I am not getting biased by investigating the code. If I have questions, I ask programmers, but I know they are biased, and I will avoid getting biased myself!

Once I have my graphs (think, interesting transaction latencies and throughput vs. load here) I meet with the development team and discuss the findings. Usually, there is one or two things they know and have been working on, and a few more they were unaware of. Sometimes, they look over my benchmark and suggest changes (could you make the ratio 80:20, and not 50:50?) After this meeting, we create our final benchmark, I modify the performance testing scripts, and now this benchmark will run as often as possible, but hopefully at least once a night. And, here is the biggest value of this effort: if there is a code change that has impacted performance in an unacceptable way, you will find out about it the next day. Not a week or a month later (How many of us remember what we did in the last month? So, why expect our developers to do so?)

Here is why I think this is the right thing to do: I have seen more bad code developed as a result of premature performance optimizations - before the team even thought they had a problem! Please don't do that. Develop your service in a clean, maintainable and extensible manner. Let me test it, and keep regression testing it. If we find we have a problem in a particular area, we can then address that problem easily - because our code is not obfuscated with performance optimization that have improved code paths that execute once a month by 5%.

I can usually do this in two - four weeks depending on the complexity of the project. Occasionally, we will find an issue that cannot be explained or understood with performance tests. At that point in time, we look under the hood. This is where performance profiling and performance modeling come in. And, both of those are considerably more complex than performance testing. Both great tools, but should be used only when the easy tool fails.

Tools, tools, tools... So, what do we use? I gave a presentation at Google Test Automation Conference in London on exactly this topic. I use open source tools. I discuss the reasons why in the presentation. In general, even if you have decided to go one of the other two routes (vendor tools or develop your own) check out what is available. You may find out that you will get a lot of information about your service using JMeter and spending some time playing around with it. Sure, you can also spend $500K and get similar information or you can spend two years developing "the next best performance testing tool ever," but before you are certain free is not good enough, why would you want to?

Final word: monitor your services during performance tests. If you do not have service related monitoring developed and set up to be used during live operations, you do not need performance testing. If the risks of your service failing are not important enough that you would want to know about it *before* it happens, then you should not be wasting time or money on performance testing. I am incredibly lucky in this area - Google infrastructure is developed by a bunch of people who, if they had a meeting where the topic would be "How to make Goranka's life easy?", could not have done better. I love them - they make my job trivial. At a minimum, I monitor CPU, memory and I/O usage. I cannot see a case when you would want to do less, but you may want to do a lot more on occasion.