Google Data

This is a guest post from Jeffrey Rosen, of Wolfire Games.

Recently, Wolfire Games, joined by four other independent video game developers and two charities, put on a pretty unusual promotion: The Humble Indie Bundle. For twelve days, you could have gone to the promotional site and paid what you want (divided any way you choose among the developers, the EFF, and the Child's Play Charity) for a bundle of independent video games, all compatible with Mac, Windows, and Linux.

The reason why I've been invited to write this guest blog post is because the site was built on Google App Engine and, according to Google Analytics, it served about 3.4 million pageviews (1 million unique visitors) in the 12 days it was live. It grossed over $1.2 million. Depending on who you ask, this is quite a lot of traffic, and we exceeded the free App Engine quota. Google sent us a bill for a grand total of $71.56.

I've been a long time advocate of Google App Engine on the Wolfire Blog, even when traffic was tiny, for a number of reasons. I really like its ease of use, and the fact that Google engineers, including Guido himself (the creator of Python), are the ones making sure everything works at 4 AM, and Google's infrastructure is responsible for data durability and security. The pay-as-you-go price combined with a generous free quota is icing.

As an independent video game developer, it's pretty crazy to have these resources at my disposal. While I'm confident in my Python web coding ability, I am definitely not confident in being a Linux sys-admin and commiting to an expensive server that may or may not handle the load properly (especially scary when critical data is involved). This upfront risk and steep learning curve would probably have been the straw that broke the camels back and turned the bundle into vaporware.

How did App Engine perform?

According to Google Analytics we did about 3.4 million pageviews (1 million unique visitors) during the bundle period on the Wolfire site (heavily concentrated around the first and last days). The traffic came in huge surges from Reddit, Slashdot, Digg, and a plethora of gaming and tech sites, and people seemed to really enjoy pushing the server by refreshing the page to update the real-time statistics. App Engine performed as advertised: additional servers were seamlessly brought up as needed and shut down when appropriate.

One thing that was really neat about the bundle is that it had a bunch of real-time statistics about purchases along with a leaderboard of the top contributions. There were many interesting donations, for example, $3333.33, $1337, $327.67, and $313.37 and watching the average donation go up and down was really addictive. I think being able to refresh the page to see the sales coming in was a really big draw for the bundle. This feature was really easy to implement on App Engine using a sharding counter. I'm really looking forward to native comet support in App Engine so the next bundle can have this update completely live, without polling or refreshing. Memcache and the datastore is built for stuff like this, so it was very fast.

How was developing on App Engine?

One awesome thing about App Engine is how it handles the deployment for you transparently and seamlessly. I felt comfortable developing, testing, and ultimately deploying new features to the site, even while it was serving say, 100 visitors per second. App Engine handles all the details of seamlessly bringing down the servers and getting the new ones in place even under intense loads. Because of this, I was able to hack on new features to the site as necessary. For example, I expanded the statistics so that you could see the platform specific average payments in real-time. That feature was started, tested, and deployed on maybe the third day of the bundle. Similarly, I spontaneously added an unplanned sixth game (Samorost 2) to the bundle on the fifth day and deployed minor tweaks dozens of times throughout the promo.

One thing that the bundle made clear to me is that raw Python code is much faster than you might expect. After I finished building the prototype of the bundle, I was getting ready to heavily optimize everything: each hit to the Humble Bundle page executed a few hundred lines of my Python code, doing all sorts of calculations for the statistics and inserting that into a massive Django template. Initially I thought that there would be no way this would be scalable and I'd need to aggressively cache the whole page. However, when I looked at the estimated CPU usage (tip: if you look at the request headers of a page while logged in as an admin, App Engine gives you some estimates on how much it will charge you for 1000 requests and how long it will take) it was something like 0.00375 CPM and was executing in milliseconds. Despite what I had assumed to be inefficient, unoptimized code, the CPM was so ridiculously low and it was executing so fast, I decided not to even bother optimizing the backend further. The only thing that was taking any time at all were the few memcache calls to retrieve the real-time list of tweets, real-time raw sales data, and other interesting stats. In hindsight though, I could have easily shaved off 30 ms by combining all the memcache requests into a single RPC with memcache.get_multi.

Was there anything App Engine could improve?

While I am really satisfied with App Engine and credit its existence and my familiarity with it for the existence of the Humble Indie Bundle in the first place, there were a few gotchas:

Due to a scheduled maintenance period on the datastore, the bundle was down for about 20 minutes (we need to store metadata about people's orders before we forward them to PayPal, Amazon Payments, or Google Checkout). Normally a message pops up to inform people when downtime occurs is going on, but this didn't activate for some reason, so we had to handle a lot of email about a blank page when people were trying to place their orders. I understand that Google is addressing this though.

The features that I would like to ask for are mostly all "on deck" on the roadmap: SSL for custom domains, comet support, and longer background tasks in particular. I'd also like to see App Engine focus on utilizing Google's infrastructure by serving non-static requests geographically like "web acceleration" services from CDNs. There are some trade-offs due to the nature of the datastore, but I think it can be done.

Thanks for having me on the blog! People can find me on the Wolfire Blog where we talk about more technical and game design topics, and check out our upcoming game Overgrowth.

Posted by Jeffrey Rosen of Wolfire Games