Google Data

This post is another entry in our on-going series of guest posts from App Engine developers. Today we have the Guardian reporting in on how they're using App Engine as part of their Open Platform.

There has been a lot of debate recently about the future of news online, some people like The Times of London and New York Times think paywalls are the answer, others are not so sure. At The Guardian we have a clear alternative and vision, we call it mutualisation. It's all about reducing the barriers between our journalists and readers and between our organisation and our partners. For software developers and online businesses we have a product sitting just in this space which we call the Open Platform.

The Open Platform has been in beta for the last year and now, today it is officially "Open for Business". Open for developers to take and co-monetize our content with us, building new applications for distributing and consuming news. It is also open for partners to build new experiences which blend content from The Guardian and their own functionality and in some cases brands into the heart of The Guardian's award winning website.

Today we're launching a brand new product and framework called MicroApps which the diagram above describes. However, just as Google dogfoods its new products before launch, so do we, and we wanted to share some of the things we've been building as MicroApps using Google AppEngine for the storage and application development part.

With 36 million unique and very engaged readers, everything we make has to scale, which is why AppEngine is ideal. With it's highly scalable architecture and features such as task queues, built for creating loosely coupled apps, and memcache, AppEngine makes an excellent companion platform for MicroApps in which the apps can run anywhere in the cloud. The examples presented here range from new ways to find content which others are finding exciting or interesting, to live responses to the TV debates to ways to bring together all of the tweets of our journalists on specific subjects.

Zeitgeist

Many people are interested in what is hot in the news and what is trending. We think this is interesting, but something a bit more interesting is in the things which are unexpectedly popular. Dan Catt used AppEngine to pull together signals from sources such as OneRiot, Twitter, comments in our system and our own internal stats to create an index of Zeitgeisty-ness. For the full "science bit" please see this overview article. We're using AppEngine for the storage and processing of a lot of data, with several task queues in play to create a page which responds over time during the current day showing you what people's attention think is the most interesting content. At the end of the day these are all archived and available so you can see what the Zeitgeist was on a particular day. The MicroApp framework allows us to host all of the application on AppEngine and display it in the network pages as if it were coming out of the content management system. This allows us to build data heavy applications using the economy of scale of cloud computing platforms while not affecting the scalability of the core site. If you want to see exactly how we made Zeitgeist, Dan talks about it here.

Election Debate Live Poll

There was recently a general election in the UK and for the first time in the UK there were a series of debates about the party platforms from the leaders of the three main parties. We felt that it would be good to capture the mood of the nation by allowing people to simply click a "+" or a "-" as to whether they liked or disliked what a leader said. This was then to be displayed as a sentiment "worm" alongside the video. Michael Brunton Spall created an AppEngine app which allowed us to scale rapidly and sustainably for the duration of the debates. At one point in time in the second debate the AJAX calls from the buttons were peaking at around 750 requests per second with an average of 454 requests per second. In total over the hour and a half of the debate over 2.7 million requests were made on the AJAX voting services which made heavy use of memcache and sharded counters. This sort of temporary load would have required a vast amount of additional temporary infrastructure for us and planning. AppEngine made that simple.

Twitter Collation

Just before The Edinburgh Festival, a really rather huge arts festival in the UK, we had a hackday and Stephen Abbott made a suggestion for something that it would be nice to see was an app that would allow the tweets about the festival of a collection of journalists and performers we trusted to be pulled together in one stream. This would need a double white list to be created, one where only if they were in the collection and they mentioned the keyword would your tweet make it into the stream. This is clearly to avoid spam and unwanted messages and to filter out journalists from talking about what they'd had for breakfast (unless it was festival related). In 24 hours using AppEngine Chris Thorpe had built a machine which fanned out queries from regular cron job which hauled in the stream of each journalist and performer (each in a single task queue action) and then processed each tweet asynchronously (again each in a single task queue action). The app performed well and was then industrialized as a MicroApp by Michael Brunton Spall and Lisa van Gelder and has been used throughout the general election and to bring together tweets from journalists within sections. We're now following and processing the streams of a large number of our journalists and are creating views of those streams with multiple watched keywords. The view of the stream can then be inserted into pages just like the one below using AppEngine and our MicroApp framework with memcache used to keep the response time fast.

Random Guardian

A flippant comment, in a serious question from Chris Thorpe to Clay Shirky, when he visited us recently at The Guardian, about where the new serendipity was in news, contained the phrase "should we just build Chatroulette for news?" He said that yes, we should definitely do that and in about 2 hours and using the beta of our new APIs, Daniel Vydra had done just that in AppEngine. The application has been hugely successful at introducing people to parts of the site they've never visited before and has led to people being mildly addicted to discovering things they never knew they needed to know, just at the click of a button. We're open sourcing the code for it very soon, so you too can start experimenting.

The MicroApp platform wouldn't exist without the hard work and brilliance of a team including Matthew Wall, Matt McAlister, Stephen Dunn, Mike Bracken, Graham Tackley, Stephen Wells and Simon Willison. We're now open for business and to prove it we're holding a hackday where we're making some very rapid prototypes for media agencies. Who knows, they may like them so much as concepts they commission them and we'll have to put them into production. That'll be fine for us, with the tools we use and the platform we have we know we have an environment to scale.

Photo by Dan Catt

Posted by The Guardian team