Google Data

SpotCloud.com is the realization of a dream originating more than 7 years ago when we first started Enomaly. SpotCloud was born from a desire to change the way cloud computing is bought and sold for the better by overcoming the inefficiencies (including large capital commitments and low utilization rates) that beset the traditional models. SpotCloud is the first multi-supplier, multi-buyer model of cloud computing, all built on Google App Engine.

SpotCloud is the first marketplace where service providers can sell their unutilized or under-utilized IaaS cloud services, and where buyers can shop competitively for these services on the basis of quality, price, and geography across a federated global pool of capacity with a single, consistent set of access and management mechanisms.

SpotCloud provides its users with the following main services:

• VM image repository & VM image provisioning services
• REST API for cloud service users, with management and monitoring functionality
• REST API for cloud service providers
• REST client to access service provider API endpoints
• Single-point unified access & authentication to all underlying cloud services
• Federated management portal, providing management & monitoring over all connected cloud services
• Cloud service quality, reliability, and performance measurement instrumentation and rating
• Runtime management service (to enable time-limited instances)
• Billing, payment, and settlement services

Why Google App Engine?
We realized early on that a traditional data center infrastructure -- even one built on our own ECP technology -- was not a fit for a platform which needs to be global from day one. We also had a strong preference for Python, given that large parts of our existing IaaS software stack is built using it. Cost was also very important: SpotCloud needs to manage potentially very large Virtual Machines, delivering them to a globally distributed group of providers and managing them in near-real time across this extended footprint. With Google App Engine we have the power, flexibility, and global breadth of the Google infrastructure at our disposal. This was a key selling feature for us. Also the cost was practically impossible to beat.

How App Engine powers SpotCloud
As you can see from the list of services provided by SpotCloud, most of our application actually runs in the background. This is made possible by App Engine task queues and cron jobs. We need to poll for stage changes across hundreds (soon thousands) of service providers -- we need a near-real-time view of instance states, available virtual hardware profiles, cloud utilization data, etc. These things are triggered by cron schedules. Each of these tasks goes into a queue. We're able to execute these tasks to allow us to keep our view of service provider resources up to date without interrupting the front-end UI and API workflows.

Another challenge is executing tasks at a particular point in the future. Running instances deployed out to a provider through SpotCloud may expire, and renewal may or may not be allowed depending on the seller’s settings. So we need a way to terminate them if their time is up. We can use the task queue API to execute this in the future. We're able to ensure our data is consistent by adding tasks to the queue in a transactional way.

Buyers upload the appliances they want to provision to the market. So we obviously have a big storage requirement. App Engine’s data store allows us to do this in a highly performant, cost-effective manner. SpotCloud can then distribute these appliances out to providers on demand. We're able to do this because of the low-cost bandwidth Google provides.

We're big fans of Django at Enomaly, and we use it in SpotCloud. We've been able to loosely couple Django with the App Engine data store, along with other App Engine components and services like urlfetch. This means that we're not "locked-in" with App Engine. The App Engine API is structured in such a way that we're able to extend any web framework we want with App Engine components that help us scale.

SpotCloud Success
Since announcing the SpotCloud private beta at the end of last year, we’ve been overwhelmed by the amount of interest we’ve gathered, and have had large numbers of service providers from all over the globe join our beta. We now have more than 10,000 servers from 40+ countries represented in the marketplace.

Since we opened the marketplace to buyers and sellers early in 2011, the market has been off to a roaring start -- as noted by The Economist, ComputerWorld, and many others. Google App Engine has performed up to our (high!) expectations, serving an ever-increasing transaction rate in the market without a hiccup. We’re looking forward, as the market ramps up even further, to seeing if we can make App Engine break a sweat! Sign up today as a cloud capacity buyer or seller at www.spotcloud.com.

Posted by Reuven Cohen, Founder & Chief Technologist, Enomaly Inc.