Stream your data into BigQuery in real-time
September 18th, 2013 | Published in Google Enterprise
Today, we live in a world where businesses are generating large amounts of real-time data from web applications that serve millions of users, online sales transactions, or customer activity created by an explosion of connected devices. Being able to react quickly to changes in the data being generated is critical to remain competitive. At the same time, businesses are gathering, storing and analyzing data -- sometimes 100s of gigabytes per day -- using legacy systems that struggle to keep up.
We built Google BigQuery to enable businesses to tackle this problem without having to invest in costly and complex infrastructure. And today this gets even easier with two key new features:
- Real-time data streaming: you can now stream events row-by-row into BigQuery via a simple new API call. This enables you to store data as it comes in, rather than building and maintaining systems just to cache and upload in batches. The best part? The new data is available for querying instantaneously. Streaming ingestion is free for an introductory period until January 1st, 2014. After that it will be billed at a flat rate of 1 cent per 10,000 rows inserted. The existing batch-based ingestion will continue to be free.
-
Query portions of a table: you can now query a specific subset of a table using a simple new @
that we call a “table decorator” in your SQL statements. Though restricted to data inserted within the last 24 hours, this capability provides significant benefits beyond just cost efficiency -- for example, in conjunction with real-time data streaming, you can now use table decorators to monitor the last 30 minutes of user activity after a new change is pushed to your application.
In addition to these features, we’ve also expanded BigQuery’s window functions to include SUM and COUNT -- statistical capabilities that many customers have asked for -- as well as regular analytic functions for calculating Correlation and Standard Deviation.
And to make the entire querying experience smoother, the BigQuery user interface has also received numerous productivity-enhancing updates. These include an expanding information panel when clicking on a query, as well as action buttons at the bottom of the query box to make it easier to edit, run, save, and show results.
You can get details about these new capabilities and examples from our Developer Blog and in our updated product documentation.
Whether it’s for capturing streams of application event logging or real-time user behavior analysis, we can’t wait to hear how you’re using BigQuery’s new features. And we hope you’ll share with our community via the #BigQuery tag on Google+.