The 'Internet Of Things' and Data Pipelines: Handling Data at Massive Scale

In May 2013, the O’Reilly Data Sensing Lab collaborated with the Google Cloud Platform and Device Cloud by Etherios, to deploy a network of hundreds of environmental sensors at Google I/O. Learn how the Google Cloud Platform was used to build an end-to-end, scaleable, and high-throughput pipeline for data collection, processing, and analysis.

Highly scalable and rapid data collection and analysis is a key need for many mobile and gaming apps, as well as for sensor networks and the “Internet of Things.” We’ll show how the Data Sensing Lab incorporates a key Google Cloud Platform pattern: a high-throughput pipeline for data collection, processing, and analysis. We use the Cloud Endpoints API to collect constantly streaming data; process large amounts of data with high throughput using App Engine, Cloud Storage, and data transformation on Compute Engine; and query many GBs of collected data in just a few seconds using BigQuery.

Místnost: Da Vinci, 19:15 - 20:00 (45 min).