Tag Archives: Google Cloud

Filling the BigQuery paddling pool from the Kinesis Hosepipe

Posted on June 21, 2016 by Shine Technologies

Introduction This blog post details how we solved the problem of analysing large amounts of HTTP request logs for one of our clients. Spoiler: we used Amazon’s Kinesis and Lambda to stream the data into Google’s BigQuery for analysis. Read on for … Continue reading →

Posted in Node.js | Tagged , , , , , , , | 3 Comments

Creating a serverless ETL nirvana using Google BigQuery

Posted on March 30, 2016 by Shine Technologies

Quite a while back, Google released two new features in BigQuery. One was federated sources. A federated source allows you to query external sources, like files in Google Cloud Storage (GCS), directly using SQL. They also gave us user defined functions (UDF) in that … Continue reading →

Posted in databases, Javascript | Tagged , , , , , , , , , , , , | 4 Comments

Pablo rocking the stage at Google’s annual cloud event!

Posted on March 29, 2016 by Shine Technologies

Last week, Shine’s very own Pablo Caif gave a presentation at GCP Next 2016 in San Francisco, which is Google’s largest annual cloud platform event. Pablo delivered an outstanding talk on the work Shine have done for Telstra, which involves building solutions on the GCP … Continue reading →

Tagged , , , , , , , | Leave a comment

Shine’s Pablo Caif to present at GCP Next 2016!

Posted on March 1, 2016 by Shine Technologies

Shine is extremely proud to announce that Pablo Caif has been invited to present at GCP Next 2016, which is Google’s largest annual cloud platform event held in San Francisco. Pablo will be presenting on the work Shine have done for Telstra, which … Continue reading →

Posted in databases | Tagged , , , , , , , | Leave a comment

Messages in the sky

Posted on October 19, 2015 by Shine Technologies

One of the projects that I’m currently working on is developing a solution whereby millions of rows per hour are streamed real-time into Google BigQuery. This data is then available for immediate analysis by the business. The business likes this. … Continue reading →

Tagged , , , , , , , , | Leave a comment

Shiners now officially ‘Google Developer Experts’

Posted on October 16, 2015 by Shine Technologies

You may have already read our previous post here about Shine’s Pablo Caif & Graham Polley being nominated to become ‘Google Developer Experts‘ (GDE). Well, today Shine are proud to announce that both guys have been officially awarded the braggable title of … Continue reading →

Tagged , , , , , | Leave a comment

Google Cloud Dataproc and the 17 minute train challenge

Posted on October 14, 2015 by Shine Technologies

My work commute My commute to and from work on the train is on average 17 minutes. It’s the usual uneventful affair, where the majority of people pass the time by surfing their mobile devices, catching a few Zs, or by reading a … Continue reading →

Posted in DevOps, Linux, Opinion, Tools | Tagged , , , , , , , , , , , , , , , , , , | 2 Comments

Video of ‘Big Data the Cloud Way’ presentation now available

Posted on August 7, 2015 by Shine Technologies

Video of the recent ‘Big Data the Cloud Way’ presentation by Shine’s Graham Polley and Shane Neubauer is now available. In it, Graham and Shane talk about how they used both the Google and Amazon cloud stacks at Telstra to … Continue reading →

Tagged , , , , , , | Leave a comment

Upcoming Shine Event: Leveraging Big Data in the Cloud using Google and AWS

Posted on June 18, 2015 by Shine Technologies

Shine would like to invite you to a special event we are hosting in Melbourne on July 8. In a one-off presentation to be held at ACMI in Federation Square, Shiners Graham Polley and Shane Neubauer will share with you … Continue reading →

Tagged , , , , , , , | Leave a comment

Test Driving Google Cloud Dataflow

Posted on February 25, 2015 by Shine Technologies

Back in June 2014, at the annual Google IO in San Francisco, Google unveiled their newest, and much hyped cloud product, Cloud Dataflow. The demo they did that day, using a live twitter feed to analyze supporter sentiment during the … Continue reading →

Posted in Testing | Tagged , , , , , , , , , , , , | Leave a comment