Tag Archives: BigQuery

Filling the BigQuery paddling pool from the Kinesis Hosepipe

Posted on June 21, 2016 by Shine Technologies

Introduction This blog post details how we solved the problem of analysing large amounts of HTTP request logs for one of our clients. Spoiler: we used Amazon’s Kinesis and Lambda to stream the data into Google’s BigQuery for analysis. Read on for … Continue reading →

Posted in Node.js | Tagged , , , , , , , | 3 Comments

Shiners nominated for ‘Google Developer Expert’ program!

Posted on September 7, 2015 by Shine Technologies

Shine are delighted to announce that our very own Pablo Caif, and Graham Polley have both been nominated to become part of the Google Developer Expert program! Becoming an expert, means undergoing a stringent evaluation and interview process, as well as … Continue reading →

Tagged , , , , , , | Leave a comment

Video of ‘Big Data the Cloud Way’ presentation now available

Posted on August 7, 2015 by Shine Technologies

Video of the recent ‘Big Data the Cloud Way’ presentation by Shine’s Graham Polley and Shane Neubauer is now available. In it, Graham and Shane talk about how they used both the Google and Amazon cloud stacks at Telstra to … Continue reading →

Tagged , , , , , , | Leave a comment

Test Driving Google Cloud Dataflow

Posted on February 25, 2015 by Shine Technologies

Back in June 2014, at the annual Google IO in San Francisco, Google unveiled their newest, and much hyped cloud product, Cloud Dataflow. The demo they did that day, using a live twitter feed to analyze supporter sentiment during the … Continue reading →

Posted in Testing | Tagged , , , , , , , , , , , , | Leave a comment

License to Queue

Posted on December 19, 2014 by Shine Technologies

A few months ago I joined an exciting project with fellow Shiner Graham Polley, who you might know from such hits as Put on your streaming shoes. This is a follow-up article, discussing the elegant way in which we solved a … Continue reading →

Posted in DevOps | Tagged , , , | 7 Comments

Shine wins Computerworld Data+ Award!

Posted on September 16, 2014 by Shine Technologies

Shine is proud to have been awarded the Computerworld Data+ award for our work with Google BigQuery.  The innovative work is a great example of using great technology to deliver business benefit.  You can see the write up of the … Continue reading →

Tagged , , , , | Leave a comment

Put On Your Streaming Shoes

Posted on August 25, 2014 by Shine Technologies

The Kick-Off Meeting It went something along the lines of: Client: “We have a new requirement for you..” Shiners: “Shoot..” Client: “We’d like you to come up a solution that can insert 2 million rows per hour into a database and be able to deliver real-time … Continue reading →

Posted in Javascript, Opinion | Tagged , , , , , , , , , , | 13 Comments

Reeling in Big Data Using Google’s BigQuery

Posted on January 18, 2014 by Shine Technologies

Introduction For me, Jaws is hands down one of the best movies ever made. It’s almost 40 years old but it still looks fantastic and the acting is phenomenal. And it’s able to boast one of the most memorable ad-libs ever … Continue reading →

Posted in Javascript | Tagged , , , , , , , , | 6 Comments