-
Recent Posts
Top Posts & Pages
- About
- Contact
- A Deep Dive into DynamoDB Partitions
- Filling the BigQuery paddling pool from the Kinesis Hosepipe
- The most important thing when picking HTTP status codes
- Orchestrating Tasks Using AWS SWF
- The Emergence of The 3 Towers: DevSecOps
- Shine hosts a successful Digital Leaders Breakfast
- Creating a serverless ETL nirvana using Google BigQuery
- Pablo rocking the stage at Google's annual cloud event!
Tags
- 5G
- A/B testing
- accessibility
- accessors
- Adobe
- Adobe Experience Manager
- AEM
- Agile
- Amazon
- amazon cognito
- angular
- angularjs
- appstore
- Architecture
- aria
- asynchronous
- Asynchronous Programming
- Awards
- aws
- Babel
- Backbone
- backbone.js
- backbonejs
- battlehack
- bem
- Best Practices
- Big Data
- BigQuery
- bluetooth
- BQ
- braintree
- Broadband
- Chart.js
- CI
- citcon-anz-2010
- Cloud
- Cloud Dataflow
- Cluster Computing
- Clustering
- CMS
- co
- code
- codedojo
- Commenting
- community
- comparison
- concordion
- Conference
- continuous delivery
- CoreLocation
- couchdb
- couchtato
- cq5
- css
- cucumber
- customer experience
- CX
- Database
- Databases
- delegate
- Design
- devops
- DevOpsSec
- DI
- dirty checking
- Distruptive
- Documentation
- DynamoDB
- dzone
- ec2
- emberjs
- empathy
- Enterprise
- es6
- ES2015
- ETL
- fiber
- firebase
- fitbit
- fullcalendar
- GCP
- generators
- gigabit
- Git
- Gitflow
- Google BigQuery
- Google Cloud
- google cloud dataflow
- Google Cloud Datastore
- Google Cloud Platform
- Google Cloud Storage
- Google Developer Expert
- Google Maps
- Google Pub/Sub
- GPS
- grunt-clientlibify
- gruntjs
- hackathon
- hacking
- Hadoop
- hardware
- html5
- HTTP
- hudson
- human-centered design research
- IaaS
- identity map
- idm
- in-app
- Innovation Session
- Internet
- Internet of Things
- iOS
- iPhone
- itunes
- jackson
- Java
- JavaOne
- javascript
- JavaServer Pages
- jenkins
- jhipster
- jmeter
- jquery
- jsconfau
- jslint
- json
- JSP
- key value observer
- kids coding
- kinesis
- koa
- lambdas
- lessons
- Location Data
- Loon
- MapReduce
- Mass Insertion
- melbjs
- meteor
- MFA
- microservices
- Mobile
- Model–view–controller
- monitoring
- Multi -Factor-Authentication
- netbiscuits
- NewRelic
- News
- ngaria
- node
- node.js
- node.js
- NoSQL
- notification
- oam
- Objective-C
- OpenCms
- Open source
- oracle access manager
- osdc2103
- pair-programming
- participatory design
- partitions
- patterns
- payments
- paypal
- Performance
- PHP
- phpunit
- Pipelining
- Play
- Play Framework
- preferred vendors
- Programming
- Projections
- Promises
- Provisioned IOPS
- purchasing
- Python
- qualitative
- R
- Rails
- rds
- re: Invent
- React
- ReactiveCocoa
- React Native
- Redis
- Redshift
- release management
- Replication
- REST
- RESTful
- Risk Management
- Ruby
- s3
- Scala
- SDK
- Security
- Sensis
- service design
- Silex
- Simple Workflow
- Simulate
- single page app
- single page apps
- sinon.js
- site monitoring
- Software Development
- Software Engineering
- spa
- Spark
- Speed
- spring
- Spring Data
- Spring Data REST
- Spring Framework
- Spring Security
- SQL
- Streaming
- streams
- styleguide
- SWF
- swift
- swipeconference
- tech
- Telstra
- testflightapp
- testing
- TLS/SSL
- Tomcat
- Touch ID
- tutorial
- UDF
- UI
- Unit Testing
- usability
- user defined functions
- ux
- wds2013
- web
- web directions
- Web Directions Code
- XCode
- YARN
- YouTube
- YOW!
Meta
Tag Archives: Big Data
Filling the BigQuery paddling pool from the Kinesis Hosepipe
Introduction This blog post details how we solved the problem of analysing large amounts of HTTP request logs for one of our clients. Spoiler: we used Amazon’s Kinesis and Lambda to stream the data into Google’s BigQuery for analysis. Read on for … Continue reading
Posted in Node.js
Tagged aws, Big Data, BigQuery, Cloud, Google Cloud, kinesis, lambdas, node.js
3 Comments
Creating a serverless ETL nirvana using Google BigQuery
Quite a while back, Google released two new features in BigQuery. One was federated sources. A federated source allows you to query external sources, like files in Google Cloud Storage (GCS), directly using SQL. They also gave us user defined functions (UDF) in that … Continue reading
Posted in databases, Javascript
Tagged Big Data, Cloud, Database, Google BigQuery, Google Cloud, Google Cloud Platform, Google Cloud Storage, IaaS, javascript, Programming, SQL, UDF, user defined functions
4 Comments
Pablo rocking the stage at Google’s annual cloud event!
Last week, Shine’s very own Pablo Caif gave a presentation at GCP Next 2016 in San Francisco, which is Google’s largest annual cloud platform event. Pablo delivered an outstanding talk on the work Shine have done for Telstra, which involves building solutions on the GCP … Continue reading
Shine’s Pablo Caif to present at GCP Next 2016!
Shine is extremely proud to announce that Pablo Caif has been invited to present at GCP Next 2016, which is Google’s largest annual cloud platform event held in San Francisco. Pablo will be presenting on the work Shine have done for Telstra, which … Continue reading
Posted in databases
Tagged Big Data, Cloud, Conference, Google BigQuery, Google Cloud, google cloud dataflow, Java, News
Leave a comment
A week in the life of a Google Developer Expert
A few months back, Shine’s Pablo Caif and Graham Polley were welcomed into the Google Developer Expert (GDE) program as a result of their recent work at Telstra. The projects they are working on consist of building bleeding edge big data solutions using tools like … Continue reading
Posted in Opinion, Uncategorized
Tagged Big Data, Cloud, Google, Google BigQuery, google cloud dataflow, Google Cloud Platform
Leave a comment
Messages in the sky
One of the projects that I’m currently working on is developing a solution whereby millions of rows per hour are streamed real-time into Google BigQuery. This data is then available for immediate analysis by the business. The business likes this. … Continue reading
Shiners now officially ‘Google Developer Experts’
You may have already read our previous post here about Shine’s Pablo Caif & Graham Polley being nominated to become ‘Google Developer Experts‘ (GDE). Well, today Shine are proud to announce that both guys have been officially awarded the braggable title of … Continue reading
Google Cloud Dataproc and the 17 minute train challenge
My work commute My commute to and from work on the train is on average 17 minutes. It’s the usual uneventful affair, where the majority of people pass the time by surfing their mobile devices, catching a few Zs, or by reading a … Continue reading
Posted in DevOps, Linux, Opinion, Tools
Tagged Big Data, Cloud, Cluster Computing, Clustering, Google BigQuery, Google Cloud, google cloud dataflow, Google Cloud Platform, Google Cloud Storage, Hadoop, IaaS, Java, MapReduce, Python, R, Scala, Software Engineering, Spark, YARN
2 Comments
Upcoming Shine Event: Leveraging Big Data in the Cloud using Google and AWS
Shine would like to invite you to a special event we are hosting in Melbourne on July 8. In a one-off presentation to be held at ACMI in Federation Square, Shiners Graham Polley and Shane Neubauer will share with you … Continue reading
Tagged Amazon, aws, Big Data, Cloud, GCP, Google, Google Cloud, News
Leave a comment
Playing with Play Framework 2.3.x: REST, pipelines, and Scala
It’s an established trend in the modern software world that if you want to get something done, you’ll probably need to put together a web service to get do it. People expect data and services to be available everywhere, in a mobile world. With the plethora of frameworks and technologies available to go about implementing a web service, it becomes a chore to try using anything beyond what’s already familiar. But every now and then it’s an enjoyable experience to dive into something new and distinctly unfamiliar. Continue reading