Big data engineering

Project Background UMG has recently adopted Google BigQuery as enterprise-wide analytics platform and currently has a number of projects for creating data pipelines which get data from various sources, perform necessary transformations and output data to BigQuery. The volume of data is assessed at 100s of terabytes. The chosen technology for these data pipelines is Google DataFlow which has been recently open-sourced by Google and re-branded as Apache Beam and currently at the incubating stage at Apache Foundation. There are two Google DataFlow/Apache Beam SDKs: Java SDK and Python SDK. Java SDK is stable and production-ready and Python SDK is in Beta, but also pretty stable. The SDK of choice for UMG going forward will be Python, because of faster delivery, more concise code and its appeal to Data Science teams. However, there are pipelines already developed using Java SDK, so we expect you to be able to work with both. Test The test is a typical de-normalization task that is performed frequently when loading data to BigQuery. The test itself doesn’t require interaction with BigQuery, as we find that final output of transformed data to BigQuery is the easy part. The transformations in Google DataFlow are more complex and this is what we would like you to do. You’ll be given 3 files in gzip-archived JSON format that we receive from Spotify API: streams, tracks and users. Your job is to develop two pipelines in Google DataFlow (one in Java and one in Python) to denormalize these three files into one flat output JSON file.