Back to Blog
July 31, 2020·Becca Refford

15 Minutes With: Andrew Ganim on Building a Data Pipeline

Keith Gregory talks to Andrew Ganim, one of Chariot's experienced software consultants, about his recent project: building a data pipeline for a multinational company.

In today’s episode of 15 Minutes With, Keith Gregory, Chariot’s AWS Practice Lead, talks to Andrew Ganim, one of Chariot's experienced software consultants.

Andrew’s most recent project was to help a multinational company better analyze their data by building a more robust data pipeline. He was brought in to clean up both their current pipeline code, and the data that was coming in. This project involved restructuring, joining, and aggregating purchase data (transactional) and behavioral data (clicks, add-to-carts, etc.) from both brick-and-mortar stores, and online sales.

Dealing with terabyte-scale data, international channels, data privacy laws from multiple countries, and all the different ways to represent retail transactions (like clearances and discounts) is extremely complex. So how did Andrew account for edge cases without leaving behind messy code? How did he partition all this data? What tools did he use?

Andrew and Keith discuss.


https://youtu.be/07Xs3gtujJc

Is there anything you'd like us to discuss in this series? Email info@chariotsolutions.com, or leave a thought below.

Comments

Ready to transform your business?

We solve complex business problems with technology. Let's discuss how we can help you build, modernize, and grow.

Start A Conversation