Back to Blog
KG
Author

Keith Gregory

Showing 53 posts by Keith Gregory.

Data

Populating a Data Lake with AWS Database Migration Service and Amazon Data Firehose

Data lakes are great for holding large volumes of data, such as clickstream logs. But such data has limited usefulness unless you can combine it with data from your transactional, line-of-business databases. And this is where things get tricky. Simple approaches, such as replicating entire tables, don’t scale. Streaming approaches that include updates and deletes require logic to determine the latest value (or existence!) of any given row. All of which has to be translated into static data files in a data lake. In this post I look at one approach to solve this problem: AWS Data Migration Service to capture changes from the source database and write them to a Kinesis Data Stream, and Amazon Data Firehose to load those records into Iceberg tables.

KG
Keith GregoryMarch 18, 2025
ProgrammingPerformance

What’s the point of Lambda SnapStart?

Lambda SnapStart is intended to improve the cold start time for a Lambda function. It’s been available for Java workloads since 2022, and was recently released for Python and .Net workloads. It works by running the initialization code of your Lambda function when you release a version, and then storing an image of the Lambda execution environment. Cold starts load this image rather than running the initialization themselves. Given that cold starts happen unpredictably, and may be measured in seconds, this seems like a win-win situation. The reality, as usual, is more nuanced. SnapStart introduces its own cold start delays, as it loads the image into the runtime. And it increases the time and effort of deployment. In this post I drill down into the nuance, so that you can decide whether it’s a worthwhile choice fo your project.

KG
Keith GregoryDecember 4, 2024
Tips And TricksInfrastructureDevelopment

Troubleshooting ECS Deployments

At Chariot, we’re fans of deploying on Amazon’s Elastic Container Service using Fargate. However, there are a few stumbling blocks for ECS deployments. In this post I cover the common problems that I’ve seen, and some of the techniques that I use to avoid tripping over them.

KG
Keith GregoryNovember 14, 2024
Data

Websockets feeding Kinesis

We recently explored a project to retrieve data from a third-party service. They didn’t offer any push capabilities such as writing to a Kafka or Kinesis stream, or even a web-hook. But they did offer a WebSocket interface, so we explored whether we could use that as our streaming source. We didn’t go that route, but I was intrigued by the idea enough to make a proof-of-concept.

KG
Keith GregoryJune 11, 2024
Programming

Lambda Four Ways, a Rosetta Stone for AWS

When I write Lambdas professionally, Python is my preferred language. It offers decent performance, a straightforward syntax, and high developer productivity. I’ve also used Java, both in demonstration apps and actual client work. But while I have some familiarity with other languages supported by the platform, I’ve never used them. So, with some downtime, I decided to implement the same Lambda in four different languages: Python, Java, JavaScript, and Go, to get a better sense of their strengths and weaknesses.

KG
Keith GregoryMay 14, 2024
Tips And Tricks

Perils of Partitioning

Partitioning is one of the easiest ways to improve the performance of your data lake, because it reduces the amount of data scanned. But implementing partitions can be surprisingly challenging, as can their effective use. In this post I look at several of the issues that you should consider when partitioning your data.

KG
Keith GregoryMarch 22, 2024
TutorialData

Transforming Data with Amazon Athena

My prior posts used Lambda to do data transformation. But what if we could use a non-programmatic tool, in keeping with the Extract-Load-Transform mindset of the modern data pipeline. As it turns, we can: Amazon Athena can write data as well as query it. There are, of course, a few stumbles along the way. In this blog post I walk through the process of aggregating CloudTrail data using SQL.

KG
Keith GregoryMarch 15, 2024
Tips And TricksData

Aggregating Files in your Data Lake – Part 2

When I ran the Lambda from my previous post against Chariot's CloudTrail repository, it took almost four minutes to process a single day's worth of data. That seems like a long time, and as a developer I want to optimize everything I write. In this post I look into analyzing the current runtime, and options for improving it.

KG
Keith GregoryFebruary 21, 2024
TutorialData

Aggregating Files in your Data Lake - Part 1

As I’ve written in the past, large numbers of small files make for an inefficient data lake. But sometimes, you can’t avoid small files. Our CloudTrail repository, for example, has 4,601,675 files as-of this morning, 44% of which are under 1,000 bytes long. In this post, I develop a Lambda-based data pipeline to aggregate these files, storing them in a new S3 location partitioned by date. Along the way I call out some of the challenges that face such a pipeline.

KG
Keith GregoryFebruary 14, 2024
ReviewTools

Putting Amazon CodeGuru Reviewer To The Test

Amazon CodeGuru Reviewer promises to "detect potential defects that are difficult for developers to find," using machine learning to identify potential problems. But how does it compare to existing rule-based tools? In this post I turn CodeGuru loose on a seven-year-old library that's in use by 3,000 people, to see what issues it flags.

KG
Keith GregoryNovember 8, 2023
Review

Java Virtual Threads: Worth the wait?

With version 21, Java got virtual (lightweight) threads. This feature has received a lot of press, but will it actually help you? In this post I review the theoretical benefits of virtual threads, and then show actual results from a benchmark.

KG
Keith GregoryOctober 30, 2023
Design

Small Data: a pipeline for low-latency decision support

In my last post, I said that I didn’t think Postgres was a good choice for a decision support database, versus a task-specific DBMS such as Redshift. In this post I’m going to take the opposite stand, and say that there are cases where Postgres is appropriate: namely, low-latency systems that contain a limited amount of data.

KG
Keith GregoryAugust 8, 2023
Analysis 2

Why Not Just Use Postgres?

My last few posts have focused on Redshift and Athena, two specialized tools for managing and querying Big Data. But there’s a meme that’s been floating around for at least a few years that you should just use Postgres for anything data-related. It may not provide all of the features and capabilities of a dedicated tool, but is one less thing to learn and manage. Should this advice also apply to your data warehouse?

KG
Keith GregoryJuly 19, 2023
Analysis 2

Performance Comparison: Athena versus Redshift

I’ve always been a fan of database servers: self-contained entities that manage both storage and compute, and give you knobs to turn to optimize your queries. The flip side is that I have an inherent distrust of services such as Athena, which promise to run queries efficiently on structured data split between many files in a data lake. It just doesn’t seem natural; where are the knobs? So, since I had data generated for my post on Athena performance with different file types, I decided to use that data in a performance comparison with Redshift.

KG
Keith GregoryMay 25, 2023
Analysis 2

Athena Performance Comparison: Avro, JSON, and Parquet

In my “Friends Don’t Let Friends Use JSON” post, I noted that I preferred the Avro file format to Parquet, because it was easier to write code to use it. I expected some pushback, and got it: Parquet is “much” more performant. So I decided to do some benchmarking.

KG
Keith GregoryMay 16, 2023

Ready to transform your business?

We solve complex business problems with technology. Let's discuss how we can help you build, modernize, and grow.

Start A Conversation