Resources

The Chariot Blog

Tutorials, analyses, and the business of software — our consultants share over two decades of knowledge.

Data

Populating a Data Lake with AWS Database Migration Service and Amazon Data Firehose

Data lakes are great for holding large volumes of data, such as clickstream logs. But such data has limited usefulness unless you can combine it with data from your transactional, line-of-business databases. And this is where things get tricky. Simple approaches, such as replicating entire tables, don’t scale. Streaming approaches that include updates and deletes require logic to determine the latest value (or existence!) of any given row. All of which has to be translated into static data files in a data lake. In this post I look at one approach to solve this problem: AWS Data Migration Service to capture changes from the source database and write them to a Kinesis Data Stream, and Amazon Data Firehose to load those records into Iceberg tables.

KG
Keith GregoryMarch 18, 2025
ProgrammingPerformance

What’s the point of Lambda SnapStart?

Lambda SnapStart is intended to improve the cold start time for a Lambda function. It’s been available for Java workloads since 2022, and was recently released for Python and .Net workloads. It works by running the initialization code of your Lambda function when you release a version, and then storing an image of the Lambda execution environment. Cold starts load this image rather than running the initialization themselves. Given that cold starts happen unpredictably, and may be measured in seconds, this seems like a win-win situation. The reality, as usual, is more nuanced. SnapStart introduces its own cold start delays, as it loads the image into the runtime. And it increases the time and effort of deployment. In this post I drill down into the nuance, so that you can decide whether it’s a worthwhile choice fo your project.

KG
Keith GregoryDecember 4, 2024
Tips And TricksInfrastructureDevelopment

Troubleshooting ECS Deployments

At Chariot, we’re fans of deploying on Amazon’s Elastic Container Service using Fargate. However, there are a few stumbling blocks for ECS deployments. In this post I cover the common problems that I’ve seen, and some of the techniques that I use to avoid tripping over them.

KG
Keith GregoryNovember 14, 2024

Ready to transform your business?

We solve complex business problems with technology. Let's discuss how we can help you build, modernize, and grow.

Start A Conversation