Showing 53 posts by Keith Gregory.
In a perfect world, there would never be a need to connect to your resources running on AWS. In the real world, it’s sometimes necessary to get your hands dirty and look at what’s happening on the actual machine, especially during development. This post dives into a few ways to connect your workstation to resources running inside a VPC. It started out as a how-to for using bastion hosts, but quickly expanded to look beyond the bastion.
Decision support databases have a number of quirks that are not obvious to the casual user, particularly someone coming from an OLTP background. In this post I look at how unbalanced distributions can impact your query performance, how you can identify imbalances, and what you can do to fix them.
It’s possible to analyze your Glue jobs using just the logs they produce. Possible. But it’s not a pleasant task: your log messages are buried in messages from the framework, and in the case of a distributed PySpark job they’ll be spread amongst multiple CloudWatch log streams. In this post I look at an alternative: AWS X-Ray, which captures and aggregates “trace segments” that monitor specific sections of your code. With X-Ray, you can easily see where your jobs are spending their time, and compare different runs.
In my last post I recommended using Avro for file storage in a data lake. It has the benefits of compact storage and a schema in every file that tells you what data it holds. In this post I show three ways to generate Avro files: one in Java, and two in Python.
I've never been a JSON hater, but I’ve recently run into enough pain with JSON as a data serialization format that my feelings are edging toward dislike. However, JSON is a fact of life in most data pipelines, especially those that receive event-stream data from a third-party supplier. This post reflects on some of the problems that I’ve seen, and solutions that I’ve used
More, cheaper, faster: our own Keith Gregory recounts the changes in big data, data storage, and data engineering over the last two decades.
Two months ago I didn't give much thought to controlling a program's access to the Internet. Then Log4Shell happened. This post looks at three ways that you can control what an in-VPC application is allowed to talk to.
It’s been a week since CVE-2021-44228, a remote code execution vulnerability in Log4J 2.x, hit the world. Hopefully by now everybody reading this has updated their Java deployments with the latest Log4J libraries. But no doubt there’s another vulnerability, in some popular framework or library, just waiting to make its presence known. This post is about Cloud features that act to minimize the blast radius of such vulnerabilities.
Amazon Redshift’s launch in 2012 was one of the “wow!” moments in my experience with AWS. Here was a massively parallel database system that could be rented for 25 cents per node-hour. Here we are in 2021, and AWS has just announced Redshift Serverless, in which you pay for the compute and storage that you use, rather than a fixed monthly cost for a fixed number of nodes with a fixed amount of storage. And for a lot of use cases, I think that’s a great idea. So I spent some time kicking the tires, and this is what I learned.
Earlier this year I wrote that Amazon Aurora Serverless allows you to implement a fully serverless application with a relational database. In this post I show you how to use the Data API to do just that.
Amazon Athena is a service that lets you run SQL queries against structured data files stored in S3. It takes a “divide and conquer” approach, spinning up parallel query execution engines that each examine only a portion of your data. The performance of these queries, however, depends on how you consolidate and partition your data. In this post I compare query times for a moderately large dataset, looking for the “sweet spot” between number of files and individual file size.
In my last post I discussed how an artifact server is the best way to publish locally-developed Python packages. In this post, I show you how to set up the AWS CodeArtifact service and use it with pip and Poetry.
Coming from a Java background, I consider the Python development process to be a bit of a mess. The pieces are all there: a central repository for publicly-available packages, a way to install the packages you want, and several ways to run your program with only those packages. But it seems that everybody has a different way to combine those pieces. So when a colleague introduced me to Poetry, my first reaction was “oh great, another tool that solves part of my problem.” But after spending time with it, I don't want to build Lambdas any other way.
Different numbers of availability zones are appropriate for different workloads. This post helps you pick an appropriate number for your needs.
“Traditional” deployment patterns separate the application from its infrastructure. Lambda deployments turn this model on its head, binding the infrastructure tightly to the running code. This can be a challenge, especially when developing in a team: it is all too easy for one developer to accidentally overwrite another’s work. In this post I look at several deployment options, and how they impact a development team.
In my last post, I looked at how you could package your Lambda as a Docker image. In this post, I show how you can use the base Amazon images to build a "traditional" Lambda and ensure that it has libraries that are appropriate for the Lambda runtime environment.
I find the SAM (Serverless Application Model) CLI extremely frustrating to use on Linux, starting with installation. But this week I learned two things that simplify both installation and operation. I’m passing them on in the hopes that they’ll be useful to you as well.
Frequent database password changes are a best practice, because they reduce the "blast radius" if compromised. However, restarting your applications in order to pick up the latest password can be onerous in a large deployment. This post describes how to implement a custom Postgres datasource that calls on IAM to generate a password whenever your application opens a connection to the database.