I’ll explain what each of these AMIs are and how we build them in due course, but first off, let’s address an important question:

Two AMIs, in THIS economy?

We know it’s crazy, who even has the action minutes to raise AMIs these days; we sure don’t. But we had a problem, or maybe an opportunity. SimKube just keeps getting better and better but configuring it can be, frankly, difficult. Building high-fidelity simulation environments requires installing and configuring a long list of tools: kind, KWOK, kubectl, Docker, prometheus, and SimKube, to name a few. So spinning up a ready-to-go SimKube environment takes some doing.

Internally, we have a configuration management repository called isengard2. It is ~48k lines of pure Ansible bliss. We use it to automate the deployment of repeatable simulation environments. It occurred to us that users of SimKube probably don’t want step one of using it to be “here’s ~48k lines of Ansible, good luck!”. It turns out there is a better way: a custom SimKube AMI.

Why not a Docker image like a normal person?

That’s a fair question. We did evaluate using a Docker image because one of our primary goals is a fast, one-click startup.

The challenge is that SimKube relies on kind, which spins up Kubernetes nodes as Docker containers. Initializing and configuring the kind cluster requires access to a live Docker daemon. During docker build, there is no Docker daemon available inside the build environment, which means we can’t just “run all the setup steps in our Dockerfile” and ship the result.

We also looked at snapshotting a running Docker environment, but that’s complicated for a different set of reasons. So after a long side quest that included Vagrant and QEMU, we realized what we actually need isn’t a container image but a prebuilt machine image that preserves the state of our configured simulation cluster. Since we primarily work in AWS, an AMI fits naturally.

Baking the AMIs

Fortunately, baking AMIs is a fairly straightforward task that our ancestors have been doing for thousands of years. We can reuse a lot of what we have already built in our configuration management system (which I will remind you is lots and lots of Ansible). We use Packer to bake our AMIs, so the first step is selecting a base image which our custom AMI will be built on top of. We chose Ubuntu 24.04 LTS for its stability, compatibility with our tooling, and long term security patching.

Using Packer we can initiate an automated build via a GitHub Action. For configuration, Packer includes a range of provisioners—including one for Ansible—so we are able to leverage our existing configuration library in isengard. The GitHub Action itself is fairly simple: it clones the repo and runs packer. This helps keep our Packer configuration sparse and maintainable. We only need to configure a handful of things: the Ansible playbook to run, the region of our builder, our base AMI, regions to copy the finished AMI to, and any cleanup activities or supplemental provisioners.

Our AMI pipeline is triggered by a weekly cron trigger, or by a manual build via a dispatch trigger3. After some bake time, we end up with a custom AMI image backed by the snapshot created during the build. That AMI is then listed on the Marketplace.

AMI patching

Shipping a public AMI means we own patching it. At a minimum, Ubuntu is going to ship security patches (we really want those) and there will be patches for other software in our stack. Instead of patching in place, we treat each AMI build as an immutable artifact tied to the configuration management repository (isengard) git hash used to produce it. Every build is traceable and reproducible.

Every time our pipeline runs, it starts fresh with a clean Ubuntu image and pulls down the latest patches so each AMI is fully up to date at build time. The result is a simple, deterministic build process that is easy to maintain. AMIs are short-lived, they won’t drift over time, and there is no ambiguity about which code produced which image.

The tradeoff is that older AMIs are never patched. If you launch an older version, you get exactly what existed at build time. For our use case, this ends up being a feature since we value reproducibility. This comes in handy for debugging thorny issues in our AMIs, especially those that manage to bypass our validation tests. We can fire up an AWS EC2 instance and watch one of our services get clobbered in real time by some bad code that I definitely didn’t write.

For the most part our AMI pipeline quietly churns out new images. To keep our account from piling up with tons of old AMIs4, we use a Packer post-processing block to deprecate old AMIs and clean up their snapshots automatically.

You said TWO AMIs

I did say that! We have two versions of our AMI. The first is the free SimKube AMI with everything needed to run SimKube including a running kind cluster and management tools. This is our free-to-use simulation environment. All the user needs to do is launch it in AWS EC2 and get right to running simulations—though you will need a trace from the cluster you are simulating.

The second AMI is our SimKube GitHub Action Runner it includes everything in the SimKube AMI but also has some extra configuration applied. We use an iterative build process, so this version is literally built on top of the base SimKube AMI5.

This cuts down on build time and allows us to patch the GitHub Action Runner software independently of the base AMI if we wish. The additional configuration in this version is the GitHub runner software and a systemd wrapper that manages it. We use this version to run SimKube in CI pipelines (via GitHub Actions). Effectively, this AMI is primed to register itself with a GitHub repo as a custom action runner when it receives the information contained in our User Data script.

A world of opportunities

Our SimKube AMI is a step forward in making SimKube approachable and easy to use. Instead of spending a few hours setting up a simulation environment, you can grab the SimKube AMI off the AWS Marketplace and have SimKube up and running in a couple of minutes. You will need to grab a trace from your production cluster, but the environment for running those simulations is available at the click of a button or at the end of a AWS CLI command.

We want to continue to extend Kubernetes simulation into CI pipelines using our GitHub runner AMI. The vision is an engineer, maybe you, checks in some change to your cluster. Then, SimKube CI simulates it based on your production cluster and sends you back metrics you can use to evaluate your change before it hits production.

Today, ACRL is already running small simulations in CI in the SimKube repo. We have developed custom GitHub Actions, available in our simkube-ci-action repo, to make launching runners backed by SimKube AMIs as easy as adding a few lines in your GitHub Actions workflow.

So maybe you find SimKube interesting but setting it up has been too much of a hassle. Or perhaps you are already running SimKube locally but want to run a dozen simultaneous simulations in AWS. The AMIs are there for you, and the SimKube AMI is free-to-use–though you still have to pay AWS for the compute (sorry).

If you want to learn more, we’ve added a new SimKube in the Cloud section to the documentation that walks through how they work and how to get started.

So get out there and simulate some trouble… before it makes it to prod!

Cheers,

Ian

Quick note for my subscribers: In an effort to try to get a few more paid subscriptions to this blog, I’m running an experiment where I keep the “paid subscribers only” period for a week, instead of just “over the weekend”. I will still publish all of my posts publicly, but if you want to read it earlier, you’ll need to be a paid subscriber. As a reminder: paid subscriptions to the blog don’t pay my bills, but they help get me and Ian to conferences, which is where we meet the people who do pay our bills. So I very much appreciate everyone who is already supporting us, and look forward to getting even more!

Last week I had the privilege of giving a talk at SRECon Americas in Seattle; the recording of the talk will be published in a few weeks, but I thought for this post I’d do something a bit different and write an (approximate) transcript of the talk for those of you who don’t want to wait and/or prefer reading things to watching videos. The talk is, of course, about SimKube: I hope you enjoy! I’ll update this with a link to the talk once it’s posted.

Thanks all for coming; the title of my talk today is “Stop Reading Changelogs: Safer Kubernetes Upgrades with Simulation”.

On Pi Day three years ago, March 14, 2023, Reddit suffered a 100-pi-minute long outage, a 314-minute outage. In the extremely detailed and well-written postmortem, it was revealed that one of the primary causes of this outage was a Kubernetes version upgrade from Kubernetes 1.23 to 1.24. Specifically, in this upgrade, Kubernetes changed one of the labels that was applied to its control plane nodes, and Calico, the service mesh component used by Reddit at the time, was using a label selector that pointed to the old label for the node instead of the new label. This caused Reddit to go hard down for around 5 hours.

Now: I was not a Reddit engineer at the time of this outage and I am not a Reddit engineer now. In fact, I have never been a Reddit engineer, and I also know for certain that there are Reddit engineers in the audience today who were employed at Reddit at the time this incident happened. So let’s take a detour to answer the question, “Who the heck am I and what am I doing here?”

Hi! My name’s David; I go by drmorr on the internet, and I use this cute little grumpy red robot icon basically everywhere. I received my PhD in Computer Science from the University of Illinois in 2014, where I was focused on scheduling and optimization problems. From there, I got my first introduction to distributed systems at Yelp, which at the time was unhappy with the amount of money they were paying to AWS, and asked me to use my optimization skills to help them pay less money to AWS. Of particular note for this talk, while I was at Yelp I wrote my first distributed systems simulator, a simulation engine for Apache Mesos that we used to try to understand some of the cost-reduction changes we wanted to make. From Yelp I went to Airbnb, doing more of the same: scheduling, autoscaling, and cost optimization, before finally starting ACRL—a small business focused on open-source research and development in distributed systems. One of ACRL’s primary projects is a simulation environment for Kubernetes called SimKube.

With that out of the way, let’s go back to the topic at hand: why are Kubernetes upgrades so challenging? In the screenshot on the right, you can see the Kubernetes release schedule, and the main takeaway from this is that a new Kubernetes version is released every fourteen weeks—that’s three new versions a year. This is an extremely aggressive release cycle, and one that is difficult to keep up with. It’s easy to find many links on the Internet like this one, entitled “Managed Kubernetes Cluster Upgrades are a Total Nightmare”. What makes them so hard? Well, the primary reason is that upgrading Kubernetes requires performing upgrades to n independent control loops in a correct-but-unknown order. Threads like the above are filled with debates about whether you should do this process “in-place” or via a “lift-and-shift” approach where you spin up a cluster on the new version and then migrate your workloads over: and to be honest, both approaches have their tradeoffs.

I want to also look at what the typical upgrade process looks like at many organizations. Step 0 in this process is “someone tells you to upgrade”. Now, that person might be your boss, but more likely it’s one of the managed cloud providers. If you run on EKS, Azure, or GCP, you are going to be forcibly upgraded on a regular cadence; in some cases it might be possible to pay the cloud providers more money to stay on an older version, but that’s a stopgap solution that only delays the inevitable. At some point, you must upgrade.

So once you’ve decided to do the upgrade, you might decide that the reasonable next step is to read the changelog. On the right of this slide is an example for the Kubernetes 1.34 changelog. The text is probably too small for you to read in the back, but buried in the middle of this 2000-line-long markdown file are two bullets: the first reads “Urgent Upgrade Notes”, and the second sub-bullet says, in parentheses, “(No, really, you MUST read this before you upgrade)”. Again, this is in the middle of a multiple-thousand-line markdown file that users are expected to read.

But OK: you’ve read the changelog. The next step is likely to spin up (or utilize an existing) test environment, where you can deploy something that looks approximately like your production workload, and play whack-a-mole with the issues that crop up until you’re reasonably confident you got them all. Then you’re ready to upgrade in prod! You might start with your lowest-risk clusters, and gradually move to more critical clusters as you gain confidence in the rollout. Whoops! You missed one. I hope you have PagerDuty configured correctly.

And now you have a problem, because there is NO supported rollback plan for Kubernetes. Your choices are either, restore from a backup (you do have a backup, right? Does it work? Are you sure?) or to roll forward and try to identify and fix your broken cluster. This is exactly the choice that Reddit was faced with in their Pi Day outage.

However, there’s one point in this upgrade process that seems promising, and that’s this test cluster idea. Test clusters seem great! Why don’t we do more of those? In my experience, there’re three reasons why: first is that provisioning a new cluster is hard. I’ve been at some organizations where you can push a button and get a new cluster in about 30 minutes. I’ve been at other organizations where you have to make 10 changes in 20 different repos, and if you’re lucky you might have a running cluster 3 weeks later. I’m sure many organizations fall somewhere on that spectrum. But in any case, even waiting 30 minutes for a new cluster introduces a lot of friction when you have to repeatedly spin up and tear down an environment while you test things out.

The second reason why test clusters aren’t as effective as they could be is that they’re expensive. Particularly if you’re doing any sort of scale or load testing (and you want to do scale testing, because this is where the most gnarly and hard-to-detect issues live), it’s going to cost a lot of money. If you go to your boss and say, “Sure, I’ll happily do this upgrade, I just need a million dollars to run a thousand-node test cluster for a month,” they’re going to come back with, “No, try again.”

And the last reason why test clusters are hard is that replicating production is impossible. See, production has a bunch of annoying things like “users” and “network traffic” that you just don’t have even in a full-fledged staging or test environment. And you’re never going to be able to replicate and test all of that stuff.

Surely there’s a better way to do this? I think so! So in this talk I’d like to introduce SimKube.

Now, on this slide is a picture of a bridge. This is a picture I took of Tower Bridge, a local Sacramento landmark; I’m from Sacramento, and—as an aside—it was 90 degrees there last week, so I’m very happy to be in Seattle where it’s in the fifties. But anyways, back on topic: why is there a bridge? Well, in many other engineering disciplines, simulation is a common tool. Civil engineers would never build a bridge without simulating it first. Aerospace engineers would never build an airplane without simulating it first. And yet, in my experience, in our discipline, simulation isn’t a tool that’s reached for very often. Why not?

I think there are a number of reasons, but a significant one is: distributed systems are already hard. And simulating them is even harder. However, in the case of Kubernetes, I think there are two things going for us: the first is the declarative nature of the system. It’s a well-known design goal of Kubernetes that you write down the desired state of the system, and the Kubernetes control loops will try to move the observed, actual state of the world to that desired state. In other words: YAML. I know we all love to hate YAML, but for the purposes of simulation, I think it’s really beneficial. We can track, in a simple, easy-for-humans-and-computers-to-read format, the changes in the desired state of the system over time. And then we can record all of those changes into a trace file, which we can then replay in our simulated environment as many times as we want! Nifty.

The second feature of Kubernetes that helps with our simulation goal is the extensible API and custom controllers that Kubernetes supports—no, you know what, I’m kidding, this one’s YAML again. See, from the perspective of Kubernetes, a “node” is just a YAML blob that’s been written to etcd. It doesn’t care if there’s any real hardware backing that node. And a pod is just a YAML blob that’s been written to etcd. Kubernetes doesn’t care if there’s a real binary behind that YAML! So, if you believe really really hard that the node object that you wrote to etcd is real, then Kubernetes is going to believe it too; and likewise, if you believe, really really hard that your pod is real, Kubernetes will believe it too. And that’s exactly what KWOK, the Kubernetes WithOut Kubelet project, which underlies SimKube does: it is a custom controller that watches for Node and Pod resources that have been created in etcd, and it walks those resources through their lifecycles.

So with that, we can finally talk about how SimKube works. This next slide is a high-level architecture diagram of SimKube: I don’t care if you understand the details, all I want you to take away from this is that on the left, we have our real, production Kubernetes cluster. Inside that cluster, we have a small agent running that’s tracking all of the changes made to the YAML files describing that cluster; at any point in time, a user can then export those changes into a trace file, which get written to some persistent storage, and then replayed in the simulation environment on the right.

Let’s actually watch a demo so you can get a better idea of what I’m talking about.

In this demo, the first thing we’re going to look at is a trace file. I’m using skctl, the SimKube CLI tool, to inspect this trace file: as you can see, it’s just a timestampped series of events, and if we drill down into these events, you can see that it’s just Kubernetes manifests. This application implements a toy “social network” application, where you have microservices for browsing your timeline, writing posts, or analyzing the social graph. That’s not super important here, what’s important is that as the Kubernetes manifests for these services change over time, those changes get stored into the trace file.

So, let’s go ahead and run this trace file in our simulator. Once I do skctl run, it spits out a bunch of metadata for our simulation, and then it launches a driver pod in the simkube namespace. The driver is responsible for downloading the trace file and replaying all of the YAML inside it. We can look at the nodes on our cluster, and you see that I’ve got two real nodes (this is using kind on my laptop), and a whole bunch of fake nodes that have been provisioned by Karpenter and are backed by KWOK. We can also go into the namespace where the simulated pods are being created to see that they’re all running; we have a little over 1000 pods right now, but this simulation will actually scale up to about 5000 pods across a hundred or so nodes.

One thing I do actually want to call out here, if we look in this namespace we see that there’s a CronJob running, which is responsible for doing some cleanup actions in the user database. Now the thing about CronJobs is that they typically have a start and an end time. You might be wondering how that works in the simulation, since there’s no application code running to complete, but if you watch you’ll notice that these pods actually do complete! This is KWOK at work again: SimKube records information about the pod lifecycles and injects that into KWOK, and it takes care of walking the pod through its entire lifecycle.

Great! Now, you might have noticed that this cluster is running Kubernetes 1.24, which is pretty old, so let’s go ahead and try to upgrade. I’m going to switch over to another cluster on my laptop that’s running Kubernetes 1.25, and re-run that same simulation. I’d like to acknowledge that what you’re about to see is not the best user experience—we’re working on it. But, we start the simulation, the driver pod comes up, and then a few seconds later—whoops! It crashed. Let’s go ahead and look at the logs.

Inside the logs, we see a traceback, and the message at the top says that the requested resource cannot be found, and returns a 404. If we scroll up a bit further, we find the culprit: the CronJob that I highlighted earlier. Now if you know your Kubernetes history, you probably already know the punchline here. Kubernetes uses versioned APIs, which is how they are safely able to evolve their user-facing APIs over time, and between Kubernetes 1.24 and 1.25 the v1beta1 API was removed from Kubernetes. Our trace file was still referencing the v1beta1 API, and when we tried to apply that to the new cluster, it crashed.

This is cool! We’ve used simulation to identify a problem with our cluster. But we can take this one step further. SimKube includes a bespoke DSL called SKEL (the SimKube Expression Language) which you can use to make targeted modifications to your trace file. The type of modifications you can make are quite complex, but for this example we’re going to do one simple transformation. Here we’re selecting all resources that have kind equal to CronJob, and we’re replacing their apiVersion with batch/v1. Then we just use skctl to apply this transformation to our trace file, and we can try running our simulation again. And this time, it works! We can watch the driver pod, which crashed after about 10 seconds last time, and this time it keeps running. We can also go into the simulation namespace and confirm that the simulated pods are all present, including the CronJob pods.

So that’s SimKube, testing your Kubernetes upgrades! We can go even further, though: we’re all SREs, we like to automate things, so what if we just automate the entire process? Imagine if you regularly collect some traces from your production infra, store them in S3, and then periodically—maybe every time you need to upgrade, or maybe even just once a month to establish a baseline—you run those traces through a simulator, maybe as a CI job or something else? Well, it turns out you can do that too. We’ve released a free AMI that you can use to play around with SimKube, and soon we’ll have a GitHub runner that you can plug into your CI pipeline as well. In fact, we do exactly this on the SimKube repo: on the right you can see a screenshot of our GitHub actions, and on every merge to main we run an end-to-end test of SimKube on a few sample traces to ensure that the system is behaving properly.

To wrap up this talk, let’s revisit the things that made test clusters hard from the beginning: first, using simulation makes provisioning clusters easy. I can spin up a thousand-node cluster on my laptop in a couple of minutes.

Secondly, using simulation makes scale testing free. I mean, not free-free, you still have to have some hardware to run it on, but compared to the cost of a full production environment, we’re talking pennies. And, lastly, replicating your production environment is impossible!

Oh. Hmmm.

Ok, look. I don’t want you to walk out of this talk thinking that simulation is a silver bullet. The whole entire point of simulation is that you’re throwing out some part of your system in the hopes that it makes the analysis of the other part easier. The trick is knowing what part of the system you want to throw out while still maintaining the right level of fidelity. So here are three examples of places where SimKube struggles.

First, anything involving networking: since there’s no application code running, there’s nothing to respond to your network requests, which means testing load or network patterns won’t work. Secondly, I lied slightly earlier in the talk when I said that Kubernetes is 100% YAML – there are some components, like the Horizontal Pod Autoscaler (HPA) which make decisions based on the results of metrics like CPU utilization or other real-time data. SimKube can’t handle that yet—but unlike the networking thing, there’s nothing technically stopping us from providing fake Prometheus metrics data to the HPA, and in fact, this is something that I’m hoping to prioritize in the next year. And, lastly, any sort of integrations with, well, I wrote cloud providers here, but this could really be any third-party service, gets tricky. See, the cloud providers don’t even provide you access to the control planes of your cluster, so getting the right data out can be challenging. And other third party systems may be similar.

So those are some areas where SimKube specifically might struggle, but what I really hope you take away from this talk is that SimKube is just one tool in your toolbox. There are lots of other tools out there that can help to cover some of these areas. To close this talk out, I want to revisit the Reddit outage from the beginning. Here’s the question: “Would SimKube have prevented that outage?”

On the one hand, maybe yes? Reddit engineers could certainly have loaded up some production data into SimKube and tried to do their upgrade, and they could have seen that the control plane node labels changed and (potentially) Calico stopped registering routes. So, maybe if SimKube had existed back then, it would have prevented the outage. On the other hand, no, SimKube couldn’t have prevented that outage. The Pi-Day postmortem highlighted a host of factors contributing to the incident, only some of which were technical. And SimKube can’t do anything about those: again, it’s just a tool, and in order for a tool to be effective, you need people who know who to use the tool, when to use the tool, how to interpret the results from the tool, how to communicate those results, and so on and so forth. So SimKube isn’t the whole picture here, but I do think (and I hope I’ve convinced you) that it can be a really powerful part of the picture. It’s certainly one that I have a lot of fun working on.

So that’s I’ll I have for you today! I’ll close with these three links: the first takes you to simkube.dev, which is the documentation site for the project; the second takes you to my blog where you can read about how SimKube has been used; and the third is a calendar link. I love getting to chat with new folks, so if you have further questions or you just want to say hi, feel free to grab some time!

Thanks for your time, and I’ll be happy to take any questions that you have.

So that was the talk! I hope you enjoyed getting to read it, even if you weren’t able to see the talk in person. I got a lot of great questions and engagement, and I’m pretty excited to see where we go next! As always, thanks for reading.

~drmorr

OK, as promised last week, this post is a follow-on to my previous write-up about running a single-node Kubernetes cluster at ACRL. After calling that post “hot garbage”, the redditor who commented on the post went on to say that he was hoping for more details and less high-level pish-tosh1. So, random redditor, this 4000-word manuscript is for you! I hope you brought your coffee.

Project Goals: Quick Recap

Before we dive into all the arrows in the architecture diagram, I just wanted to quickly review what the goals of this project are:

1. Run a Kubernetes cluster in the cloud,

2. as cheaply as possible,

3. and schedule a workload on it. Lastly,

4. it should be one command to completely destroy and recreate our entire stack if we need to.

To recap what I discussed in the previous post, we’re running k3s on a single AWS EC2 spot instance. We’re not using EKS, a) because it’s expensive, and b) because we’re supposed to Kubernetes experts over here and not just some random hack jobs, so we should probably have some expertise running Kubernetes. We’re running on a spot instance a) because it’s cheap, and b) because I wanted to know if it could be done. If you’re following along at home and you’re not running on a spot instance, you can probably do away with a lot of this complexity; but on the flip side, running on spot serves as a forcing function to build the automation and tooling to easily recreate your entire state from scratch, which is in my experience an extremely underrated superpower.

In my previous post, we got to the point of “having a running k3s node”, but we didn’t actually get to the point of “running a workload on it”, partly because I ran out of time, and partly because I thought that once I had the node up and running, running the pods would be easy. It’s just YAML, right2???

Anyways, in the last post we also didn’t define what workload we’re going to be running on our Kubernetes cluster. I have quite a few things I eventually want to run, but my first goal was just to run a simple XMPP server. I’ve wanted a good internal “chat” solution for a while, I really don’t want to use Slack or Discord, and it turns out that XMPP has been living a healthy thriving life for the past 15 years, even if most people don’t realize it. So after a bit of research, I decided that I was going to get Prosody running inside my Kubernetes cluster. It’s small and lightweight enough that “configuring Prosody” wouldn’t take too long, but it would also exercise a bunch of features that I’m going to want later: specifically, ingress, certificates, and persistent data.

I also also didn’t explicitly discuss the fourth goal in my last post, but this is really important to me. I’m a very big proponent of the “GitOps” pattern3. If you can check in your desired infrastructure state into version control, and it’s easy to recreate and/or get back to a “known good configuration”, that makes everything so much easier down the line. So for me at least, it’s worth it to spend some more time up front doing things “right” to (hopefully) make my life a little easier in the future4.

So with that said, let’s go into the nitty-gritty details.

Revisiting Middle-earth

In my last blog post I explained that I have two repos named after famous locations from Lord of the Rings: Moria handles my “infrastructure as code” and Isengard is my server configuration management repo. Since that post, I’ve added a third git repo, Mirkwood, which contains all of my Kubernetes manifests5. You can see in the above diagram how each of these repos interact with components in AWS, shown by the colored solid arrows (green is moria, blue is isengard, orange is mirkwood, and the dashed lines indicate physical hardware connections or network connections); feel free to refer back to that diagram as we continue.

Moria

My last post did a pretty good job of covering the basic AWS infrastructure setup, so I’ll just quickly recap it here. In Moria, we use Pulumi to create the following AWS resources:

1. A single bare EC2 node as an SSH proxy/bastion host/NAT gateway into my VPC.

2. A single-node AutoScaling Group (ASG) which contains the k3s server node, registered as a spot instance.

3. A persistent EBS volume which is auto-attached to the k3s server node on first boot (via a systemd service), which serves as the data volume for k3s.

4. An internal zone in Route53 which I can use to store a DNS A record pointing to the k3s server.

5. An AWS Lambda function, run as an ASG Lifecycle Hook which updates the A record whenever the k3s server is terminated and recreated.

Since the first post, I’ve had to add a number of other AWS resources as well into Moria to support various tooling further down the stack. XMPP requires TLS these days, which means I need a certificate; I considered whether to use the CA that I set up earlier in the year for this purpose, but those certificates are for a very different purpose than what I want now. I also considered whether to create a second CA for “internal services” but that also didn’t feel like a good approach. Fortunately, Let’s Encrypt is a thing; I’ve been using it for a long time for my personal website, and it has good integrations with Kubernetes, so this felt like the right approach.

Certificates need to be signed for a particular domain, however, and Let’s Encrypt needs to know that you own the domain that it’s signing certificates for. But my Kubernetes cluster is inside a private VPC that Let’s Encrypt knows nothing about! So this means I need to use Moria to add

6. Two more hosted zones67 into my infrastructure; a public zone that Let’s Encrypt can use to verify ownership, and a private zone that will host the A records for my XMPP server. Note that we don’t actually configure those A records with Moria, however, because the IP address of Prosody can change.

I also didn’t talk about this at all in the last post8, but Moria also manages

7. All of the AWS IAM (Identity and Access Management) permissions for this configuration.

8. An encryption key in AWS KMS (Key Management Service) to help with secure secret storage.

IAM permissions are some of the most arcane and convoluted things I have ever had to work with, but essentially every component above requires a corresponding IAM policy that enables it to perform its task. The proper, most secure way of managing these permissions involves things like short-lived temporary credentials and OIDC and JWT and a whole bunch of other ugly acronyms, but frankly that stuff gives me heartburn every time I look at it, so for right now I just have a bunch of “service account” IAM users with static credentials configured in Moria. This is marginally less-secure, but I make it easy in Moria to rotate the credentials, and try to keep “what they can access” scoped as narrowly as possible. Someday in the future when I can afford to hire an infrastructure security engineer, I’ll make their first task to rip out all the service account users and replace it with the OIDC thing9.

I’ll lastly mention one note about the single-node spot instance: previously I had configured my ASG to launch a t3a.medium, which uses an AMD processor, has 2 CPUs, and 4GB of RAM. My expectation was that, since this is a (relatively) small instance type, it would have pretty high availability, since AWS can probably bin-pack that more-or-less anywhere. That expectation was extremely incorrect; the k3s server node was getting disrupted between 5 and 10 times a day. Since then, I’ve expanded to allowing any of t3a.medium, t3.medium, t3a.large, or t3.large spot instance types10, and using the “price capacity optimized” allocation strategy: this essentially tells AWS “pick an instance type that is relatively cheap, but also is relatively unlikely to get interrupted”. Since I made that change, I’ve been running a t3.medium spot instance for the last 3 days with no interruptions!

Isengard

In the last post, I gave an extremely high-level overview of my Isengard (Ansible) configuration management setup. Just as Pulumi or Terraform let you define AWS resources as code, Ansible lets you define software resources as code. In Ansible, you write all of your software configuration using a combination of YAML files and Jinja templates. These configuration steps are bundles into “roles” which are then rolled up into “playbooks”. The idea is that a server can have multiple roles, each of which has its own software and configuration installed, and the playbook is what tells Ansible which roles should be applied to which hosts.

In Isengard, we have three playbooks:

1. The SimKube playbook: this is used to quickly and repeatably create SimKube environments. It’s also what’s powering our SimKube AMI and Github Action Runners, which I’ve been teasing in several posts, but will be getting its own blog post very soon now. So I won’t say anything further about this.

2. The bastion playbook: this playbook manages the “entry point” into our private VPC.

3. The k3s server node playbook, (also used to generate the AMI that packer builds for the k3s root volume).

In the previous post, I mentioned that I was using SSH tunneling through this bastion host, but that has since changed. I configured Tailscale, and I have no other words for it beyond “freaking magical.” I understand (somewhat, at a high level) how Tailscale works, but the experience has been so unbelievably good that I wish I’d done this years ago. Everything Just Works; I now have my bastion host configured to forward both DNS lookups into my private VPC, as well as to expose the internal routing tables from my VPC to any other client in the tailnet. It really is magical, there’s just no other way to describe it.

I am also running the fck_nat NAT gateway on my bastion node, to allow everything inside the VPC to talk to The Internet :tm:. AWS does actually provide a native/built-in solution for this, but it’s frankly hard to describe it as anything other than price gouging. A single NAT gateway instance costs $30/month just to exist, and then they charge you on top of that for all of the traffic that goes through the NAT! For comparison, I’m running fck_nat on a t4g.nano, which works just fine and costs me $3/month.

On the k3s server node, I am obviously using Ansible to install k3s, kubectl, and other supporting tools. I also configure the k3s systemd service here. This is what my k3s config looks like:

---
disable-helm-controller: true
disable-network-policy: true
tls-san: "k3s-server-0.uswest2.acrl.dev"
secrets-encryption: true

I disable helm because, ew gross. I disable the network policy controller because I don’t care about network policies, and that controller expects a stable IP address which we obviously don’t have. The tls-san block gives a stable domain name to issue the Kubernetes certificates for11, and the secrets encryption line enables AES at-rest encryption for Kubernetes Secret objects.

For a brief period in time I also specified a stable Kubernetes node name in this config, but that ended up causing more problems than it solved: whenever the node was interrupted/restarted, it would remember all of the state of the node from before it was interrupted, which led to some extremely weird “stale state” issues. More on this later.

For what are probably “premature optimization” reasons, this config file is not baked in the k3s AMI, but instead is generated as an ExecStartPre script in systemd; the script looks up the tls-san name from a tag on the EC2 instance, which allows me to dynamically detect the hostname and potentially run more of these clusters in the future.

There is also an ExecStartPost k3s systemd hook which is configured by Isengard that does the following:

1. Applies the EBS CSI Driver “not-ready” taint to the node (if this doesn’t mean anything to you yet, don’t worry, just keep reading).

2. Cleans up any Node resources that are left over from when the spot instance was restarted.

The latter point is necessary because Kubernetes doesn’t actually ever delete nodes that are stored in its database, it just lists them as NotReady in perpetuity. This isn’t really a problem, except that it looks ugly and takes up space. If we were not using k3s, we would configure the cloud-controller-manager to monitor the state of our AWS nodes and delete the ones from the Kubernetes database that don’t exist anymore; but k3s ships with its own, slimmed-down cloud controller manager that has some other nice features, and I didn’t want to disable that. So instead I just stuffed the cleanup into this post-run hook.

Mirkwood

Whew! We’re almost done here! We’ve got one last git repo to cover before we can have a working messaging app! Mirkwood contains all of my Kubernetes manifests, and it uses kustomize to provide the configuration management tooling12.

There are (currently) five applications running on my cluster:

1. The AWS EBS CSI driver

2. The AWS node termination handler

3. cert-manager

4. The external-dns operator

5. And, lastly, Prosody itself, aka, the thing we’ve been trying to run this whole damn time.

The EBS CSI driver watches for Kubernetes Persistent Volume Claims (PVCs) and handle the creation, deletion, mounting, and unmounting of those block devices into Kubernetes pods. This is necessary so that my XMPP/Prosody pod can have persistent data across pod restarts. The EBS CSI driver must be running before any pods that request PVCs, so the CSI driver pod tolerates the taint that we created as part of the k3s configuration, and once it’s running and healthy it removes the taint so that other pods can schedule.

The node termination handler watches for spot interruption warnings (which AWS provides with a 2-minute time window) and triggers pod and node cleanup ahead of the interruption. This is necessary so that the EBS CSI driver can actually cleanly unmount the EBS volume before all the hardware is rudely yanked out from underneath it13.

Cert-manager, as we’ve discussed, is responsible for talking to Let’s Encrypt and getting certificates for my XMPP server. It’s operating using the DNS-01 challenge mode, wherein the Let’s Encrypt issuer creates a challenge, and then cert-manager creates a TXT record in my public acrl.dev hosted zone to prove that yes, in fact, I do own the zone. Cert-manager then stores the certificate as a secret, which gets injected into the Prosody pod. It also automagically handles getting new certificates ahead of the expiry date, which is pretty slick14! I’m a big fan of cert-manger.

Lastly, external-dns handles setting the DNS entry for my Prosody pod in my private acrl.dev hosted zone. The details here are a little subtle: we create a Kubernetes LoadBalancer Service15, which normally would be handled by the cloud controller manager to create a network load balancer resource in AWS. But that stuff’s expensive! We don’t wanna run no network load balancer here. So instead, we use the builtin ServiceLB controller in k3s, which more-or-less just exposes the right ports on the host where the pod is running via iptables rules. The external-dns controller just looks up the IP address of the Prosody Service and writes that as an A record into Route53.

And now we (finally!) have everything we need in order to run our single Kubernetes pod. The Prosody pod requests a 5GB persistent volume to store all of its data, grabs a certificate from cert-manager, uses a ConfigMap to inject all the Prosody configuration, and runs as a single-pod Deployment. And, it all works!

Putting all the pieces together

There’s two bits that I didn’t cover in the above descriptions, which are a) automation and tooling, and b) secrets management. While the details are slightly different for each of my three repos, the high-level bits are the same. For automation and tooling, we use just, which I have increasingly fallen in love with over the last few years. I have justfiles in (almost) every repo, and any time there’s a complicated command that I need to remember, I drop it into the justfile. So if I needed to recreate this entire setup tomorrow from scratch it would be three commands16:

~moria > just apply
~isengard > just apply
~mirkwood > just apply

Not quite the one command I started off with in my initial goals, but honestly it’s pretty good imo. I also have GitHub Actions configured in each of these repos17 to automatically run the above just commands whenever code is changed (or, in the case of moria and isengard, on a periodic schedule to rebuild the AMIs and update the running hosts).

On the secrets management front, each of the four systems (moria, isengard, mirkwood, and k3s) have slightly different methods for dealing with them. Any secret data is stored encrypted at rest, and the user interface is more-or-less the same for each, it’s just the backend that changes. Moria uses Pulumi secrets, which are stored encrypted (using local credentials) in the Pulumi state file in S3. Isengard uses Ansible secrets, which are stored encrypted (using local credentials) in the Git repo itself. Mirkwood uses SOPS to store sensitive data in the Git repo itself, this time encrypted the AWS KMS key that we set up with Moria18. To make each of these three systems easier to work with, I have just targets configured to read or write encrypted data.

So that’s it! After several months of nights-and-weekends work, I now have a usable Kubernetes cluster that is running a single XMPP server in a somewhat reliable fashion, all for about $50/month of AWS bills. Is it cheaper than Slack? No. Is it a better user experience than Slack? Also no. Did I have fun? Lmao, nope, this was annoying as all hell to get configured. Was all the time I spent on this worthwhile, when I could have been using it to make SimKube better? Well… I think you know where this is going.

Anyways, hopefully this 4000-word guide (of sorts) might be useful to someone else down the line who is seeking to embark on a similar endeavour. Next time we’ll be returning to our regularly-scheduled SimKube content!

As always, thanks for reading.

~drmorr

Issue #1: OOMs on First?

This first issue actually sparked a long rant on Mastodon, which I started off by saying,

Sometimes problems are hard to solve because of something inherent in the problem domain that makes it hard. Other times, the problems are hard because we make it hard all on our own. This tale is one of the latter ones.

And it’s true, I stand by every word of that post. And since not all of you read Mastodon, and I’m still kindof upset, I figured I’d recap the issue here.

The problem that we’re trying to solve here is, how frequently is your pod or application running out of memory (commonly called an OOM)? This seems like a useful thing that you might want to know, because it might, say, indicate a memory leak, an increase in data or requests, or even just misconfigured resource requests. Maybe you’d want to know if this is happening several times a week, or day, or hour. Anyways, Kubernetes exposes a whole host of metrics from basically every single part of the system, and if you install kube-state-metrics (KSM) you can get even more! Surely among those hundreds of thousands of metrics, there’s one that tells you how often your applications run out of memory?

Wrong! There’s not one, there’s actually four1. They are:

1. kube_pod_container_status_terminated_reason: this one comes from KSM and is a binary value that is one if the container is terminated because it was out of memory, and zero otherwise.

2. kube_pod_container_status_last_terminated_reason: this one also comes from KSM and is a binary value that indicates why the given container terminated last, including if it OOMed.

3. container_oom_events_total: this metric comes from the Linux kernel, and reports the number of times that a container has run out of memory

4. The Kubernetes event stream: this is sortof cheating, because it’s not a metric exactly, but Kubernetes emits an event any time a container runs out of memory, and that event stream can be queried and stuffed into some other database somewhere else down the line if you want.

This all seems great and stuff, maybe there are a few too many metrics here, but we can just pick one and move on with our lives, right? Wrong2!

It turns out that none of those four metrics give you the information that you want, and thus there is actually no way—in Kubernetes—to tell how frequently your pods are OOMing. 😖😖😖

I go into details in the Mastodon thread, but lets break it down real quick here, too. Metrics (1) and (2) are both gauges, not counters, which means they can go up and down. The first metric goes back to zero whenever the container state changes3. The second goes back to zero whenever the container terminates for a second time for some different reason. That means you can’t just slap a delta function on it and call it a day, because sometimes that delta is negative! The third metric actually doesn’t work at all. It just always emits zero. There’s a GitHub issue about this where the answer is “welp, tough luck”. And to round it out, the last “metric” (event) doesn’t tell you which pod or container is affected, it just tells you “hey something on this node OOMed, oopsies!”

So there you have it. Or rather, there you don’t have it, because it’s literally impossible4 for no good reason.

Issue #2: When is closing time again?

This one’s a quickie that I learned about yesterday. Pods in Kubernetes have a deletionTimestamp field which indicates (as you might expect) when the pod was requested to be deleted5. Except! That’s not actually what it indicates.

The deletion timestamp shows the time when the pod was requested to be deleted plus the pod’s termination grace period delay6. Fine, OK, whatever, just subtract the termination grace period delay from the deletion timestamp and you get the time when the pod was requested to be deleted. A little bit annoying, but not a big deal. But wait! It gets better!

If the pod finishes all of its work and shuts down before the termination grace period expires, the deletion timestamp is updated to the actual time the pod went away. Hope you like time travelling, baby, cuz we’re going B2k to the F4e!

Issue 3: When is a map not a map? When it’s been merged!

I spent an inordinate amount of time troubleshooting this issue yesterday and today. The short version is: you’ve got some custom resource in Kubernetes that creates some pods7. Inside your custom resource, you have a pod template spec, which defines how and where that pod should be created. Inside the pod template spec, you have a node selector, which specifies which node(s) the pod is eligible to run on. The node selector is a map, it looks something like this:

nodeSelector:
  simkube.dev/type: virtual

This node selector is saying that the pod can only run on nodes that have the simkube.dev/type=virtual node label.

Now, suppose you perform the following sequence of actions:

1. Change the node selector to appliedcomputing.io/simkube-type: real (Why are you making this change? Who knows, who cares, it’s not important right now).

2. Re-apply the custom resource, using…

3. kubectl apply --server-side8

What is the resulting node selector on your newly-applied custom resource? Is it:

nodeSelector:
  simkube.dev/type: virtual

or is it

nodeSelector:
  appliedcomputing.io/simkube-type: real

or, lastly, is it

nodeSelector:
  simkube.dev/type: virtual
  appliedcomputing.io/simkube-type: real

If you guessed the last one, you’ve been doing this software thing too long, you really ought to go do something else with your life so you don’t lose what little bit of sanity you have left. But also, if you guessed the last one, you were 100% correct, thanks to the slightly counter-intuitive merge semantics of server-side-apply in Kubernetes. This might be a problem for you if, for example, it’s impossible for something to be virtual and real at the same time.

Fortunately, this last one is fairly easy to fix, you can just add a field to the custom resource specification that tells Kubernetes “hey don’t do this”. But if you’re not expecting it, I hope you enjoy spending a couple days of your life pondering increasingly-unlikely scenarios, like gremlins and gamma rays.

Anyways, that’s all I’ve got for this week! Come back next week to read more about how hard it is to actually run applications on Kubernetes.

Thanks for reading!

~drmorr

Alright, it’s been a long time since I’ve posted a good rant on here, so buckle up buttercup. This one’s about version control.

VCS? CVS? WTF?

Ok, a quick primer if you don’t do software on the regular: version control is how we keep track of dozens or hundreds of changes being made to a codebase by dozens or hundreds of engineers. It’s also, thanks to GitHub, how most open-source projects publish their work. It’s also also, mostly accidentally, how most software is backed up. The whole idea is, “we’re tired of mailing around SimKube.zip.v2.final.2.final.v0.4.no_really_final_this_time1; hey, computers are good at tracking things, what if we made the computer track our computer changes?”

The first “version control system” (aka VCS) I ever interacted with was called, confusingly, CVS2. It was (apparently) revolutionary for its time, but I only ever had bad experiences with it. See, one of the main things that version control is supposed to solve is “what happens if two (or more) people change the same file at the same time?” One could argue that dealing with conflicts like this is the sole job of version control, because if there are never any conflicts it doesn’t matter. Needless to say, CVS was not very good at dealing with conflicts.

I moved fairly quickly from CVS to Subversion (aka SVN); I still didn’t really know what I was doing, and I honestly didn’t spend much time with SVN, but I do remember that it made merge conflicts ever so slightly less painful than CVS. However, this was around the time that Git was becoming extremely popular, and once I tried Git I never looked back. The major improvement that Git made was the idea of “branches”3: you could have multiple—dozens, hundreds, even!—of different work streams going on the same codebase all at the same time, and none of them would ever conflict with each other, because they all lived on separate branches! It was truly revolutionary in my experience. Branches were cheap, lightweight, and easy to navigate back and forth, and you (the user) got to choose when you wanted to deal with the conflicts4. It was still a pain to deal with the conflicts, but at least you had some control over it.

Then GitHub came along, and became synonymous with Git, and that’s basically been the state of the world for the last two decades. Nevertheless, it hasn’t been all roses and daisies over here in Git-land: Git is a distributed version control system, which introduces certain complexities, because (who know) distributed systems are hard5. Also, especially in the early days, the Git user experience was… challenging. If you didn’t really understand the internals of the system, it was very easy to get your codebase into a bad state that was very difficult to recover from. There were limited affordances for new users to learn the system, and there’s honestly been endless ink spilled on that topic which I don’t need to revisit here. Suffice to say, there’s been a lot of effort in recent years put into making Git more approachable for newcomers, which is honestly great to see. And thanks to GitHub, basically everyone in software now interacts with Git on a daily basis, and it’s difficult to see that changing anytime soon.

So we’re done then, right? Version control is solved. Blog post over.

Wellllllll..... not quite.

See, Git itself is very unopinionated about how you use it. There’s lots of ways to accomplish the same thing task, and teams adopting Git were left on their own to figure out what workflow worked best for them. Specifically: when you write code you usually want someone else to review that code before it “goes live” or “gets deployed” or “put in front of users”. Ostensibly, at least, having a second pair of eyes on your code is a good way to catch bugs6. When you have dozens or hundreds of branches floating around (because they’re free), figuring out how to review the changes on a specific branch is a little challenging. But then GitHub came along and said “The One True Way to interact with your code is through forks, merges, and pull requests!”. And ever since then, the “pull request” model is the only thing that most software engineers have ever encountered.

Now, pull requests in-and-of-themselves aren’t awful. Basically it shows a diff of changes between your branch and whatever is “in production”, and then you can leave comments on the changes and have a discussion about them. The problem is… How to put this nicely… the GitHub UI sucks. Badly. It always has, and it’s only ever getting worse7. So while PRs themselves are fine, the experience of working with them on GitHub is unpleasant. The main issue is: someone requests that you change all your variable names arbitrarily, so then you sigh and go make the change because you’re effing tired of fighting about trivial nonsense all day every day. Then you make a new commit and give it a description like “addressing review comments”. Meanwhile, during the time you’ve been having this back-and-forth, 10 other people have made changes which means you need to merge their changes back into your branch before you can get it reviewed again and then once you’re ready for someone else to look at your code again, nobody can tell if you actually addressed any of the comments on the PR or not, and also your commit history is filled with things like “fixed things” and “merged so-and-so’s changes back in” and “fixed more things” and “wrote a test” and etc.

The thing that is frustrating to me is that it doesn’t have to be this way! All we need is a way to a) make targeted changes to a commit without having to create a new commit, and b) a way to see what changes were made since the last time we reviewed the code. But, because of the way that GitHub PRs work, (a) is incompatible with (b). And because GitHub is ubiquitous, it’s very difficult to do anything about it.

Enter jj stage right

It might surprise you to learn that a number of large, prominent companies have looked at this model and said “this sucks, we’re going to do it differently.” Google, Facebook, and a number of other companies have stopped using the PR model, and instead use a concept of “stacked diffs”. The short version is instead of reviewing an entire branch at a time, you can instead review each commit in isolation. When you’re working on a new code feature, you create a “stack” of changes, and each change gets reviewed independently. But—and here’s the crucial part—while your heartless engineers are ripping apart your current changes, you can keep developing new code on top of your previous changes. And then, when you go back to address their totally unreasonable comments, you just “drop down” to a lower level of the stack, address their changes, and then the rest of your change stack automagically adjusts itself to incorporate those changes.

It’s hard to overstate how much of a game changer this model is. People who’ve worked with stacked diffs will do almost anything to keep working with them in the future. Just like Git made branches cheap and easy, tools that support a stacked diff workflow make “editing code anywhere in your history anytime” cheap and easy. The problem is, it is extremely difficult to make this model work with GitHub.

Not for lack of trying, though: a number of tools have emerged over the years that try to blend the “stacked diffs” mindset with Git/GitHub. Sapling was (one of) the first Git-compatible VCS that tried this approach; a more modern attempt is Graphite, which basically writes their own entire UI on top of the GitHub UI. I’ve tried both (along with several others) and bounced off of them pretty quickly; they’re just clunky and don’t actually make the experience of managing code any nicer.

BUT. In recent years, a new project called Jujutsu (aka JJ) has come out of Google which is changing all of that. It is a completely new model for doing version control that is, nonetheless, totally Git-compatible, and it’s really gaining a lot of traction8. JJ does two things extremely well: first, it completely rethinks the Git UI/UX to streamline common operations. Things that take two, three, or more commands in Git are a single, well-documented command in JJ—specifically, it makes it extremely simple to navigate to any point in your change history, make changes, and then navigate somewhere else, and have everything else auto-update. Secondly, JJ makes a “conflict” a first-class concept in the system: when you make (or merge in) changes that conflict, JJ understands that what the conflict is and asks you to resolve it. However, you can defer “resolving the conflict” as long as you want; this is in contrast to Git, where if there is a conflict in your code, you must immediately drop everything and resolve the conflict right then and there before you can do anything else. Honestly, given that the sole goal of a VCS is to help manage conflicts, it’s a bit mind-blowing to me that it’s taken us this long to start treating “conflicts” as first-class citizens.

So that’s it, right? You’re using JJ for everything now, and I can finally stop reading this dumb post?

Well, again, no.

I’ve tried to completely switch over to JJ three separate times now, and I keep running into issues which make me switch back to Git. This isn’t a knock against the JJ team, because it’s a young, very ambitious project, and they’re very aware of all these issues, so I have a lot of optimism that these things will get fixed at some point in the future. But for right now, I’m unable to make the switch, which is doubly-frustrating, because now any time I’m doing something extra complicated with Git, I’m like “but this would be so easy with Jujutsu!”

For posterity, here are the three big blockers I keep encountering:

1. Pre-commit hooks: I use pre-commit extensively to do code linting, formatting, and other static analysis checks. I also run the same pre-commit checks during CI to prevent “bad code” from accidentally getting merged. Unfortunately pre-commit hooks don’t make a lot of sense in JJ world; there are some efforts to support “pre-push” hooks instead of “pre-commit” hooks, but none of those have actually landed yet, and every time I push my code up to GitHub and then realize that I forgot to run my checks locally first and they’re all failing, I get real sad.

2. Branch management: while JJ itself doesn’t care about branches (called “bookmarks” in JJ-land), GitHub still cares about branches A LOT. But because JJ doesn’t care about branches, the tooling to keep branches in sync with GitHub is lacking. The tug alias that some users have come up with does help some, but it doesn’t always work, and keeping your bookmarks correctly pointed at the right code is a lot of manual juggling that I find to be really disruptive. Again, I think the JJ team is aware of this and is working on it, but it’s not there yet.

3. Merging code: This is related to the previous point, and is the reason I bounced off JJ most recently. I normally use a “rebase” workflow on GitHub, to avoid a bunch of pointless and unhelpful “merge commits”. But when you rebase and merge an external system like GitHub, JJ is not currently able to track the ways in which your code has changed, and you end up with a whole bunch of dangling references/branches that (again) you have to clean up manually9, which again is a bunch of frustrating busywork.

There are some other issues that I have with JJ, mostly around the learning curve for its revset language and other configuration aliases/options10. But honestly, a steep learning curve doesn’t bother me that much, I can get over that. It’s the more fundamental “workflow issues” that are the bigger problem right now.

All that said, I’m very excited for the future of Jujutsu and I will probably continue watching its changelog and trying it out every few months until some of these issues get resolved. It really does feel like the first genuine, foundational improvement in how we do version control in twenty years.

As always, thanks for reading!

~drmorr

Let’s talk about subscribers! As of today, there are 225 subscribers to this blog1, and 14 of those are paid subscribers. This is pretty great! I’m excited that all of you are out there reading this…. whatever it is, and even more excited that 14 of you are willing to financially support the work we’re doing at ACRL.

As I’ve said before, this publication doesn’t pay our salaries, but it does help cover for our conference and travel budget, which gets us new clients which do pay our salary. However, managing the accounting for these paid subscriptions does take up a non-trivial amount of time, especially around tax time. Given that we don’t have a huge number of paid subscribers, it has been an ever-present question in the back of my mind if it’s worth the effort2.

So, for the remainder of this year, I’m trying an experiment: I’d like to increase the readership of this blog in general3, and specifically, I’d like to increase the number of paid subscribers. I feel like if we could get between 50 and 100 paid subscribers to the blog (which would bring in around $2500-$5000/year), that would be enough to make the accounting effort during tax season worth it. But, if we can’t get there, I don’t think it’s worth my time to manage the paid subscriptions.

So that’s my goal: for the rest of the year, I’m going to be trying a bunch of experiments with the blog to try to increase readership in general, and paid subscriptions specifically. All the content here is still going to be released to the public on Monday afternoons, but I’m going to be experimenting with some other “paid benefits” for subscribers, and we’ll see what happens!

Also, if you’re currently one of my free subscribers, and you feel like you’ve gotten some value out of what I’ve written, would you consider upgrading to a paid subscription4? You’ll get a warm fuzzy feeling in your heart, the ability to comment on my hot garbage, and maybe some other benefits in the near future.

Subscribe now

Thus concludes my meta-blog-intermission! As always, thanks for reading.

~drmorr

Put all this together, and things get really confusing really fast, especially when your CA engineer needs to work with his CA counterpart to write a CA because the CA couldn’t get new credentials from the CA, which is a thing I have definitely never had happen to me.

All this is to say that ACRL has been a CA for a while: we’re a California-based company that has definitely autoscaled some clusters, and given the circumstances of my departure from my previous gig, you could argue that ACRL is also a corrective action. But there’s one CA that we haven’t been before, which is a Certificate Authority; however, late last year that changed! In this post I’m going to talk about the why and how of it all.

Why on earth do you need to issue certificates?

OK, so let’s take a step back: what is a certificate12? I alluded to this briefly in the introduction, but a certificate is a cryptographic primitive for performing encryption and authentication. It has two parts, a “public key” and a “private key”; the public key, as you might expect, can be shared publicly, but the private key needs to be kept secret. There’s a lot of math involved which I won’t go into, but the basic idea is that you give your public key to someone else, and then you can prove to them that you are the owner of the private key by decrypting a bit of data that they encrypted with the public key. This is one of the underpinnings of the modern internet; most websites these days use public key cryptography to provide a secure connection to their site, and in fact many browsers will flash scary warnings at you if you visit a site that doesn’t use this security measure.

It turns out that this is just the beginning; for a long list of reasons, we’ve established a chain of trust with these certificates, so that very often you’re no longer verifying that you have the private key, you’re verifying that you have the private key and that private key is trusted by some third party. The third party has their own certificate, which is trusted by another third party, and so on and so forth. These third parties are called “Certificate Authorities”, and one of the reasons why they exist is to make certificate revocation easier.

Imagine, for example, that you accidentally shared your private key on GitHub. Whoops! Now anybody who happened to look at your GitHub repo while it was there has your private key, and they can pretend to be you! They can send encrypted messages as you or pretend to be you when visiting websites. You can take the private key down, but it would be really great if there was some way to signal to the entire world that if anybody ever uses that private key again, they are a bad person and should feel bad. That’s (one of) the functions of a Certificate Authority: they maintain revocation lists where you can look up and see whether a certificate is still “trusted”.

Again, all very cool technology, based on a lot of interesting math, but why is ACRL a Certificate Authority now? Well, the answer is simple: SimKube.

Oh come on. You’re telling me your open-source Kubernetes simulator needs to be able to issue certificates?

Well, not exactly. See, SimKube itself isn’t very useful on its own; if you wanted to, for example, compare Cluster Autoscaler and Karpenter, there are a lot of extra components that you might want to install to make that comparison easier, and some of those components might be hosted in a private container registry on AWS. So you need some way to give folks access to that container registry. Now, AWS has a way to manage permissions and authentication already: it’s called IAM (Identity and Access Management), and it’s some of the most arcane nonsense you will ever have to deal with34.

Fortunately for us, AWS provides a way to make it even more arcane: IAM Roles Anywhere. The sales pitch for IAM Roles Anywhere is, essentially, “What if we took an incredibly complex permissions management system and hot-glued an incredibly complex cryptographic identity scheme on top?”

The nice (????) thing about this, and the reason why ACRL is now a CA, is that it gives us a one-step process to grant access to parts of our private AWS account. If a client needs some internal tool or component to make simulation easier, all I have to do is send them a certificate. They don’t even need to have their own AWS account! They just install the cert and they’re off and running. And then, at some later point when they no longer need access to ACRL’s AWS account, I just revoke the certificate and AWS won’t let them in anymore. Cool beans!

I’m still not convinced that any of this is necessary, but OK, at least tell me how you did it.

The process of setting up a certificate authority that works with AWS IAM is non-trivial; there’s quite a lot you need to think about, especially if you want to do it securely. Fortunately for us, someone else already did all the hard work! A German company called Q-Solution has published an open-source Terraform module for creating a certificate authority and easily using that authority to generate public/private keys.

The steps for setting it up were relatively straightforward5:

1. First, I created a second AWS account for the CA; these certificates aren’t guarding anything particularly sensitive right now, but they definitely could in the future, and if an attacker somehow gains access to my primary AWS account, I don’t want them to be able to muck around with certificates. This is probably overkill, honestly, but as one of my trusted colleagues has repeatedly told me, this whole CA scheme is insane and ridiculous overkill, so why not go all in on it?

2. Once I had the second AWS account, I really wanted to make sure that I got notified whenever anybody used the account for anything. For this purpose, I created a CloudTrail log (you get one free!), and then set up EventBridge to send notifications from CloudTrail to the Simple Notification Service (SNS), which emails me. Note that CloudTrail is configured in the main account (aka the management account), but it actually aggregates events from all the different accounts.

This was incredibly annoying to get working: ACRL is using AWS Single Sign-On (SSO) to access both the main account and the CA account; when you sign in to an account with AWS SSO, it logs a Federate event to CloudTrail. But also, users can access the CA account from the CLI, which doesn’t go through SSO, but instead uses an GetRoleCredentials call; so I needed to monitor two separate types of events, confusingly, neither of which are an AWS API Call, but instead are an AWS Service Event via CloudTrail6. And lastly, it turns out that GetRoleCredentials is a read-only event, which isn’t tracked in EventBridge by default; instead, you have to turn on tracking of read-only events in EventBridge, and the process to do this is a semi-secret flag that you can only set from the command line, and not through the AWS console7.

Anyways, once I got all that set up, I decided I hadn’t gone deep enough down the rabbit hole, and also made this janky “login monitoring system” alert me if anyone logs in using the AWS root user. Because why not. But finally, we can move on to

3. Point the certificate authority Terraform module at my brand-spanking-new AWS account, run it, and immediately get a million emails saying “Someone just logged into your AWS account!!!111!1!11one”

The certificate authority module is pretty nice actually; when you first set it up, it runs a bunch of AWS Lambda functions to generate your root certificate and signing certificates, and then anytime I need to generate a “client” certificate, I can just use the provided Python script to do so. Managing the revocation list is also not too bad, any time I need to revoke a certificate I just add its SHA to the revocation list, and trigger the “revocation Lambda”, and voila, nobody can use that certificate to access my account anymore. Pretty neat!

Huh. I guess that is kinda cool.

Thank you. I’m glad you finally agree. Anyways, the best part of all this is that now any time anybody wants to know if ACRL is a CA, I can confidently answer “Yes”, regardless of what definition of CA they are referring to8.

As always, thanks for reading.

~drmorr

Ok, before I start off I need to acknowledge the reddit user who complained that my post last week was “hot garbage.” Thank you for your kind words! I have never aspired to produce anything else, but I do want to point out that everything I write here is 100% hand-crafted, human-written hot garbage. I put up AI artwork sometimes but the words are all real human words. Just in case there was any confusion about that.

Anyways, in this week’s episode of hot garbage, I want to talk about software versions, but maybe not in the way you might be expecting. There’s a well-known class of “Software Blog Post” arguing about versioning schemes: “SemVer sucks! Use CalVer!” “No, CalVer is terrible, just use WTFVer!” And etc, ad nauseam, vim vs emacs style. That’s not what this post is about. Instead I want to talk about the psychological effects of software versioning12.

Releasing new versions is scary…

There’s a running joke/meme in the Rust community that none of the crates or libraries in the Rust ecosystem will ever reach “version 1”, and will be in a weird “alpha” state in perpetuity. It’s kind of funny that many foundational libraries are on version 0.1884.23, but I think (based on my own observations) that’s actually a symptom of a deeper psychological issue: namely, “releasing a new software version is scary.”

I tend to believe that as a general rule, scientists and engineers want to produce good things that are high quality and that will make people’s lives better. And we’ve created this weird association between “putting a version tag on something” with “this thing is ready to go”. And I can kinda understand how we got here: back when new software was released on a physical disk that you had to go into a store and purchase3, there was a lot of pressure to make sure that the bits contained on that physical disk were perfect, because if they weren’t it was extremely difficult to fix them after the fact4. And even now, when we put a new version on something, we’re kindof implicitly saying “This set of bits is ready for consumption, all those new features I was working on are complete and ready for someone to use.”

And that’s kinda scary! You’re putting yourself out there in a way that feels a bit uncomfortable. What if it’s broken and busted, or there’s some bug or corner case that you didn’t think about, or, or, or…?

…but it doesn’t have to be.

But here’s the thing: whatever versioning scheme you’re using, the numbers are free. It’s not like we’re going to run out of numbers. If you find a bug or an issue with version X.Y.Z, you can just fix it and then release version X.Y.Z+1. We no longer have a software distribution problem like we used to5. It doesn’t have to be scary to release a new version.

I’ve been thinking about this a with one of the internal tools I’m developing for use with our clients; there’s been an aggressive set of feature development and bugfixes, and this has resulted in an aggressive set of new versions so that my clients can get access to those new features/fixes. I’ve often released several new versions in a single day, which feels… a little excessive. But at the same time, like, who cares? Number goes up. Install the new version. Life goes on.

I also ran into this with SimKube yesterday: I released a new patch version of SimKube, version 2.4.3. It was a small bugfix to ensure that simulated DaemonSets get scheduled on the right subset of nodes. I wrote some tests, they all passed, I released the new version, I installed it… and it was broken. UGH. I went through this whole narrative in my mind: “What kind of hack job am I? I can’t even release a new software version correctly. I know, nobody else is using this, I can just cover up how incompetent I am, I’ll just yank the version, force-push a new change that fixes the bug, and re-release 2.4.3. Nobody will know!”

But then I paused. Actually, really, who cares? Nobody is staring at my release notes going “Wwwwoowowowoww drmorr really screwed that one up!” Nobody cares. I fixed the bug, I fixed the test that should have caught the bug but didn’t because I wrote it wrong, and I released SimKube 2.4.4 an hour later. Problem solved, life moves on.

Anyways, the whole point of this post is that we should normalize this: version numbers are free, and we’re not going to run out. Don’t feel bad if your last version has a bug. All software has bugs. Fix the bug, make a new version, and move on with your life.

Long list of caveats

Because I just know that someone is lurking in the background trying to decide whether to sign up for a paid subscription so that they can call this post hot garbage in the comments, I do want to acknowledge that, while version numbers are free, sometimes “releasing things” isn’t. There are a whole bunch of settings where it’s still somewhat challenging to get your new version into the hands of your users: mobile development (or really anything that has an “app store” or “marketplace”) typically has a long lead time for releasing new things, because it has to go through some kind of third-party audit, whether that’s just “automated tests” or “human review”. And even after you’ve released something, many people can’t or won’t upgrade to the new version, so you’re kindof stuck maintaining the old thing for a long time anyways. See also: air-gapped systems, embedded firmware, stuff that is literally going to the moon, etc.

The point of this post is not to say we should just throw shitty code out there willy-nilly because you can fix it later6. The point is to say, “do the best work you can, and when you think it’s ready, don’t have so much anxiety about releasing it.” I’d way rather have people using my slightly-buggy code, and have to release a new version to make it slightly less buggy, than to have nobody ever use it because I’m too scared to make the new version.

So anyways, to sum up: version numbers are free, we should do more of them. As always, thanks for reading, and tune in next week for more hot garbage from ACRL, Inc!

~drmorr

Hello! Hope you’re all having a great start to your new year. As I hinted in my recap post, there are a lot of exciting things happening over here. In this post, I want to talk about one of them: a fun project I worked on over the holidays1 and in the early part of the year to create ACRL’s very first permanently-running Kubernetes cluster2!

I’m expecting you all are having two distinct reactions to the above statement: a) “You’ve been in business for 2.5 years as Kubernetes experts and you don’t even have a Kubernetes cluster? What kind of hackjobs are you?”, or b) “Don’t you barely even do any real work? Why on earth would you need a Kubernetes cluster?”

My answer to those questions are “The best kind,” and “We were getting bored over here.” Anyways, in the remainder of this post, I’m going to pull back the curtain on some of our internal infrastructure and talk about how we got here.

No, but really–why do you need a Kubernetes cluster?

There’s an oft-repeated adage of sorts in the industry that “most companies” don’t actually need Kubernetes, and that trying to run a k8s cluster just adds a bunch of complexity that will distract from your core business offering when you’re small. I actually don’t totally disagree with this viewpoint, and have repeated it myself from time to time, but there are two specific reasons why I decided to ignore this advice for ACRL:

1. We actually are supposed to be Kubernetes experts, and it is a good use of our time to make sure we stay up-to-date with how to run and operate the technology we are supposedly expert at.

2. As we’ve grown, I’ve built up a lot of “bespoke” services, and there are a bunch more that I want to run: for example, my website exists as a janky mess of Docker Compose files, and it’s always a little nerve-wracking to make changes to it. I also want to start hosting things like “an internal message/chat app” or “VictoriaLogs, because we already generate more logs than is humanly-feasible to comprehend”. And what I’ve realized is that I can have N sources of complexity to manage each of these N services/applications in slightly different bespoke ways, or I can have 1 source of complexity that manages N services for me in a uniform way. It’s the classic “pets vs cattle” problem; and while ACRL is admittedly still quite small, it turns out that it doesn’t take a very large number of pets before you have too many pets. Kubernetes provides a well-supported platform for turning all of your dogs, cats, fish, and turtles into cows3.

So, given those two things, I’ve wanted to get this cluster up and running for a while, and finally decided it was time to take the plunge.

So what’s your architecture, bro?

Now, I knew going into this that running a Kubernetes is not for the faint of heart. It does have a huge amount of complexity, and it’s almost guaranteed to cost you an arm and a leg4. So my goal when setting the cluster up was to do it as cheaply and with as little human intervention needed as possible. In this section we’ll talk about my architectural choices so that everybody on the Orange Site can mock and ridicule me for doing things the wrongest possible way.

What Kubernetes distribution?

There are a LOT of different options for “how to run Kubernetes” ranging from “just do the steps in Kelsey’s book, don’t worry, you’ll have a working cluster in 99 years” all the way to “just pay out the nose to run on EKS”. I landed somewhere in the middle: I don’t want to pay out the nose to AWS, and I want/need access to the control plane (which you don’t get with EKS), but I also want something that does “most” of the work for you. For this purpose I elected to use k3s, which is a lightweight Kubernetes distribution that bundles all of the components (API server, controller-manager, scheduler, kubelet) into a single statically-linked binary that you can “just run”5.

Using k3s makes setting up the cluster itself extremely easy: ship a binary somewhere and run it, done. The default configuration is even pretty sane out of the box6! The bigger question is how to set it up in a persistent, reliable, automated, and inexpensive way. And it turns out that this is where all the complexity lies.

A descent into Moria

Those of you who know me know that I’m a huge Lord of the Rings geek. So when I initially set up all my internal infrastructure tooling at ACRL, I didn’t have to reach far to come up with names for my repositories. We have two internal repos for doing “infrastructure as code” and “configuration management”. The first repo is named “moria”, and handles all of our IaC needs using Pulumi7. Moria is where all of our AWS config is managed: S3 buckets, EC2 instances, etc. The second repo, named “isengard”, uses Ansible8 for configuration management9: in other words, what software, tools, and configuration files should be installed on my hosts.

So, when I started getting k3s set up, the path seemed10 pretty straightforward: launch an EC2 instance in moria, and install k3s on it via isengard. The first obstacle appeared when I started looking at costs: running a single EC2 instance with 2 CPUs and 8GiB of RAM would cost me ballpark $60/month (and that doesn’t include storage). And I want (eventually) a whole cluster of these things! So I made the design decision to run the entire cluster (including the control plane) on spot instances11. Doing this lets me run a single node for ~$10/month (depending on spot price fluctuations). That’s much better!

The only problem is that spot instances can be taken away literally at any time, and I want my cluster to be (somewhat) resilient to disruption. Obviously the “best” way to do this would be to run etcd, a distributed datastore that is designed for resilience, but that would add a whole bunch of complexity and expense that I’m not ready for yet, so I took a middle ground: the k3s data volume would be a persistent EBS volume that gets automatically re-attached any time the instance restarts. So add another line of code to Pulumi and off we go!

Oops! Turns out that storage is expensive. A 100GB EBS volume (probably the minimum of what would be “acceptable” for this use case, once I get things actually running on the cluster) costs $8/month, and a 30GB root volume adds another $2.40. Welp, my costs just doubled. ALSO, I just made life harder for myself, because there’s no built-in way to re-attach EBS volumes to hosts in the event of disruption.

No big deal though, we can just write some more isengard code to handle this. The way we handle this is, each k3s host is tagged with the ID of the EBS volume that “belongs” to it; a couple of systemd scripts run on instance startup to look up the tag, attach the volume, mount it to the right place, format it if necessary, and then start k3s. Easy peasy12!

The next problem we need to deal with is that the API servers need to have a stable network address if we’re planning to access them from anywhere “external”. My first thought was “DNS is for lusers, and also it’s always DNS”, so I tried to just assign a static private IP address to the control plane instance. However, because we’re running the control plane node as a spot instance we have to stick it inside an ASG if we want the instance to automatically re-create itself on disruption, which means that—even though the ASG has a max size of 1—we can’t assign a private IP address to it. So, back to DNS it is.

But now we have a problem: how do we update the A record for the instance? Eh, no big deal, we’ll just use throw more money at AWS and use an internal hosted zone in Route53, and create a ASG lifecycle hook and a lambda function to update the A record on instance launch. Net additional cost: a few cents/month.

AMI dreaming?

The last step in the process is answering the question, “how do we get k3s re-installed on the nodes when they’re disrupted?” The naïve solution is “just re-run the Ansible playbook on boot”, but this is obnoxious because it a) uses up a bunch of compute credits for my burstable EC2 instance, and b) it takes even longer for the control plane to become available after interruption. So the solution here is to use packer13 to bake an AMI14 with all the required software packages installed. This part actually was pretty easy, thanks to all the hard work Ian has been doing to create a SimKube AMI for GitHub Actions. It took a half-hour or so to modify the AMI baking pipeline to make it more generic, and now we’re baking an AMI for k3s as well! We do this weekly, so we can pick up security updates and other package updates to the underlying OS.

Of course, we don’t have anything to actually clean up stale/old AMIs, and turns out that these snapshots are also kindof expensive, so currently I log into our AWS console every couple of weeks and delete all the old ones, a process which is completely sustainable from now until the end of eternity. Net additional cost: another $5/month or so, probably.

We do want the k3s node(s) to reference the most recent AMI when new ones are launched or old ones are disrupted, and we also don’t want to have to manually update the ASG Launch Template for k3s to point to the new AMI. Fortunately, AWS provides a key-value store that you can reference inside a launch template so that it always points to the latest AMI. All we have to do is update the value stored there whenever a new AMI bake is complete, and problem solved. Fortunately because AWS is so benevolent, this service is free.

Cool Kubernetes cluster, bro; what’s it do?

So there you have it! ACRL is running a (one-node) Kubernetes cluster for around $35/month. Not too bad if I say so myself.

What? What’s that you say? Is it actually running any services or applications? Lmao, of course not. That stuff costs money! Also I got nerd-sniped this week into solving another totally different problem which will absolutely become its own blog post sometime in the future. So, yes, we are in fact spending $35/month for nothing.

Also, there’s the little, small, tiny—miniscule, really—problem that the cluster is living inside our private VPC, which means it’s not actually accessible to anybody from the outside. My current solution to this is to use SSH forwarding and a manual entry in /etc/resolv.conf to point to the internal VPC DNS resolver. This is… not actually sustainable, and isn’t even worthy of being called an “interesting choice”, it’s just dumb.

I think the solution here is to set up TailScale, but that’s another $6/month and a whole bunch more configuration that I don’t have any time for right now. So instead, we’re just gonna keep running a pointless Kubernetes cluster that does nothing for the foreseeable future. But maybe at least you can use this article to inspire some “interesting” architectural decisions of your own.

As always, thanks for reading!

~drmorr

Whale hello there, happy new year! I hope you all had a good holiday season! It’s been a minute since I’ve written anything here—I’ve been meaning to write this post ever since, well, before Christmas, but I had a bunch of travel and illness at the end of last year, and it’s taken me.... several weeks, lol, to get back into the swing of things in the new year. In any event, we’re back now! As is tradition, I want to take this post to do a recap of last year’s significant accomplishments and events, and then briefly discuss my goals and plans for the upcoming year.

Before jumping in, I do want to briefly acknowledge something: the world is a fucking shitshow right now. I don’t talk about it much on my blog, but things are hard out there, and I do really hope you are all taking care of each other. And, while I am personally optimistic about ACRL, I am… less optimistic about other things. It’s a difficult balancing act to hold both of those things at the same time, but I hope that the work that we do at ACRL is, in some small tiny way, an act of resistance against all of the bad things out there.

Anyways I don’t know how to segue from that, so I’m just going to do the world’s most awkward transition: anyways!

2025 by the numbers

As always, we’ll start off by looking a bunch of statistics and numbers that are, ultimately, meaningless.

• The big news of 2025 was that the company doubled in size! Ian has been doing a fantastic job, and I’m very happy to have him on the team. Looking forward to lots more good stuff this year!

• I only had 26 meetings in 2025 (where I remembered to take notes). That’s a 17% decline in meetings from last year, which I feel like is a net improvement in things that could have been emails (or naps).

• I completed 296 issues on my task list! That’s, like, 5 tasks a week, or just about one per day! If you want to work with somebody who is really good at creating tasks and then crossing them off, I’m ur guy! In a (slightly) more serious vein, in last year’s recap I mentioned needing a better routine for task tracking. Early on the year, I switched to Linear for task tracking, and I’ve been very happy with it. It’s pretty lightweight, easy to use, has some nice GitHub integrations, and is fairly inexpensive. It also has a good set of keyboard shortcuts, which I’m still learning but make your life easier once you get the hang of them.

• In terms of code written, I only made 483 contributions on GitHub. This is slightly lower than in 2024 (which was, in turn, slightly lower than 2023). If we extrapolate this out, this means I will stop writing code sometime in 2030. Presumably this means I will have struck it rich and can retire. Thanks, linear algebra!

• We can also look at “lines of code written”, which we all know is the best metric for software engineering, and it totally un-gameable1. Looking at my major repos, we have a total “source lines of code” count of 107,630. This is 90k more lines of code written in 2025! According to my estimates, this cost about $3 million to develop, took 75 months, and required 30 people. And this doesn’t even count work that I did for my clients! Not bad for a couple o’ hacks2. If you want to hire somebody who knows how to write lines of code, get in touch!

• The above two statistics don’t include any of the private work I did for clients in 2025; I had two clients that I was working with, and that kept me very busy throughout the year. Honestly it’s a miracle I managed to get any other code written at all, lol. I did have the opportunity to do some very successful work for Astronomer, a small company you may have heard of.

• By the way, just in case you were wondering, ACRL is now a Certificate Authority!

• My blog was a little less active in 2025 than it was in 2024; I only wrote 20 posts last year, but I also produced a whole heck of a lot of video content3, which is a new area for me, and is a lot harder and more time consuming than written content. Even so, I have been slowly but steadily adding new subscribers: I have 220 total subscribers, up 30 from last year! And 11 of you are paid subscribers: as always, thank you so much for your support. I don’t finance my business off of Substack, but I do finance my coffee addiction, and that’s not nothing. We also had two (2) posts make it big on the Orange Site: How to log in to ECR from Kubernetes the right way in April, and and Make the Easy Change Hard in August. So if you want to view content that the Orange Site thinks is valuable, uh, check those two out I guess.

• I’m still keeping up with the conference attendance: I was at the SoCal Linux Expo in the spring, and (of course) KubeCon in the fall! I also made it to the Kubernetes Community Days event in San Francisco, which was a delightful smaller/more focused event than KubeCon is. My talks at KubeCon didn’t get accepted, but I was able to give a talk as well as an impromptu lightning talk at Cloud Native Rejekts before the main KubeCon event. Also at KubeCon, we sponsored the first-ever “Can U Fit In The Kube?” challenge, which was ridiculous and delightful and a ton of fun.

• My first clinic project with Harvey Mudd College wrapped up in spring of last year, and was a great experience! I’ve been sponsoring another project this year, which is also going very well; I have a great team that I’m working with, and I’m very excited about the project that they’re working on.

• Towards the end of the year, ACRL published its first public postmortem for a product that is (still) not publicly available4. We just want to re-iterate that we’re deeply sorry for the impact that this issue caused, and to emphasize that we are making changes to prevent this from happening again.

• Lastly, I just want to shout out to the ACRL BOD5—I’ve had an informal BOD ever since the company started, and I would not have gotten anywhere near as far along in this endeavour without them. But, last year, we took a step towards making the informal BOD (slightly) more formal: I’m now paying them for their time! I also added one additional person to the BOD, who I am extremely excited and grateful to have. Thank you so much for all your help so far.

There were a lot of other things that happened in 2025, too many to write about here, but just know that no matter what pointless metrics you choose to measure things by, it was a very successful year for the business!

Plans for 2026

Let’s also take a brief look forward as to what’s coming in 2026! I’m pretty excited about the upcoming year, I think there’s some great things happening, and I can’t wait to share them with you:

• SimKube development continues: if you’ve been following along, you might have noticed that “core” SimKube development has slowed down quite a bit, which honestly is a good thing. We have a core piece of software that works pretty well and has had a lot of bugs and kinks ironed out over the last year. There are a bunch of planned new features for SimKube, but the main focus for the last 6 months has been “how do we make this thing easier to use?”, and I expect this to continue into 2026. ACRL is (still) probably the only team in the world that can actually use SimKube effectively, and I’d love to make it so that this is not the case. To this end, we have a number of projects in flight:

  • A GitHub action runner that you can plug into your CI pipeline so that you can run simulations of your project on production data

  • A scraping tool similar to kemu that allows you to more easily replicate or clone a production environment into a simulation environment

  • A variety of other quality-of-life improvements to make the system easier to work with

  • [STRETCH GOAL] Maybe some kind of UI? I am increasingly coming to the conclusion that to really make SimKube take off, it needs a way to interact that isn’t just a CLI. We need ways to inspect, modify, and extend trace files, we need controls to start, stop, and re-run simulations, and we need dashboards to see the results of simulations. All of this is possible right now, but having a unified interface will be a huge improvement. The only slightly concerning challenge with this goal is that I am not, have no interest in, and will never be a front-end person, so to make this a reality I’m gonna need to find someone to do it for me.

• Client work will also be continuing in 2026: at the encouragement of my BOD, I am exploring ways to transition away from consulting work and move more towards “having a product I can sell”; this has always been the long-term goal, but I also do need ways of paying the bills. So I’m happy to say that I have one client that I’m continuing to work with in 2026, and several other potential clients in the sales pipeline! I’m hoping to have some more blog posts to share based on client work in 2026 as well.

• We6 are also spending a bunch of time ~writing a lot of YAML~ shoring up some of ACRL’s core infrastructure. It’s probably overkill, but I’m trying to start incorporating as many “best practices” for infrastructure management; we’re using Pulumi for our infrastructure-as-code tooling and ansible for configuration management. On this front, ACRL is actively working towards running its very own permanent Kubernetes cluster, which I will have a very exciting blog post about in the coming weeks.

• Conference attendance will continue in 2026 until morale improves. I’m going to be presenting at SRECon in Seattle in a couple months; I’m very excited to go back to Salt Lake City for KubeCon in November, where we will be presenting “Can U Fit In The Kube 2”; and I am debating attending RustConf in Montreal for the first time! I’m also hoping that KCD SF will happen again, because I had a great time there. In any event, there will be lots of opportunities to meet up with us in 2026.

• Blogging will continue in 2026 until morale improves. I am very grateful to all of you for continuing to read this publication, and I really enjoy getting to share the work that I’m doing with you all here. I’m also hoping to get Ian regularly contributing to this space in 2026, so you can get to know him a bit better as well.

I’m sure 2026 is going to have its own twists and turns as well, but at least to start off with, that’s where we’ll be going! Follow along this year to learn how the plans worked out :)

Warpping Up

These wrap-up posts are always one of my favorite things to write on the blog: when you’re in the weeds as a business owner/small business person, there are a lot of hats to wear and a lot of things to do, and sometimes it can be frustrating and demoralizing that “we’re not moving faster”. So this is always a great way for me to look back at the year and go “Wow, we really did accomplish a lot!” It’s very validating to see: I’m two and a half years into this journey, and I’m so excited to see how successful it’s been so far. Of course anything can happen, who knows if this thing will still be around next year, blah blah blah, but I’m optimistic and excited!

On Wednesday, November 26, 2025, while testing changes to ACRL’s SimKube CI Runner1, an ACRL employee discovered an intermittent failure in the runner. This failure caused approximately 50% of the simulations scheduled on the runner to fail, resulting in failed actions in users’ CI pipelines, which prevented new deploys of mission-critical code. We at ACRL take our responsibility as the world’s leading provider of Kubernetes simulation analysis very seriously, and we understand the severe impact this incident had on users of our CI runner. We deeply apologize for this incident, and are committed to taking whatever actions necessary to restore trust with our customers. In the remainder of this post we will outline the timeline of this incident, a detailed analysis of the underlying causes, and the remediation steps we have taken to prevent a recurrence of this incident.

Timeline of events

The aforementioned ACRL employee discovered the issue late Wednesday afternoon on the 26th. However, because the following day was Thanksgiving, the investigation was postponed until the following week under the hypothesis that it was likely a transient error, it’d probably go away if we didn’t look at it too hard, and we had a lot of Thanksgiving food to eat.

On the following Monday (December 1st), during our regularly-scheduled company all-hands, we re-triggered the CI pipeline once and it succeeded, whereupon we decided the problem had fixed itself. It wasn’t until Thursday, December 4th, when the incident re-occurred that we decided to bother spending some time investigating. We then spent most of the afternoon troubleshooting until we found the inciting factors2 and identified a series of remediations. Those fixes were published at some point later on, when we got around to it.

Background and terminology

SimKube is ACRL’s simulation environment for Kubernetes. It is designed to allow organizations to study changes in their production Kubernetes clusters in a safe and isolated environment. One way of using SimKube is as a dedicated step in CI pipeline; this would enable users to check for regressions or bugs in their Kubernetes code before it is deployed.

The SimKube CI runner is published3 as an Amazon Machine Image (AMI)4, which contains a complete SimKube environment. The runner can replay trace files contained in the codebase, and will check the outcome of the simulation to see if it’s Succeeded or Failed. The symptoms of this incident were that periodically, a simulation would report as “failed” after completing its entire run. The SimKube driver pod (the component responsible for running the events in the trace file) would report the following error, along with a stack trace and a panic:

timed out deleting simulation root sk-test-sim-driver-sn295-root

The “simulation root” is a Kubernetes custom resource which acts as a “hook” to hang all the other simulation objects off of. The simulation root exists to make for a one-step clean-up procedure: because of Kubernetes garbage collection, when the root is deleted, all objects owned by the simulation root will also be deleted.

Detailed analysis

The first step we took in our investigation was to study the trace file running in the simulation. This trace file (also available as an example trace in the SimKube repo) creates a single CronJob, lets it run for three minutes, and then deletes the CronJob. The CronJob is configured to create a new pod every minute, and the pod sleeps for 30 seconds before terminating. This trace file is used to test the pod lifecycle management features of SimKube.

We investigated the log files from all the relevant controllers, including the SimKube driver pod, the Kubernetes controller manager, and the Kubernetes API server. The results were, to use the technical terminology, extremely f*$&ing weird. The SimKube driver pod had dozens of log lines which looked like the following:

INFO mutate_pod: mutating pod (hash=10855072724872030168, seq=66) pod.namespaced_name=”virtual-default/hello-simkube-29414550-tcr49”
INFO mutate_pod: first time seeing pod, adding tracking annotations pod.namespaced_name=”virtual-default/hello-simkube-29414550-tcr49”

What do these lines mean? Well, the SimKube driver registers itself as a mutating webhook so that it can redirect simulated pods to the fake nodes and apply other labels and annotations to them. The hello-simkube pod is the one that’s owned by the simulated CronJob. What’s curious about these log lines is that they repeat over, and over, and over again, even after the CronJob object itself has been deleted! At first we thought this meant that the CronJob hadn’t actually been deleted, but after some further study we realized that the pod name was the same for every single one of these log entries: in other words, the SimKube mutating webhook is trying to mutate the same pod for 10 minutes, well after the simulation was over and everything (supposedly) had been deleted.

The next clue came from the Kubernetes controller manager logs:

 “syncing orphan pod failed” err=<
        Pod “hello-simkube-29414550-tcr49” is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations), `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
        @@ -140,7 +140,9 @@
          “TerminationGracePeriodSeconds”: 30,
          “ActiveDeadlineSeconds”: null,
          “DNSPolicy”: “ClusterFirst”,
        - “NodeSelector”: null,
        + “NodeSelector”: {
        +  “type”: “virtual”
        + },
          “ServiceAccountName”: “default”,
          “AutomountServiceAccountToken”: null,
          “NodeName”: “cluster-worker”,
 > logger=”job-controller” pod=”virtual-default/hello-simkube-29414550-tcr49”

This is a standard error that gets returned when something (a user, a controller, etc) tries to update a read-only field. In this case, it’s showing that something is trying to update the pod’s node selector after the pod has already been created, which is not allowed. There are two curious things to note in this log entry: first, the timestamp is after SimKube has deleted the CronJob, and it states that the pod has been orphaned, which means it’s not owned by anything. In other words, the CronJob really was deleted! Secondly, we got lucky in that some of the additional context shows that the pod has been scheduled to a node, that is, cluster-worker. This is not one of our simulated nodes! This is a real node! That shouldn’t happen.

The last clue came from the API server logs, where we discovered that the SimKube driver mutating webhook had been configured to fail open5. This means that, if the webhook fails (for whatever reason), the pod object will be allowed through anyways. Specifically, we saw that the webhook was failing because of a certificate error.

The certificate error immediately cast suspicion on cert-manager, which is the component that manages all of the TLS certificates for SimKube. Cert-manager is quite a complex bit of machinery, but is nevertheless required because mutating webhooks must communicate over TLS, which means they need certificates. In SimKube, we create a self-signed certificate issuer for this purpose. Cert-manager is actually a very robust tool, and has the really nice feature that it can auto-inject certificates into your webhook configuration if you apply the cert-manager.io/inject-ca-from annotation, which we do in SimKube. Investigating the cert-manager logs, everything seemed like it was working as designed at first, until we inspected the timestamps more closely. Then these two lines stood out:

I1204 18:29:07.814009 attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1204 18:30:11.466829 successfully acquired lease kube-system/cert-manager-cainjector-leader-election

By default, cert-manager, like many other components in Kubernetes, operates in a semi-HA fashion. There is one “leader” pod and a number of hot standby pods. That way, if the leader pod crashes or gets evicted, one of the standby pods can immediately take over. Kubernetes provides a distributed locking mechanism to ensure that only one pod can be the leader at a time. Until the lease is acquired, the cert-manager pod can’t do any work. What’s interesting to note here is that it took almost a minute to acquire the lease; and moreover, the simulation start time on the runner was 18:29:41, which means that the first CronJob pod, created at 18:30:00, was created before the cert-manager injector could provide the SimKube mutating webhook with its certificate.

So that’s one mystery answered: if the webhook didn’t have a certificate, it can’t apply the proper node selector, and because it fails open, the pod gets scheduled onto a real Kubernetes node instead of the intended fake node. But why and how does this pod become orphaned and stick around in the cluster until the SimKube driver times out?

Now that we knew the mechanism for the failure, it was easy to develop a local reproduction: delete the cert-manager injector pod from the cluster, start a simulation, and then after the first CronJob pod was created, recreate the cert-manager injector pod. This simulates6 the effect of the injector waiting for the lease. In fact, the first time we did this, we didn’t recreate the injector pod until after the simulated-cronjob-sleep-pod-that-got-scheduled-on-a-real-node-by-mistake7 had finished, and in this case it was correctly cleaned up and the simulation finished as normal.

Repeating the test locally, we observed that the critical failure only occurs if the cert-manager injector pod comes up while the CronJob pod is running. Since we had a reliable way to reproduce the error, we decided to take a quick peek at the kubelet logs and saw this log line repeated over and over again:

Failed to update status for pod” err=”failed to patch status
...
<long status update message>
...
for pod \”virtual-default\”/\”hello-simkube-29414879-r22m5\”:
pods \”hello-simkube-29414879-r22m5\” is forbidden: node \”karpenter-worker\” cannot update labels through pod status”

Aha! This is the last piece of the puzzle: kubelet is trying to update the status of the pod to say that it’s finished running, but it can’t. The error message is slightly weird, it’s saying that kubelet is sending a modification to the pod labels to the pod status endpoint, which is forbidden because pod labels aren’t part of the pod status. What’s strange about this is, if you look at the actual update kubelet is sending, there are no label updates.

I suspect those of you who’ve written admission webhooks are nodding along by now. The flow of data looks like this:

kubelet status update -> API server -> SimKube mutating webhook -> 
API server -> kubelet

In other words: because the SimKube mutating webhook was subscribed to both CREATE and UPDATE events8, it intercepted the kubelet’s status update, said “hey, this pod doesn’t have any of the right simulation labels or the proper node-selector on it, lemme add those!” The Kubernetes API server received the modification and said (in the logs) “Hey, you can’t add a node selector on an UPDATE!”, and said (to kubelet) “Hey, you can’t add a label from the /status endpoint!”, and said (to the mutating webhook) nothing9. Kubelet continued to retry the status update for the pod every 10 seconds until the simulation driver terminated.

Wait, but why did everything clean up after the simulation crashed? Well, once the simulation driver pod terminated, there was no longer a mutating webhook in place to add labels to the pods based on a status update, so the update went through, Kubernetes realized the pod had completed, and it deleted it to finish its cleanup.

Remediation steps

After conducting this detailed analysis, ACRL engineers identified the following remediation steps:

1. Stop running cert-manager in HA mode, because our one-replica cert-manager injector pod definitely doesn’t need to be spending up to one (1) minute trying to claim a lock that nobody else is holding.

2. Configure the SimKube driver mutating webhook to fail closed: we basically never want a pod that is designated for a simulated node to get scheduled on a real node, because that could cause all kinds of issues.

3. Configure the SimKube driver mutating webhook to only listen to pod CREATE events, not UPDATE events. Once the simulated pod is running, the driver never makes any further changes, so there’s no reason to listen for updates.

4. Modify the SimKube simulation controller to wait for the driver pod to receive its certificate before continuing with simulation setup.

5. Improve our logging and metrics monitoring infrastructure so that it’s easier to identify and troubleshoot these issues in the future.

As is common with incidents of this nature and scale, there was no single point of failure that caused the issue; had any one of these remediations been in place, the incident would not have occurred. To prevent future recurrence of this issue, and to enable defense in depth, we will prioritize getting these fixes in place at some point in the future when we feel like getting around to it.

Conclusion

ACRL cares strongly about the experience of the zero customers who are using this SimKube CI Runner action. We deeply apologize for the impact that our failure had on your CI pipelines and deploy process, and will be issuing refunds to all zero of customers who tried to use our runner image during the period of this outage. Please feel free to contact our support team if you have any further questions or concerns about this outage, and rest assured we will strive to do better next time.

~drmorr

The vibes

It’s unfortunate that KubeCon was in Atlanta the same weekend that the US government announced disruptions to flights at major airports around the country. I was lucky in that my flight was “only” delayed 2 hours1, but my HMC clinic students this year had their flight canceled, which was a major bummer. A few of them were able to make it out anyways, but it was definitely disappointing. I was having trouble telling how much the FAA disruptions impacted the conference as a whole, though—it still seemed like the event was well-attended, so aside from a lot of worrying on social media, it was still a big, loud, chaotic event just like normal.

While we’re talking about vibes, of course AI was everywhere. All the vendors were advertising their agentic thingmajiggywhatsit, and (at least based on the abstracts) more than half the talks were about AI. I also heard from a bunch of people that the conference felt way more “vendor-focused” than in the past. I’m not sure if I agree with this or not—KubeCon has always as far as I can remember had a veneer of vendors pretending to be open-source so they can sell you something2—but I did see more people complaining about it. The interesting technical content and hallway track conversations are still present, but I think folks are realizing that you have to work a lot harder to find it. And to be clear, I’m not complaining about this3, I think this is just the natural progression for events that are as large as KubeCon and have as much money swirling around.

In terms of my personal experience, just like last year I went to very few talks, so I have a big backlog of things that I want to catch up on when the videos are posted to YouTube. Unlike last year, where I spent a lot of time on the vendor floor, this year I did the pre-game thing where I set up a ton of meetings ahead of time with folks that I wanted to talk to. I think this was a much more effective strategy, and I had a lot of really energizing and productive conversations with folks. It was also really fun to attend with my new coworker, we had a really good time.

The talks

But let’s go over the talks I did manage to attend: I only went to five, I think, but they were all 🔥. I’ll give a quick overview here, in order of appearance:

• Slurm Bridge: Slurm Scheduling Superpowers in Kubernetes by Alan Mutschelknaus & Tim Wickberg, SchedMD: this talk was a deeply technical talk about how to incorporate the OG distributed scheduler (aka Slurm) into Kubernetes. If you’re not familiar, Slurm is a fairly old tool used in many different high-performance computing (HPC) contexts. It is, I’d say, harder to use than Kubernetes and has an “Old Unix Sysadmin” appeal to it, but it’s also extremely good at what it does4. To this end, there have been a lot of efforts to integrate Slurm and Kubernetes, and this talk described one such effort called SlurmBridge. The high-level framework for SlurmBridge is that workloads (Pods) are scheduled by Slurm but launched by kubelet. I don’t have a ton of personal experience with Slurm, but this seems like a really interesting approach and I’d love to play around with it some more!

• Zero Downtime Migration of Monolith To K8s Using Sidecar and Container Lifecycle Hooks by Deepak Kosaraju & James Dabbs, Procore: the next talk I went to was by a former coworker who’s now at ProCore Technologies, which is a construction management software company. This talk was a very hands-on, practical introduction to a problem that many companies face, namely, “We’ve got a giant monolithic code base and we want to figure out how to run it on Kuberentes”. The talk included a lot of practical advice about how to do autoscaling and configure timeouts and application behaviour to minimize downtime when Kubernetes, you know, does its Kubernetes thing. There’s also a GitHub repo where you can follow along and try out a bunch of these techniques on your own!

• I’ve Got 99 Problems and They’re All Controllers by Tim Goodwin, UC Santa Cruz: you might recognize the name of this presenter, I gave a joint presentation with Tim on our Kompile project at KubeCon last year. This year Tim was back talking about (wait for it) Kubernetes simulation! Tim is approaching this problem from a slightly different perspective than I am: instead of trying to simulate the whole cluster for the purposes of analyzing the control plane, Tim wants to simulate the control plane for the purposes of testing a single Kubernetes controller. To accomplish this, Tim built a tool called Kamera5 which provides a fake Kubernetes API server that you can wire into your controller(s). The fake API server can automatically produce events for your controller in any arbitrary order, and check the outputs from your controller to see if it does the right thing. In essence, it’s a big Kubernetes controller fuzzer, and it’s incredibly cool. This was one of my favorite talks from the conference and I definitely recommend that you watch it.

• From Panic To Peace: Making K8s Controller Observability Suck Less by Cat Morris & Derik Evangelista, Syntasso: this was another talk about building and testing Kubernetes controllers, which reviewed a lot of very practical “best practices” for how to write controllers and make them robust, observable, and debuggable. It also included a fun Shrek story throughline throughout the talk, which was a fun touch. Definitely recommend this one as well if you’re in the Kubernetes controller/operator space!

• Evicted! All the Ways Kubernetes Kills Your Pods (and How To Avoid Them) by Ahmet Alp Balkan6, LinkedIn: this was far and away the best talk I saw at KubeCon this year. The whole premise of the talk was “Kubernetes has a lot of ways to kill your pods, and generally people don’t like it when their pods get killed, so let’s enumerate all the ways this can happen so you can be prepared.” He discussed 9 different pod eviction pathways, at least 8 of which I’ve personally experienced. He then demonstrated that none of these eviction pathways are aware of each other or play nicely together, and also that existing Kubernetes primitives for managing pod eviction are WOEFULLY inadequate, one of my personal pet peeves7. If you watch one talk from this conference, make it this one. It’s so good.

• Evolving Kubernetes Scheduling by Eric Tune & Wojciech Tyczyński at Google: this talk was in (almost) the last slot on Thursday afternoon, but it was still really well-attended! The presenters in this talk discussed the future of Kubernetes scheduling to handle the needs of batch and ML workloads running on Kubernetes. This is, as previously discussed, a challenging problem, because Kubernetes was never designed to handle these types of workloads, and is completely unaware of things like hardware/network topologies that are critically important to HPC/ML workloads. The scheduling framework is also missing many primitives that are necessary to operate in this environment. The quote from the talk that resonated the most with me is that, “The Kubernetes scheduler is designed to schedule one pod on one node at a time, and we now need it to schedule groups of pods on groups of nodes at a time.” They discussed a number of proposals and extension points that they are hoping to add to the Kubernetes scheduler to handle this use case. Personally, I am somewhat unconvinced that kube-scheduler is up to the challenge, but I’m still very interested to see where this ends up!

The vendor floor

So that was it for the talks I attended! I also promised to discuss the video that I included at the beginning of this post. As you may have noticed, if you’ve been following along, I did a huge8 video marketing campaign leading up to KubeCon this year, and the second-to-last video was of my good friend and colleague Liz trying to climb into a tiny cardboard box to see if she could “fit in the kube”. This was, objectively, hilarious, and I had the objectively genius idea to see how many other people at KubeCon could “fit in the kube”.

So $20 and one extremely tiny FedEx box later, we launched the first-ever “ACRL Can U Fit In The Kube” challenge. I paid five people $50 each if they could “fit in the kube”, defined somewhat subjectively as “being able to more-or-less close the lid of the cardboard box without it ripping”. We had a lot of people stop by our guerilla marketing “booth”, and I handed up a bunch of SimKube business cards, and got to meet and laugh with a bunch of really cool people. This was probably the highlight of the conference for me: it was so fun to see peoples’ reactions and double-takes when they walked by our “booth”, and I think maybe brightened some people’s days as well in what can be an long, gruelling, exhausting week. So that’s the story of our first-ever KubeCon promotional challenge, and it will absolutely be back next year with “Can U Fit In The Kube 2”9!!!

Aside: ACRL video campaigns

Also speaking of videos, I thought it would be interesting to briefly cover some stats about the marketing videos that I posted in the lead-up to KubeCon. It was definitely eye-opening for me, and a window into a whole world of advertising that I’ve heretofore never experienced. For a quick summary, we posted five videos, one per week, starting at the beginning of October. Each video I posted here, on LinkedIn, on YouTube, and on my social media sites (Hachyderm and Bluesky)10. I spent $100 on LinkedIn to “boost” them for a week, and I also spent $100 on YouTube for the first one, just to compare what sort of reach and viewership we got on the two platforms. The results were somewhat surprising! Here’s a few stats:

• Video 1: “What’s SimKube, Precious???” - Lord of the Rings is near and dear to my heart, so this was a very fun video to make. On LinkedIn, we got 16k “impressions”, 5000 views, 14 reactions, and 1 comment on the post. On YouTube, we got 25k views, and 43 “likes”. The big difference between YouTube and LinkedIn is the degree of targeting that you can do. LinkedIn has very fine-grained controls over your audience, in terms of who the video gets shown to, whereas YouTube was basically “are they in the US” and “what gender”. So even though YouTube got five times the views, I kindof suspect that most of the people who watched the video probably didn’t even know what Kubernetes is.

• Video 2: How to pronounce skctl - this was definitely my least favorite of the video series; it took a long time to make, and the result wasn’t quite what I’d envisioned. Nevertheless, I got a lot of good feedback from the video, and other people seemed to genuinely enjoy it. We got 17k “impressions”, 4800 views, 18 reactions, and 7 comments on the post on LinkedIn.

• Video 3: The Bluth family does Kubernetes - This was my favorite video in in the series. It was short and easy to film, and resonates with a cultural touchpoint that I think is familiar to a lot of people in my “target market”. Also I just love Arrested Development. We only got 10k “impressions” and 4000 views (possibly because I was experimenting with a smaller target audience for this post), but 22 reactions and 3 comments. The folks who did watch this video seemed to love it!

• Video 4: Mission Impossible XII: Upgrade Kubernetes - I was expecting this post to be a lot more popular than it was, particularly given how painful and near-to-home the Kubernetes upgrade problem is for folks, but I actually got very little engagement with it. We had 12k impressions, but only 2500 views, 5 reactions, and no comments. I don’t know if if people were just getting tired of SimKube content, or if there was some other factor outside of my control that was impacting viewership.

• Video 5: Can you fit in the cube? - Of course, the video that spawned our “challenge” at KubeCon. My friend Liz filmed it on a total whim, and (I found out later) wasn’t even intending for me to post it11. Nevertheless, the video was (imo) brilliant, and outperformed every other post we made, with 29k impressions, almost 9000 views, and 16 reactions—still no comments, though.

• Video 6: Can U Fit In The Kube challenge recap - This is the same video at the top of this post, and the “boost” campaign is still ongoing. So far, we’ve gotten 9k impressions and 1700 views, along with 5 reactions and 2 comments. It “feels like” this video hasn’t gotten as much engagement as I was expecting, but I don’t know why: maybe folks are just not on LinkedIn as much now that KubeCon is over? No idea.

So anyways, that’s the summary of my KubeCon experience! It was a good conference, despite a few rough patches here and there, and I’m definitely looking forward to returning to Salt Lake next year!

As always, thanks for reading. We’ve got some banger content coming up in the next few weeks to close out the year, so follow along if you want!

~drmorr

I have a calendar set up here if you want to book some time: https://cal.com/drmorr/kubecon

Filmed by my Developer Relations Consultant Elizabeth Ponce and her partner!!

https://cal.com/drmorr/kubecon

https://cal.com/drmorr/kubecon

(Btw, if you’re wondering why there are so many videos here recently, you can read my recent post about it, which I don’t think got an email announcement because of the AWS outage)

You may have noticed a couple things about this blog over the past couple years: a) I exclusively write long-form posts that are probably a touch too rambly, with wayyyyy too many footnotes1, and b) my last two posts have both been video posts that are thinly-disguised advertisements for SimKube.

Have no fear, gentle readers, my rambly, footnote-laden, erratically-posted long-form text content isn’t going anywhere! These video “things”2 are just part of my brilliant-but-subversive strategy to get many people to give me money so that I can coast comfortably for the rest of my life3.

In seriousness, though, despite my love of long-form text blog posts that go into way too much detail, it’s relatively well-known that a lot of the world prefers videos, and I would like to be able to reach those people. Also, KubeCon is coming up in just under a month4 and I am trying to get as many people as possible to be aware of SimKube before KubeCon so that when I’m there I can have a solid list of folks to meet with.

Schedule a meeting with me at KubeCon!

In this post, I thought it might be interesting to briefly talk about my process with these videos.

Videos are hard, maaaaaan

I’ll tell you a secret: I don’t like making video content. Maybe that’s obvious. I’ve also done a decent amount of it over the years, so I know JUST ENOUGH to know how hard it is, and also how to make really bad videos. So for this SimKube series, I really decided I wanted to lean into the “cheap, goofy, barely-TikTok-worthy” vibe, partly because it’s easy5, and partly because it leans into the slightly-off-the-beaten-path model of ACRL in general. My goal has been to make videos that take no more than an hour to film, and under 2 hours to edit.

I have accumulated a fair bit of equipment over the years to help with this goal. I have a nice camera and tripod because I still occasionally do photography as a side hobby (though much less than I used to). I have a couple different microphones: one is the built-in mic for my Bluetooth headset, which does hardware noise cancellation on the mic itself6, and another directional mic that I picked up several years ago the last time I had to make some videos. And lastly, I have downloaded DaVinci Resolve, which is a professional-grade video editing software that comes with an extremely capable free/community edition that is easy to learn and use. Given that I am explicitly not trying to make professional videos here, this is perfect for my use case.

And that’s pretty much it for my equipment and setup! Now let’s talk about the process.

Script? What script? Where we’re going, we don’t need scripts

All of the videos that I’ve filmed thus far have basically come down to an idea in my head, a few simple notes in Notion, and then I just sit down and improv the rest. This helps me keep to the low-budget, low-time-investment process. But I do spend some amount of time up front thinking about who my audience is and what I want them to take away from it. Aside from just “brand awareness”, my main goals with the videos are a) to make somebody laugh, and b) to highlight some aspect of SimKube that they might not be aware of or thought about. My first video tried to answer a really basic question that someone brand new might ask: “What’s SimKube?” This led me straight to the Mashed Taters meme from Lord of the Rings, and since I already have a vast collection of LotR Legos, the rest of the video fell out extremely naturally.

The second video tried, in a roundabout way, to identify one of the main problem domains that SimKube is designed to help with: scalability and capacity planning. It then quickly devolved into a wacky left-field bit about how to pronounce various CLI tools and why we’ve (collectively) decided that we should end all our CLI tools with ctl. The inspiration for this video is ffmpeg guy, which I continue to maintain is one of the finest pieces of art and satire on the Internet. This video, unfortunately, took way too long to both film and edit, and I think it’s too long to be an effective advertisement as well7.

The third video (coming this Wednesday) definitely hit the sweet spot for time and complexity. A friend and I filmed it in about 45 minutes, it took just about an hour and a half to cut and edit together, and it is, objectively, hilarious. Aren’t you excited to watch it?

Being hypocritical is fun and easy

Once the videos are ready, I post them on a variety of social media platforms; here (obviously), YouTube, LinkedIn, and TikTok8. They go out on a weekly schedule; I’ve been posting them on Wednesdays at 10am. Once they’re posted, I spend $100 to boost the video on LinkedIn.

Everybody: But David, don’t you hate advertising with a fiery burning passion?

Me: Sure do.

Everybody: Soooooo you’re paying for LinkedIn advertising because…?

Me: Because I’m a hypocrite, obviously.

Moral/ethical qualms about advertising aside, it has been fascinating to peek into the world of targeted advertising. I can definitely see why people and companies dump so much money into it. For my first video, I also put $100 into a YouTube promotion, because I was curious to see how it compared to LinkedIn: the main difference is that on LinkedIn, you can do some very specific targeting for who you want to see your videos: what industry, what level of experience, geographic location, etc. etc. etc. On YouTube, I got to select “geographic location”, “gender”, and “age range”. So while I got about twice as many “impressions” on YouTube as I did on LinkedIn, I got more “watches” on LinkedIn that YouTube, and I can just about guarantee that nobody who saw the video on YouTube is going to care (or even know what the heck I’m talking about).

The future of video at ACRL

So anyways, that’s why you’re seeing videos show up here more regularly. The rough strategy here is to produce a bunch of silly, fun short in the ramp-up to KubeCon. If I get one person at KubeCon to tell me “I saw one of your videos on LinkedIn, it was hilarious”, I will consider that a win. Post-KubeCon, I am hoping to transition to some more informational video content. As much as I love my long-form blog posts, there are a lot of people out there who are going to bounce off them hard—and that’s totally fine! Different people have different preferences and learning styles, but that means that if I stick to just long-form rambling posts, I’m neglecting a potentially large audience. So while the frequency of videos will probably lessen after KubeCon, I’m still hoping to have ~1 video/month with some short informational content about SimKube and/or the work that I’m doing around here. Along with the occasional quirky Lego-filled SimKube advert, because those have been heckin’ fun to make.

As always, thanks for reading (and watching)! Until next time,

~drmorr

Come chat with us at KubeCon in Atlanta!

https://cal.com/drmorr/kubecon

Part I: ProtonMail

So, as part of my attempts to avoid Google products as much as possible, Applied Computing uses ProtonMail for its email provider; I’ve also used ProtonMail for my personal mail for a long time, and I’ve been pretty happy with the product, so it made sense to set it up for ACRL as well. And it’s worked great as an email provider: but what I find somewhat lacking is the calendar software.

The core problem is that it is exceedingly difficult to share your Proton calendar(s) with other people so that they can see your availability. The only way you can share a calendar with a non-Proton user is via a read-only ICS link. The problem is that usually these days you really want your calendar to be (partially) externally-writeable, either via scheduling software like Calendly (to let other people book time on your calendar without having to play the “Oh I’m free these times when are you free?” game) or with other calendars (because, say, maybe you are working with a few different clients who each have their own calendars and you need to sync events between all your clients as well as your non-work calendars so that everybody doesn’t all schedule everything all over each other all willy-nilly—but I don’t know anybody in that position).

(It’s me. I’m in that position. I actually wrote a simple Google Apps script that allows me to sync events across all my different calendars, because I couldn’t keep up with manually syncing everything. It gets around the ProtonMail read-only limitation by just creating duplicate events and inviting the ProtonMail account user to those events. It’s not ideal, but it works.)

But anyways, that’s not the point of this story.

Part II: Google

As much as I would like to be completely de-Googled, it’s actually kindof tough to run a business and not have a Google account, because most of the world runs on Google Docs and Google Slides. So a while back, I created a Google account associated with my ACRL email address, which I mostly just use when I need to collaborate with someone else or share a document with them. It’s fine, it’s not my preference, but I got other things to worry about, whatever, life goes on.

However, today I was redoing my business cards because I need to print a bunch more for KubeCon and the last batch that I made was missing a few important things3. And what I really would like to put on the business cards is a QR code with a link to Calendly (or something like it) so that people at the top of my sales funnel can just click two buttons and immediately get some time booked with me, instead of having to take the trouble to write an email, because nobody uses email anymore, or at least that’s what all the kids these days tell me.

So I was revisiting the “nothing is compatible with ProtonMail” rabbit hole when I happened to click through to the Google account that was associated with my ProtonMail email address and discovered that… all of my ProtonMail calendar events were also on my Google calendar (reminder: the one I never use).

Part III: WTF?

Again, maybe it’s obvious to you reading this, but it sure wasn’t obvious to me in the moment. How were all of my calendar events getting synced with Google? I double-checked at least three times, and I wasn’t actually sharing my magic ProtonMail ICS link with the Google account. Moreover, all of the semi-private information (event titles, attendees, etc) were appearing in the Google account, and I definitely wasn’t syncing those. What was going on???

After staring at it for a little while, I realized something important: not all of the events were getting synced with my Google account. In particular, if I created an event, it wasn’t on my Google calendar, but if somebody else had created the event and invited me to it, then it would appear on my Google calendar.

How was this happening? I don’t have a Gmail address associated with ACRL (a Google account is not the same thing as a Gmail address). Did I at one point set up sharing somehow and then forgot, and then maybe a setting changed or got removed and now I can’t undo it? What was going on? The most frustrating, confusing, and annoying thing about this is that I couldn’t just delete my Google calendar, because that would delete all the events on the calendar, and then send cancellation messages to all the other attendees. I definitely don’t want that to happen! But this is even more confusing now, because it means that the events aren’t just getting synced to my Google calendar, but Google also somehow thinks it’s the owner of the events.

I went through a lot of crazy (in hindsight) hypotheses: did I have some weird DNS record4 set up that was also sending calendar events to Google? Did I have an email forwarding rule set up from ProtonMail to Google5? I went so far as to create a brand new Google account that was associated with an unused ACRL email address, and then I created an event from my personal email and invited that user. Annnnndddd… the event didn’t show up!

Ok, now I was really confused, before it finally hit me: Google was man-in-the-middling me! See, it’s not every event that someone else created that showed up on the Google account calendar, it’s just events that are created by someone else using a Google account. If both users are on Google, it just assumes that you’re using their product and will auto-create the events on your Google calendar, even though the calendar invite email gets sent somewhere else entirely. This is also why all of the calendar event details appear on the Google calendar, and why Google will notify users if you try to delete the events off of your Google calendar.

It’s obvious in hindsight, but it was a real head-scratcher for me for a bit this afternoon. And it was a good reminder that if one party in any sort of online interaction is using Google, then all parties are using Google, whether they want to be or not6. And not that it really matters, but I can also tell which of my contacts are using Google products or not by checking to see whether their calendar events show up on my Google account.

Anyways, now that that mystery is solved, I can get back to the problem of “figuring out how to make Calendly work with ProtonMail”78 so that I can print my dang business cards. Isn’t this fun???

Tune in next time for another rant about my problem of the week! Until then, thanks for reading!

~drmorr

Problem statement: how much will we save by doing X?

Anyone who’s worked on a platform team has probably at some point had their boss ask them “our platform is expensive, how can we make it less expensive?” When you’re dealing with Kubernetes and the cloud, there are a lot of variables at play, and it can be really tough to know in advance which knobs and levers are worth tuning. That’s where simulation comes in! If you can spend a couple hours running a simulation to see some projected savings, using real production data, that could potentially save you a lot of time and wasted effort. We’ll see why in just a minute.

First, some background on the specific scenario I’m demonstrating today: in recent years, the big cloud providers have started offering ARM-based compute resources with similar-or-better performance characteristics to their equivalent x86 machines, for a slightly-reduced rate1. The details of the different architectures aren’t super important for this post: all you need to know is that they aren’t compatible, so it may involve recompiling a lot of code as well as figuring out how to serve multi-platform Docker images. All of this is doable, and in fact quite a few of the big tech companies have done it, but the key point is that it’s a non-trivial amount of engineering effort to enable. So the natural question you might ask is, “Is the savings worth the effort, and over what time horizon?” And that’s the question we’re going to tackle today in this post.

The problem setup is straightforward: I’m going to use the same DeathStarBench trace data that I used in my comparison of KCA to Karpenter; I’m using the latest version of Karpenter for autoscaling, and I’ve configured SimKube to “look like” AWS by offering the full suite of AWS EC2 instances for scaling.

We’re going to run three different simulations: the first is using only x86 machines (as in my KCA vs Karpenter comparison, we’re using m, c, and r class nodes of 6th and 7th generation). In the second simulation, I have configured a few of the deployments in the trace to be “ARM-compatible”, and added in the equivalent Graviton nodes into the mix; and in the last simulation, I only run on Graviton nodes.

Why did I set things up this way? Well, in any large Kubernetes deployment, you probably have some workloads that don’t care about the underlying architecture: Python code, for example, is generally architecture agnostic2. It’s a high-enough level language that the architecture details are abstracted away and it should “just work”. Golang is a step up on the difficulty ladder: you will need to recompile your code, but Go offers very good cross-compilation support, so it’s probably not “too hard”. Java code is probably similar to Golang in this regard. Highest on the complexity ladder are things like C/C++/Rust: especially if they have a lot of low-level system requirements, it’s probably going to be a lot of work to port from x86 to ARM.

So, this simulation is answering the question, “How much will we save if we just port the easy stuff to ARM?” compared to “How much could we hypothetically save if we ported everything to ARM?” Since this is all made-up for illustrative purposes, I just picked a few DeathStarBench deployments at random to be the “easy stuff”, but you can hopefully see how this mirrors the types of conversations that occur in many organizations.

Here’s the cool thing: because all my workloads are fake, they don’t actually care about the underlying architecture. In fact, the nodes are fake too, so they can report whatever architecture I want3. If you were to do this experiment using real nodes and real hardware, it would probably take you a few weeks to even just get things to the point where you could run a test; and I did the same work in an afternoon.

The results: is the work worth it?

Now that we understand the problem statement, we’ll look at the results. First, we need to establish our baseline: how much does it cost if nothing uses Graviton?

Figure 1 shows both the total node count as well as the node composition in this experiment. You can see that predominantly, we are using c6a.48xlarge instances, which Karpenter has presumably chosen because they’re one of the cheapest instance types that supports the large number of pods that we’re running. We can multiply this set of instance type data with the publicly-available on-demand pricing for each of these instance types to find out how much everything costs. Doing so, we end up with a total price of $149.22 for this twenty-minute slice of time4.

So what happens if we port the easy stuff over to ARM? I looked at the five largest (in terms of pod count) deployments in the simulated trace file and allowed them to run on Graviton nodes5; everything else was restricted to x86. Re-running the experiment yields the following:

You can see in Figure 2 that we are definitely using Graviton nodes now6! We’re running a lot more (but smaller) nodes, and the most common instance type is the c6g.16xlarge. That seems promising, how much did it cost? We run the numbers and find out that it costs… $150.72. Basically the same as before7! Sure glad we didn’t put in all that engineering effort to save $08.

So OK, doing just the easy stuff doesn’t help in this case, and we clearly don’t have the time to migrate everything to Graviton right now9, but just hypothetically speaking, what happens if we were able to wave a magic wand and move everything over to Graviton? Figure 3 shows the answer:

Now everything is running on Graviton, and our total EC2 spend is predicted to be $138.24 for this 20-minute time slice. Now we’re talking! There’s that 10% discount showing up finally10. Is that really worth the 6-12 months of concerted engineering time it would take to port all these workloads over to support ARM? I mean, maybe, maybe not, we’ve now moved from the realm of engineering into policy and priority decisions—but the point is, you can make better decisions now because you spent a few hours running some simulations and collecting data first.

A cool epilogue: SimKube plus KubeCost

Even though this was kindof a “toy” example, hopefully this is helpful to see how you might be able to use SimKube to make decisions around all the different cost levers that are available in Kubernetes and the cloud. I wanted to close with one cool observation: if you were going to repeat this for your own workloads, you could do what I did, and collect all the data, and then multiply it out by hand. It’s not hard to do. BUT, there are also existing tools out there to help you track your Kubernetes spend that make it a lot easier! Possibly one of the more well-known ones is KubeCost, which was recently acquired by IBM. KubeCost integrates with all the different cloud providers and can pull pricing data from the public APIs, as well as take into account EBS volume usage and any of your negotiated volume discounts or other cost-impacting factors: and you can just, you know, install KubeCost into your simulated environment and get some costs out! Here’s a screenshot of me doing exactly that:

I dunno, I think this is pretty nifty, and it really shows how you can start to use all of your existing tools and components in your simulation environment as well, to take your simulation analysis and data collection to the next level.

Anyways, I hope this was an interesting read for you all, and that it gives you some ideas for how you can use SimKube in the future! As always, thanks for reading!

~drmorr