This is the third post in our Five Key Autoscaling Metrics series; in the first post, Ian broke down the Workload Committed Capacity metric, or “why your utilization isn’t as good as you think it is”. Last week, I talked about measuring the “scale-up” performance of your autoscaler from the perspective of your users: namely, how long they have to wait before they can see their AI-generated cat pics. This week, we’re going to look at the inverse: how much your users notice when your autoscaler scales down (and why Kubernetes makes it virtually guaranteed that your users will notice).

Can you please design an autoscaler on the whiteboard?

If I’ve learned anything from doing endless architecture and/or system-design interviews (on both sides of the table), it’s that what software development looks like in an interview has nothing at all to do with what software development looks like in real life. See, in an interview, typically you’re asked to “design a system” for the interviewer, who can then ask you arbitrary questions like “how many requests per second can your whiteboard handle?” In real software engineering, “design” is a naughty word and most people skip right past it. But I digress; just for the sake of argument, let’s suppose that somebody asked us in an interview to design an autoscaler that gracefully scales down. How would you do it?

Well, let’s think about what can go wrong on scale-down: if your pod isn’t doing anything, you can just kill it. If it is doing something, then maybe you should let it finish before you shut it down. But how do you know if your pod is doing something?

Spoiler warning: you don’t. You’re the platform, you have literally no idea what kinds of applications are running on top of the platform, so it is literally impossible for you to understand whether it’s safe to terminate your application. So anyways, I guess we’re going to fail this system design interview, because most interviewers don’t like being told “the problem you’re asking me to solve is impossible”1.

Just kidding, we actually can do something: we could (gasp) ask the application whether or not it’s ready to be terminated2! And then we could give it some kind of, idk, “termination grace period” or something so that the application has a reasonable amount of time to shut down but it can’t greedily just hang on to its resources forever.

So, putting my interviewer hat back on for a minute, I say “Huh. That seems like a reasonable approach, but how can you tell whether it’s successful? In other words, how do you know whether your autoscaling policy is actually impacting your user experience?”

And then I take my interviewer hat off and I respond “Users? What are those? I’m a platform engineer, I don’t care about the users!”3

I thought this post was supposed to be about metrics

Ok, I’m kindof burying the lede in this post4, but the actual answer to the hypothetical interviewer’s question is, “You measure it!” Specifically, you need to understand a) what kind of errors your application can throw, and b) how many of them are happening. In some cases, this is easy: is it a webserver? Then it probably throws a 500 error when the HTTP request abruptly gets canceled. Is it some kind of queue processor? Maybe the task it was working on ends up in a dead letter queue. In other cases, you may need to think a little bit more creatively about your application’s errors and how to measure them.

But! It’s not sufficient to just “count errors”, because your application can throw errors for any number of different reasons. What you really want to be able to do (because this is all about autoscaling, remember) is connect the errors that get thrown to the actions taken by your autoscaler. Seems easy, right5?

Welllllll it turns out that in today’s world, about the best you can do is correlation, not causation. You “probably” have some metrics getting emitted about your 5xx response rate for various pods, and you might also have some metrics from your autoscaler indicating whether it took a scale-down action. You can then do some extremely ugly PromQL to apply some kind of envelope function to your error rates metric, so that you can produce a count over time of “errors occurring during scaledown”.

Normally I would try to be helpful and actually tell you what that PromQL is, but that honestly sounds harrowing enough that I don’t want to try, and I give you 10-to-1 odds that your monitoring platform just chokes on the query anyways. So that’s not great. But hey, maybe we can do better?

What if we just asked them

I’m going back to a point I made earlier in the post: your application is the only thing that knows when it’s safe to scale down, and it’s also the only thing that knows if it’s throwing an error or not. So what if we tied those two things together? What if we made a communication channel to your pods that tracks “disruption safety”? What if we “just asked”?

Then we could actually do two things at once: we could have a single built-in metric to track the number of pods that get disrupted outside their “disruption safety” window, and we could have a good autoscaling flow:

1. Autoscaler: hey I need to scale some stuff down, those pods looks good. Hey pods, can I scale you down?

2. Pod 1: Sure, go ahead!

3. Pod 2: Uhhh I’ve got some cleanup I need to do, check back in five minutes.

4. Pod 3: Nope, I’m never going away, I am immortal! 😈😈😈

5. Autoscaler: OK, I’m gonna shut Pod 1 down now. Pod 2, thanks for getting back to me, I’ll try again in five minutes. Pod 3, you are not immortal, you’ve got 10 minutes before I pull the plug. Make the best of it.

I’m going to spend a lot more time on this in my rant about pod disruption budgets next week, but unfortunately something like this just isn’t possible today. The best we can do are hacks like Karpenter’s do-not-disrupt annotation or messing around with pod deletion cost values.

But here’s the thing: there’s already some precedent for Kubernetes communicating with pods: we have three different probes to monitor one half of a pod’s lifecycle, and while it’s frequently confusing what the difference is between startup probes, readiness checks, and liveness checks, doesn’t it seem a bit unbalanced that we don’t have any probes for the scaledown side of things? It wouldn’t be that hard to create a disruptionReadiness probe that various other Kubernetes components could query to coordinate scaledown6.

Did u just write a KEP?

Ehhhhh, the thing is, is that it actually would be pretty hard to create that probe. Not because the code would be that complicated, but because the process of getting any changes made to the core Pod spec is incredibly burdensome these days7. But sure; I guess if any of the Kubernetes authors are reading this and want to write a KEP for this, I wouldn’t complain. I do think I’d be remiss to mention how most organizations handle scaling down today, though. There are three steps:

1. Add middleware that does retries.

2. Set an exorbitantly-large value for terminationGracePeriodSeconds.

3. If you’re feeling extremely up to the challenge, add a preStop hook to gracefully stop doing work and shut down the application before the pod is terminated.

Though to be honest, most people stop at step 2, because step 3 is hard and people are lazy. And so, given that we’re supposed to be talking about metrics for autoscaling, here are two that you can track to monitor your scale-down performance: first, track your error rates. You probably don’t even have to do any sort of fancy PromQL envelope function, if you just overlay your error rate metric with your replica count on Grafana, it’ll be pretty clear if you’re getting a burst of errors due to scaledown. And second, maybe track how long your pods are sitting in a “waiting for deletion” state because you’ve set your termination grace period seconds to 400 years.

As always, thanks for reading, and if you do get a job based on my hot interview tips, remember you owe me 10% of your signing bonus.

~drmorr

Welcome to the second deep dive in our “Five Key Metrics for Kubernetes Autoscaling” series! In last week’s post, Ian broke down the node workload committed capacity metric: how this is a useful proxy for “wasted money” and why nobody is actually computing this correctly. In this week’s post, we’re going to do the same thing, but on the flip side of the cost-reliability coin. Specifically, how can your autoscaling behaviour impact the actual end-user experience for your application?

We’re going to do this via a metric that nobody’s ever heard of before, called “Mean Time to Stop Sucking” (MTSS). Just like in last week’s post, I’m going to argue that, despite the name, this is a real metric that you ought to be tracking in your autoscaler, and then explain why nobody does.

But first, let’s revisit the “cost-reliability” coin, since it’s been a minute1 since we’ve talked about it.

Heads, it’s cheap. Tails, it’s not broken.

Like I mentioned in the intro post, there are two reasons why you might be interested in autoscaling: cost, and capacity planning. The first reason is the one that everybody talks about, but in my favorite thought experiment of all time, I can make your infrastructure extremely cheap by turning it all off. Or, I can make your infrastructure extremely robust and reliable by provisioning a c7i.48xlarge for every pod in your cluster, but I don’t think your CFO will like that very much. Ideally you’d like your infrastructure to be “somewhat” cheap, but also “somewhat” reliable, but how do you know where you are on that curve?

Ian’s post last week covered the “cheap” side of things; this week and next week we’ll cover the “reliability” aspect. I’m actually going to break “reliability” into two sub-components, that is, “scaling up” and “scaling down”, because they’re both important and you need to measure different things for them. On scale up, you need to know if your users’ experience is degraded (and for how long) because it’s taking you too long to provision new capacity. On scale down, you need to know if your users’ experience is degraded because they were actively doing work and your autoscaler rudely interrupted them.

So there you have it: autoscaling metrics two and three! We’re all done here, you can go home now.

“Mean Time to Thing” metrics considered harmful awesome

Just kidding. Let’s talk more about scaling up.

In Kubernetes, this is how it works: your vibe-coded project gets posted to the orange site and all of a sudden five million AI bros are hitting your homepage. Unfortunately, you only have two pods backing that homepage; so those pods’ CPU utilization goes through the roof, but fortunately you configured the Horizontal Pod Autoscaler (HPA) to watch for overloaded pods, so after sending back a few million 503s, a bunch of new Kubernetes pods get created. Unfortunately, you’re still on the AWS free tier, so you’ve only got a single t3.medium instance, and it’s all full! However, you did actually set up Karpenter to scale up more nodes, so it sees all these pending pods and requests some GPU instances from AWS, because your vibe-coded app needs GPU instances to display your home page for some reason. Sadly, Anthropic has sucked up all the GPU instances so it takes 10 minutes to get some, but this is a fine and normal experience these days. ANYWAYS, once you’ve finally gotten your compute hardware, your pods can get scheduled on them—just kidding, actually you have to install 10 years of security updates before Kubelet will start. NOW your pods can start!

You’re done, right? Your users can now click buttons on your vibe-coded home page? Ha! Hahahaha! Nope. Sadly, Claude Code wrote your app in Java, which means it takes another five minutes to load your entire database into memory, and do all of the JIT compilation things, and then finally the AI bros can see a vibe-coded picture of a cat.

OK, so this situation probably isn’t great, but how can you make it better? Well, the first step is, you need a metric to game! In most organizations that I’ve worked at, the metric folks look at is “time from pending pod to scheduled pod”2. Even this metric is somewhat challenging to track, but Karpenter, at least, has an alpha timeseries called karpenter_pods_unbound_time_seconds, which tracks the time from pod creation to pod binding.

But as you can see, this metric really doesn’t capture the performance of your autoscaling. In fact, mostly what this metric captures is the cloud provider provisioning delay. That’s, like, a tiny fraction of what your user cares about! So in the post, I’d like to propose a different metric, called “Mean Time to Stop Sucking”: this is the length of time between when a user makes a request and when their user experience stops being utter dogshit3.

WAHHHHHH, that’s so hard to measure, I don’t wanna!

I’ve been in a number of these conversations with folks, and whenever I’ve pitched this as a metric to monitor, most people have said something to the effect of “That’s a really hard metric to measure, and also we don’t have control over half the things in there. We should just monitor the bits we have control over.”

And, like, excuse me. You’re distributed systems engineers. Your literal job description is to do hard things. And also: there’s lots of stuff in life you don’t have any control over, but you monitor anyways4. The first step to “improving things” is asking Claude to fix your broken everything understanding them. If you don’t understand where all the delays in your scaling are coming from, how on earth do you meaningfully expect to do anything about them?

Anyways, maybe that’s a little bit unfair. This is actually a really hard thing to measure, at least in a useful way; if you just measure MTSS as a single number, you know that things are bad, but you don’t understand why things are bad. But to understand why things are bad, you need to correlate/connect events from multiple different sources. You need to know when your users make requests, which you probably don’t have a good way to track unless you’re running some kind of service mesh. You need to link that time to your horizontal autoscaler response, which is based on a bunch of aggregated metrics data from cAdvisor. Then you need to connect the HPA scaling behaviour to your cluster autoscaler response, and finally you need to understand your application performance on startup. So that’s like, five different metrics you need to track just to track one single metric5!

The good news is, with various distributed tracing infrastructure, you actually can track all of this stuff and get a good breakdown of your scaling activity. Setting up the distributed tracing infrastructure is left as an exercise for the reader.

We have achieved suckitude! Now what???

So OK, now you understand a) that your application sucks, and b) why your application sucks. How do you make it suck less? Step one is the hardest step: “convincing your CEO that having an application that sucks is bad for business.” But this isn’t an advice column, so I’m not going to dig into that one any further.

What I will talk about is how you, as (purportedly) a member of your platform team, can have a positive impact on your scaling up time. It might feel like you don’t have many levers to pull, but it turns out there’s actually quite a lot you can do here! Here are some ideas:

1. Use a faster node autoscaler: as I pointed out a long, long time ago, there is a marked difference in scaling speed between the Kubernetes Cluster Autoscaler and Karpenter. There’s a pretty good chance that “just” switching to Karpenter is going to have a noticeable improvement on your cluster scaling speed. You don’t even have to be on AWS for this, there’s a Azure Karpenter Provider that’s been around for a couple years now, and even a GCP Karpenter Provider!

2. Optimize your node provisioning time: most of the cloud providers are “pretty fast” at providing you with hardware when you ask for it6. And yet, the number of folks I’ve talked to who are like “it takes 5+ minutes for a node to become available” is incredibly high. It turns out that as much as we talk wanting about our compute hardware to be cattle, not pets, actually doing that is, well, pretty hard. So usually when a new node comes up, there’s a lot of “stuff” that needs to be done before it can join your Kubernetes cluster: installing OS security updates, node configuration based on environment (prod vs staging), region (US vs somewhere else), etc. All of that stuff takes time, and if you can make it go faster then you can improve your MTSS. I don’t have a ton of specific advice here because it likely depends on a lot of specific architectural decisions you’ve made about your infrastructure, but one thing I’ve seen be effective for some organizations is to bake as much of your architecture into an AMI as possible.

3. Use a faster horizontal autoscaler: one of the more common reasons7 that folks reach for KEDA is that you can configure the metrics evaluation loop to be more performant than the vanilla HPA. The more often you evaluate your metrics, the faster you can react to changes in those metric values.

4. Use better horizontal scaling metrics: in what is the definition of the word “irony”, every platform engineer on the planet will tell you that you should not scale on pod CPU utilization, and also every platform engineer on the planet scales their pods on CPU utilization. If you scale on something else (requests per second, queue depth, really anything at all is better than CPU utilization), you’re going to have an earlier signal that load is increasing, which means you’re going to have an earlier response time, and your application is going to suck less. KEDA is also a great choice if you want to scale on arbitrary non-CPU metrics.

5. Scale your entire dependency chain at once: this lever is the corollary to #4. A very common pattern is that in your microservice architecture, you have a chain of dependencies, A→B→C→D. Service A starts getting increased load based on user requests, which it eventually solves by scaling up. Service B then starts getting increased load based on requests from service A, so then it starts scaling up. This continues until service D has scaled up, and then—and only then—will your users stop seeing errors. The better way to do this is to scale your entire dependency chain at once as soon as service A starts seeing increased load. The hard part here is identifying what your dependency chains are, but there was a very cool paper published a few years ago that provides some insights on how to do this.

6. Make your application start faster: I’m pretty sure that you, as a putative platform engineer, had a visceral reaction when I said this. “But we don’t control the application code, bro! Those are the product teams, and we try as hard as possible to never talk to them!” Well, OK, I think I’ve identified one problem in your organization already. But also, even if you absolutely can’t change anything at all about your application code, there’s still stuff here you can change: how many init containers are you running in your pods? How many sidecars are you running that all need to be healthy before your pod readiness check succeeds8? Maybe get rid of some of those.

7. Use a predictive autoscaler: this is the thing that literally everybody jumps to. “If we could just predict when our load is going to occur, then we don’t have to do any of the hard work of actually making our application suck less!” I think folks really like this approach because it sounds sexy as hell, and also probably because implementing a predictive autoscaler is likely going to get you a promotion. But unfortunately I have bad news and worse news for you: a) you’re probably never going to get it right, because doing predictive autoscaling is really, really, really hard to do well, and b) even if you do get it right, it’s probably not going to solve all your problems9. That’s not to say that predictive autoscaling is always a bad idea or that it’s never going to work well, but it wouldn’t be the first thing I reach for in my toolbox. However, if you do decide to go this route, I recommend paying a vendor instead of trying to implement it yourself: there are a couple vendors who do this really well, and about 67 other vendors who do it extremely poorly, and if you pay me a bunch of money I might tell you which is which.

So anyways, there you have it, ACRL’s simple seven-step guide to making your application scaling suck less! Tune in next time to find out all our secret strategies for scaling back down once you no longer need less suckage.

As always, thanks for reading!

~drmorr

At ACRL, we get to talk to a lot of clusters… I mean people. Many of those people manage clusters at scale1, which we don’t recommend, but we all have our burdens. In conversations with these individuals, who will remain nameless to protect the guilty, we often hear something like, “at made-up-for-this-fake-example Corp., we’re only utilizing 67% of our capacity”.

And to that I want to say: “What do you even mean, bro?”

I’m a bit of a student of economics, by which I mean I lost a lot of money in crypto scams and now I have to do this to survive. But I once had an economics professor who would respond whenever you mentioned a rate like productivity, efficiency, or utilization with a simple question: “Of what?” This was a polite way of saying: “Tell me the damn numerator and denominator, and do it right now.”2

Specificity matters, and at scale, it matters a hell of a lot. Today, let’s give ourselves permission to be a bit pedantic, because we can’t afford not to.

A cluster (under)utilized

In the lead-in to this series, teased the five key metrics for evaluating autoscalers. We’re starting off with committed capacity percentage: a cost-oriented metric that reveals how much of our provisioned capacity we’ve committed to workloads.

“Capacity Utilization” sounds like a straightforward efficiency metric. Unfortunately, as an industry, we are pretty abstract about what utilization is and how we measure it. If you Google cluster utilization, you will get a terrible AI overview, followed by an ad, followed by a fistful of different definitions, including some of the following:

These are all perfectly valid calculations, but they all measure different things. When someone says, “Our utilization is 57.62%,” I still have to ask “Of what?”

For our five metrics for evaluating autoscalers, our interest in utilization is as an inverse proxy for waste. For a comparable workload, a higher utilization metric should indicate we are provisioning less excess capacity. It does not prove that those uncommitted resources are removable or that all the requests are necessary. We’ll get to that in a moment.

Show me the money

I’m going to spoil this right out of the gate. If we had to choose just one metric for the cost component of our five metrics for evaluating autoscalers, it would be Committed Capacity Percentage. Well… sort of.

Where:

• R = Total scheduled pod requests

• C = Total provisioned node capacity

Typically, committed capacity is calculated separately for CPU and memory. Here, “requests” means the resource requests of non-terminated pods on the nodes we are measuring. Unscheduled pods represent unmet demand rather than commitments against the current nodes and should be tracked separately.

Why this ratio? We love money, dude3.

We’re paying somebody good money to provision nodes. Any capacity above what is required to support our workload is a cost worth investigating. The numerator tells us how much capacity workloads have claimed. Our denominator shows the full provisioned node capacity.

Waste REVEAL thyself

Before we start hunting down waste, we need to understand where that capacity went. Total provisioned capacity is not all available to pods. First, we have to reduce our provisioned capacity by host reservations and eviction headroom, giving us our allocatable capacity. Pod requests can claim up to all the remaining allocatable capacity.

It’s important to note that kubeReserved and systemReserved belong to a different class of reservation and are not included in pod requests (our numerator). Requests from DaemonSets, however, are pod requests and are included in the numerator. If we were to use allocatable capacity as the divisor, we would lose visibility of the overhead inherent in kubeReserved and systemReserved. Some of this overhead is 100% necessary.

In most cases, the capacity of a node has two primary dimensions: CPU and memory4. How the scheduler fits workloads onto nodes is called bin packing and, since we have two dimensions, we actually have a 2D bin packing problem5. Bin packing is a fascinating, complex domain; however, it is quite easy to visualize potential excess capacity with a conceptual diagram. To illustrate the complexity of this problem see the following diagram: efficiently bin packing a bunch of squares of equal size is quite easy, but inefficiencies emerge quickly with real workloads: we’ve got long, short, big, small, squares, and rectangles creating many inefficiencies in the packing of this node6.

Bin packing is just one place inefficiencies appear. Other sources of excess capacity may include slow or inefficient scale-down, node/pod shape misalignment, and stranded resources due to scheduling constraints. When these cause additional resources to be provisioned for the same requests, committed capacity falls. Committed capacity percentage can show us that one or more of these symptoms are present, but it will not identify the cause.

We also can’t tell from this metric alone if the excess capacity is waste. Some headroom is necessary for a well-functioning cluster; it aids in faster scheduling and overall resilience. That’s why the goal is not 100% committed capacity percentage. An organization might have a lower provisioned capacity because of a valid reliability concern, which is also why this is just one of our five metrics. That all sounds great, right?

Wait a second. Some of our allocatable capacity is claimed by DaemonSet replicas on each of our nodes. What if we increased our node count while keeping total provisioned CPU and memory unchanged? In this scenario, our DaemonSet replica count would rise and committed capacity percentage would increase. That seems like the opposite of what we want in a metric for evaluating autoscalers. We would prefer a measure that decreases as overhead increases7. Additionally, our output metric (the numerator) should be closely aligned with the actual application workloads; after all, the platform is designed to serve customers, not the other way around8!

With that said, we are pleased to announce, Workload Commitment Percentage, a modified committed capacity metric which addresses this loophole and is more closely aligned with the actual business objective of your clusters.

Where:

• R = Total scheduled pod requests

• D = DaemonSet pod requests

• C = Total provisioned node capacity

By keeping the denominator as our total provisioned capacity, we still have the broad cost basis for our metric. We exclude DaemonSets from the numerator so as DaemonSet overhead rises our Workload Commitment will fall.

PromQL Example

Disclaimer: This probably won’t work at scale or maybe at all. But hey, knock yourself out.

avg(sum(kube_pod_status_phase{phase="Running"}
  * on (pod) max by (pod) (kube_pod_container_resource_requests{resource="cpu"})
  * on (pod) max by (pod) (kube_pod_owner{owner_kind!="DaemonSet"})
  * on (pod) group_right max by (pod, node) (kube_pod_info)
) by (node) / on (node)
sum(kube_node_status_capacity{resource="cpu"}) by (node))

Point of order: Ian, are you stupid?

If you talk to your platform teams, nobody actually tracks this metric, and why not? Is it because we’re stupid? No, it’s because it’s hard to do. Here’s a quick comparison of the other metrics that are easier to track, and why we recommend doing the hard thing anyways.

Why not requests / allocatable?

What we are calling Scheduler Saturation helps us understand how much of the schedulable budget is claimed. For our cost-oriented comparison, we want the full provisioned resource budget in the denominator. Necessary host reservations still occupy capacity on the nodes we are paying for.

Why not usage / requests?

Request efficiency helps us determine how closely requests match observed consumption. It can expose oversized requests, which are worth investigating. But it can’t see an empty node. Add an idle node without changing the pods, their requests, or their usage, and this ratio stays exactly the same, but our bill won’t. Additionally, requests are not limits, so usage over 100% of requests is possible and a value below 100% isn’t necessarily waste.

Why not usage / allocatable?

Allocatable Utilization tells us how busy the capacity available to pods is. This is a helpful metric for investigating resource pressure.

For evaluating node autoscaling, this leaves something to be desired. The denominator is only our allocatable capacity, which is not what we actually pay for. A useful metric for answering a different question.

Why not usage / provisioned?

Measuring usage against the full provisioned budget can help evaluate our overall infrastructure efficiency. We choose requests because we’re interested in how efficiently the autoscaler accommodates the resource demands of the cluster. Whether that demand is appropriately sized is another question that is absolutely worth asking.

Let me break this for you

Great, we have a metric. Let’s game it.

If we simply traversed our cluster and increased the requests on our application pods, we could improve our Workload Commitment without doing anything useful like, I dunno, reducing total cloud costs.

Larger requests also reduce the placement options for pods and make node consolidation harder for node autoscalers. If the changes produce more nodes, costs could increase.

There are other ways this percentage can mislead us:

• A cluster-wide average can hide resources stranded on specific nodes. You can’t take CPU from one node and supply it to a pod on another node.

• An autoscaler could leave unsatisfied demand and earn a good score by keeping its existing nodes full. So we would see a high workload commitment percentage but many pods awaiting capacity9.

• A higher percentage on expensive nodes could cost more than a lower percentage on cheaper nodes. CPU and memory percentages are not denominated in dollars.

For autoscalers to reduce spend, they must make or induce changes at the node level: fewer nodes, cheaper instance types, or both. Other pricing changes can reduce spending too, but workload commitment doesn’t measure them.

Use this metric to compare provisioning under comparable workload and reliability requirements. Don’t treat it as proof of savings or a ranking system for otherwise unrelated clusters.

Why track it anyway?

Maybe you are thinking, “Well, Workload Commitment is imperfect, maybe we shouldn’t worry about tracking it at all.” However, this is a useful signal that is cheap(-ish) to calculate. You can get all the request data easily from Kubernetes’ native metric sources. Workload Commitment is a great frontline metric and provides a good indicator of avoidable cluster costs10.

Conclusion

Workload Commitment Percentage is our first look at the cost side of evaluating autoscalers. It tells us how much provisioned capacity has been claimed through requests for non-DaemonSet workloads. It can’t tell us if the requested amounts are sensible, if excess capacity is justified, or whether our applications received the capacity they needed quickly enough. That’s why we have four more metrics on the way.

Thanks for reading the ACRL blog. Be sure to subscribe to catch the rest of the series!

Cheers,

Ian

Earlier this year, I had the opportunity to chat with Courtney Nash1 at SREcon, and one of the things she reminded me is that “it’s OK to say that you’re an expert”. So, what am I an expert in? Well, for the last decade-plus, I’ve been working with autoscalers for distributed systems. I built Clusterman at Yelp—originally for Apache Mesos, and then migrated to Kubernetes; I’ve contributed to both the Kubernetes Cluster Autoscaler and Karpenter; and I’ve helped a lot of companies do autoscaling better. So, I think it’s safe to say that I’m something of an autoscaling expert.

Autoscaling is a bit of a funny topic: it’s one of those domains that seems like it shouldn’t be that hard, but once you start digging into it, you realize that there are a lot of subtleties. Do you scale up reactively, or do you try to predict when new resources will be necessary? How long do you keep unused resources around before scaling down? How do you put guardrails around your system so it doesn’t scale out of control, and then what do you do when your applications run into those guardrails naturally? How do your answers change when you’re in the middle of an incident or outage, instead of normal operations2?

Thus, I thought it would be interesting over the next several weeks to deep dive into autoscaling: why do we do it, and how do we do it well? At ACRL, we’ve identified five “key metrics” that you should be tracking for your autoscaling engines, and—spoiler—they might not be the normal ones you think of. In this post, we’ll do a quick overview, and then spend future posts digging into some details. If you’re curious, follow along below!

Why do we autoscale?

Before we talk about anything else, it’s important to understand why we autoscale. There are two main reasons: the first, which is the one everybody talks about, is cost. When you were a small baby startup, you had lots of AWS credits and not that much infrastructure, so you just massively overprovisioned everything, because why not? It’s all free anyways. At some point, however, you ran out of AWS credits and you have a LOT more infrastructure, and your VP of Finance comes to you with a bill. What do you do? Just shove some autoscaling in there! It’ll be easy! Take your tech lead a couple weeks at most. Massive savings! Problem solved!

But it turns out to not be that easy; two weeks later, your tech lead has “shoved some autoscaling in there” and you’ve just had your first autoscaling-related outage. Whoops. So then you kick off a months- or years-long effort to balance your cost savings efforts and your reliability efforts. Better write some OKRs about that!.

The above example highlights something that I’ve said hundreds of times: cost and reliability are diametrically opposed. You can make your system extremely robust by purchasing all of the largest, most expensive compute that you can get your hands on3, but then you’re gonna get another visit from your VP of Finance. On the flip side, you can make your system extremely cheap by shutting off all your compute, but then you’re going to get an unpleasant visit from your CEO because they no longer have a company. Obviously these are extreme examples; in reality, you’re going to be constantly balancing the tension between these two extremes.

However, there’s another reason for autoscaling that most people don’t talk about, that I might argue is even more important. Even if you’re running at peak capacity all the time, one of these days your little startup is going to run into limits. You’re a hyperscaling growth company, after all! You double the number of users of your product every other week! Congratulations! You’ve won capitalism!

Back in the previous century, you would handle this by going through an extremely rigorous data analysis exercise called “capacity planning”. Unlike today, when you can ask AWS for a new machine and get it approximately instantly4, it would take between 3 months and a year between when you “place an order for new hardware” and “when it shows up in your datacenter” and then another several weeks or months after that to when it was actually available for use. So you would need to predict, somewhere between 3 months and a year in advance, how many compute resources you would need. And if you got it wrong, welp, good luck.

Autoscaling makes that easier; it doesn’t obviate the need to do capacity planning, but it gives you more leeway if and when your predictions don’t pan out. This isn’t a cost thing; it’s a “long-term viability of your business” thing. And, at least to me, that’s an extremely compelling reason as to “why you should think about autoscaling even when your business is small.”

What things can we autoscale?

Conversations around autoscaling tend to be very confusing, because everyone is talking about something slightly different in the process. As an industry, we’ve landed on three different types of scaling activities, which I’ll briefly overview here and then make the probably controversial claim that “they’re all the same”.

1. Node autoscaling: this is what we were studying in my comparison of the Kubernetes Cluster Autoscaler and Karpenter. How many compute resources do we need at any given time, and can we get those from your cloud provider of choice? In “most” organization, node autoscaling is the purview of the platform or infrastructure team(s).

2. Application (horizontal) autoscaling: if node autoscaling is managing your hardware, application-level autoscaling is managing your software. How many copies of your webserver do you need running at a time? How many shards does your database need? Application-level autoscaling is typically handled by the team that owns the application or service; some applications can scale really well (e.g., your stateless webserver), and some do not (scaling out read replicas for a database is generally not “too hard”, but scaling out write replicas is typically much more difficult).

3. Vertical autoscaling: also called “right-sizing”, vertical autoscaling is a less-frequently-used form of autoscaling5. If application scaling is answering the question, “How many copies of my application do I need?”, vertical scaling is answering the question, “How many compute resources does a single copy of my application need?” Vertical scaling is typically also owned by the application owner, but as mentioned, it’s a less-frequently reached-for tool. It should also be noted that, if implemented poorly, there is a nasty feedback loop between vertical and horizontal scaling, which means that you can’t use both unless you do it intelligently.

But OK, back to my original claim, which I said was going to be controversial: these three types of autoscaling are all the same thing. Or, to put it slightly more precisely, they’re all answering the same question: “How do I ensure that my application has the resources it needs to do whatever it needs to do right now?” The only reason we’ve split “autoscaling” into these three different buckets is to make it easier to reason about6.

In an ideal world, however, we’d have a single autoscaler that behaves as follows7:

1. Ooops, we can’t handle the application load right now!

2. Which thing will resolve the situation? Is it: a. Add more hardware b. Add more software c. Make the software bigger d. Some combination of the above e. Some combination of the above while also kicking out some other software that isn’t needed right now

3. Do the thing you identified in step 2.

If you really wanted to get fancy, you’d through a machine learning algorithm on top of that and add some predictive capabilities to your autoscaler, but I’d argue that in 98% of cases, as long as you can provision new resources quickly enough, predictive scaling isn’t needed.

But anyways, now that we know why we autoscale and how we can autoscale, let’s wrap up this post by answering the question, “How do we know if we’re doing a good job?”

The five key autoscaling metrics

Most organizations look at one metric for their autoscalers: provisioned capacity versus requested capacity. This is, basically, only looking at the cost side of things. The higher your utilization, the fewer resources you need, and the less money you have to pay AWS. But as we’ve already identified, this is an extremely reductive way of looking at things. Autoscaling is a tool that we are using to find our place on the cost, reliability, and capacity-planning spectrum, and if you’re not paying attention to metrics that address all three of these areas, then you’re flying blind. So let’s briefly talk about ACRL’s Five Key Metrics for Autoscaling :tm:

1. Committed capacity percentage - I already described this metric, but just for the sake of putting it in a nice list, your capacity metric measures “how many resources you’re wasting”. Higher committed capacity = less wasted money. Easy. ‘Nuff said.

2. Error rates - If capacity commitment is measuring your cost, error rates measure your reliability. When you get a burst of traffic and you have to scale up, do you throw a bunch of errors until the capacity comes online? When you start scaling down, do you interrupt a bunch of in-flight requests and force retries? Or do you do graceful scaling, both up and down8?

3. Mean Time to First Byte (MTFB) - this is a concept I borrowed from the front-end community, where they have the concept of “mean time to content”: basically, how long does it take from when a user enters a URL in their browser to when they can do something meaningful on the webpage? MTFB is a similar concept: how long does it take from when an autoscaler decides to scale up, to that application or service actually doing useful work?9. If capacity and error rates are tracking your cost and reliability metrics, MTFB is tracking your “scaling responsiveness”.

4. Application disruption - this metric is one of my personal pet peeves, because Kubernetes gives you basically no ability to control for this. But, ideally, your autoscaling engine isn’t causing a bunch of disruption to running pods that is forcing you to repeat a bunch of work, because that’s just wasteful. Stay tuned for the longer rant on this one.

5. Application churn - this final metric is not often talked about, but anybody who’s written an autoscaler (or, heck, any kind of engineering system) has dealt with the problem: your autoscaler thinks you need two and a half replicas, but unfortunately, you can’t have two and a half replicas, so instead you get two. And then three. And then two. And then three. And then two. And then.... Anyways, go look up PID controllers10.

So there you have it! Track those metrics, optimize your autoscaler for all five of them at the same time, and boom! You’re all done!

Ahahahahah. Haha! HAHA! I can’t even say that with a straight face11. Anyways, we’re going to take a closer look at each of these metrics over the next several weeks, and we’ll give some examples from our past experience about why they’re important and what can happen if you don’t pay attention to them.

As always, thanks for reading.

~drmorr

Editor’s note: I’ve been informed that the introduction to this post is convoluted and confusing. If you’d like, feel free to skip it and go straight to the first subsection.

When I was in grad school, I spent a LOT of time thinking about mixed integer programming; in fact, my PhD thesis was primarily focused on developing new techniques for solving (certain classes of) integer programs). In short, if you’re unfamiliar with this concept, an “integer program” is just a system of equations that encodes certain “state” as variables that can only take certain (integer) values. The goal is to find the “best” set of assignments to those variables so as to maximize some objective or goal.

See, the thing is, though, integer programming is hard. Like, really hard. Like, as hard as a lot of other really hard problems. And so when you’re confronted with one of these problems, one of the things you might do is “try to make it easier”. For mixed-integer programming, one of the ways you can make things easier is called the “Big-M Method”. The idea is, “Well, OK, it’s really hard to find integer solutions to the equation x = y, but maybe instead we can replace that equation with x − y ≤ Mz and x − y ≥ Mz for some other variable z and some extremely large constant M”.

The problem with this approach is that M has to be “big enough” so that your solver can’t “cheat” and find a solution to these equations where x ≠ y. BUT, and this is the important piece for the rest of this entire post, which I promise is relevant, the bigger you make M, the harder the problem is to solve in practice1, specifically because M doesn’t have any meaning or relationship to the problem you’re actually trying to solve. And therein lies the problem with Kubernetes.

Weight, weight, don’t tell me

So I’m pretty sure at least all of you are going “WTF is drmorr on about today?” and are ready to ragequit your browser. But if you bear with me2 I promise I’ll make it all make sense.

Let’s back up. I work on this extremely complex distributed system called “Kubernetes”. And because Kubernetes wants to be all things for all people, there are a lot of different ways that any individual user can configure Kubernetes. And in many cases it is impossible3 for the authors of Kubernetes to know how end users might want to configure Kubernetes; questions like “is this thing more important than that thing?” might reasonably have different answers for different people. And whenever we run into one of those questions, instead of doing the hard work of understanding our customers, we just do the distributed-systems equivalent of Big-M and say “Screw it, we’ll just slap a priority on that sucker and let the users decide.”

Here is a (non-exhaustive) list of where I’ve encountered this behaviour:

• The kube-scheduler scoring step: kube-scheduler operates on pods one-at-a-time4, and for each pod it goes through a long pipeline which first filters out nodes that the pod can’t schedule on, and then scores the remaining nodes. There are a lot of different tests that are run to compute the score of a node: things like “how many pods are already on the node”, “how many resources would be left on the node if the pod gets scheduled there”, “are there any ‘soft’ affinity or anti-affinity rules in play”, etc. Now the thing is that each of these tests are weighted, and the weight controls how much the test contributes to the final score. And of course, because different users might have different goals and priorities, these weights are user-configurable.

• The Cluster Autoscaler priority expander: Same deal as kube-scheduler, different problem. Cluster Autoscaler needs to pick one of N different node groups to scale up when there are pending pods in the cluster; it first filters out the invalid ones and then selects the node group based on a weighted rank of the remainder, and the weights are chosen by the user. This one’s fun because in 98% of cases, it reduces to picking the first value in a sorted list; the values of the weights are completely irrelevant.

• The Pod Priority mechanism: Kubernetes allows you to specify priorities on your pods themselves, so that if there isn’t enough space in your cluster for a pending pod and it’s a higher priority than another (currently running) pod, the lower-priority pod will get evicted and the higher-priority pod can then take up the newly-freed slot5. This one’s particularly fun because setting pod priorities can have unintended knock-on effects to your pod’s QoS class.

• Pod Deletion Cost annotations: If you didn’t want just one way of setting pod priorities, don’t worry, here’s another! You can set the controller.kubernetes.io/pod-deletion-cost annotation on your pods to tell Kubernetes, hey, when my ReplicaSet scales down, delete the pods with lower deletion cost before the ones with higher deletion cost! As a fun bonus, Karpenter’s balanced consolidation mode also now relies on this annotation to determine the cost of consolidating a node.

I’m sure there are others, but these are ones that I’ve personally interacted with (and been bitten by) in the past. The problem in each of these cases is the same as the Big-M problem in integer programming: these numeric weights a) have no meaning except that which is assigned by the users of the system, b) changing the value of the weights can have unintended consequences in distant parts of the system, and c) the temptation is to make these weight values really, really large to signal “Hey this thing is REALLY IMPORTANT, don’t touch it”, and bigger numbers = bigger problems.

Let’s look at a couple of examples.

Example 1: kube-scheduler node scoring

The first example of how this can bite you is in kube-scheduler; here, the weights are being applied to the different tests that kube-scheduler executes as part of node scoring. Let’s imagine that there are five different such tests. For our purposes, a test T_i is a real-valued function that takes some arbitrary inputs about the state of the cluster and the node under consideration; in other words, in math-y lingo, T_i: K×n → R.

Each test is then given a user-specified weight wi∈R, and the score for a particular node in the cluster is calculated as the weighted sum of the tests evaluated for that node. Moreover, the selected node is the one that maximizes this weighted sum:

Now, let’s suppose as the administrator of the cluster, you want to ensure that pods always get scheduled on cheaper nodes before they get scheduled on more expensive nodes6. Let’s say that T₅ returns the cost of the node, so you set w₅ to the largest negative number you can think of, say, -100. Then you go about your day, and all of a sudden you get paged because some pod got scheduled to an expensive node before a cheap node! What! How can this be???

Well, in this example, you didn’t care about any of the other tests, so you just left w₁,...,w₄=1. Each of the other tests T₁,...,T₄ returns the value 300, and since the expensive node of course costs $10, the final node score is 200. Turns out that for all the other (cheaper) nodes, the other tests returned 1, so the weighted sum for those nodes was -96.

“Well, ok, that can’t be right,” I hear you say from all the way over here. “Lemme just adjust some of those other weights real quick!” And bam! You have just entered the 12th circle of hell without even realizing it.

Example 2: pod deletion cost and balanced consolidation

In the previous example, there is a single hypothetical user that is controlling all of the numeric weights. So even though the weights don’t have any semantic meaning to the system, there is a single user that can assign some semantic meaning to their choices. This second example is more insidious, because it involves multiple users with competing incentives getting to pick the weights.

In Karpenter 1.14, the Balanced Consolidation disruption mode was introduced. The high-level idea here is that Karpenter shouldn’t disrupt nodes where the benefits don’t outweigh the drawbacks. Since Karpenter’s whole shtick is “save money”, the authors used “the amount you would save by disrupting the node” for the “benefit” side of that equation. On the “drawbacks” side of the equation is “how much disruption does this cause”, and the entire calculation boils down to comparing the “benefits / disruption” ratio to 2.

The problem is that the “disruption” part of this ratio is (essentially) just a weighted sum of the pods on the node, and the owners of the pods (who may or may not be on the same team, and who may or may not have an antagonistic relationship towards each other) get to set those weights, via the pod-deletion-cost annotation. So if engineers on one team see their pods getting frequently disrupted by Karpenter, they’re going to increase the weight of those pods so that some other team of losers gets their pods disrupted instead. And then that other team is going to play a tit-for-tat game until both teams have set their deletion costs to INT_MAX and are furious with each other, and congratulations, you’re back in the 12th circle of hell7.

Who could have foreseen this?

I’d like to argue in this post that the underlying cause of all these problems (including the Big-M problem at the beginning of the post) boils down to two factors:

1. Allowing (essentially) unbounded range on your weights

2. Using unit-less values for your weights

The first problem is somewhat obvious: allowing big numbers means there is always the temptation to just say “well THIS thing is really important, so lemme 10x the weight!” And then later on when the math fails you, you go and 10x some other really important thing, and the cycle continues ad infinitum.

The second problem I think is more fundamental. You’re trying to compare fundamentally incomparable things by assigning them numeric weights. You, as the external operator, give those weights some semantic meaning, but math doesn’t know what that semantic meaning is: all math sees is numbers, and numbers are easy to compare8! The “right” answer here is “don’t compare incomparable things” but I admit that isn’t always practical, so my second-place recommendation is to always make sure the things you’re comparing have units. As we all learned in physics class, unitless numbers are bad.

For example, in the Karpenter balanced consolidation case, an arguably “better” notion of disruption might be something like “time needed to get all the pods on the node back to Running”. Then you can consider the ratio of “dollars saved / time spent rescheduling”, and that lets you start to semantically-encode some of your external priorities into your policy engine. It might be fine to spend 2 minutes rescheduling all the pods if you’re going to save $100, but if you’re only going to save 50 cent, it might not be worth it9.

My last suggestion here is, if you do have to make up some unitless integer weights, consider dramatically constraining the range of values those weights can take on. Nobody can reason about the difference between something with weight 147938964 and 147938965, but people can actually think about the difference between, say, 1 and 3. And as a corollary to this, if you find that you only need a few weight values, maybe see if you can get that number down to two, because then you can replace it with a boolean: I claim that nobody ever needs to set a pod deletion cost in [-2147483648, 2147483647]. All anybody ever actually cares about is “delete these pods first, then delete those ones”, which is just two states.

In closing, just remember, kids: the next time you think about adding in a user-configurable integer weight value to your distributed system (or your integer program), “Dr. Morr says NO!”

As always, thanks for reading.

~drmorr

Huh? What’s a bazel?

So Google has this questionable habit of building something internally, and then releasing a worse version of it as an open-source project, and then throwing a lot of marketing money at the open-source version until it takes over the world. In this post, I’m not talking about the most famous example of this2, but instead a lesser-known project called “bazel”, which is based off an internal Google project called “blaze”.

Bazel, if you’re lucky enough to have never heard of it before, is a tool for building software. It’s kinda like Make, or CMake, or Ninja, or any number of other build systems. The singular task of a build system is: take a bunch of source files written in text, and produce one or more executables at the end. Bazel adds onto this goal, in that it creates “reproducible builds”, which means essentially, if I give you the exact same set of inputs to the build system that I used on my computer, and you build it on your computer, you will get a bit-for-bit identical binary executable at the end3.

There’s a variety of reasons why you might want that, ranging from software provenance and supply-chain security to caching and build timing, and I’d just like to acknowledge that Bazel (probably) is solving a really hard problem that some people in the industry need solved. But, and this is my central thesis for this blog post, the user experience as someone who just wants to compile some code and run some tests is terrible. Here’s ten reasons why.

Reason 1: You have the wrong bazel installed

When I first started working with bazel, I was on a Mac, so I did the natural and normal thing that you do on MacOS: brew install bazel. Then, later, I needed to upgrade bazel and was greeted with an arcane message that said I needed to

cd "/opt/homebrew/Cellar/bazel/9.0.0/libexec/bin" && \
    curl -fLO https://releases.bazel.build/9.1.1/release/bazel-9.1.1-darwin-arm64 && \
    chmod +x bazel-9.1.1-darwin-arm64 && \
    cd -

What? What year is this? I thought homebrew was just supposed to handle upgrades for you? Why are we downloading release targets by hand and installing them in homebrew’s directory tree???

Anyways, not to spoil the punchline, but apparently the actual tool I needed to install is called bazelisk. Now to be fully transparent: if you read the Bazel docs4, this is their recommended approach, but I’m placing this complaint at the top of my post because it is emblematic of Bazel’s apparent disdain for the people who are actually using the software.

If Bazelisk is the tool that developers are supposed to use, why is there a separate homebrew target for bazel? Why does nothing in the tool itself ever tell you, “Hey, you might have installed the wrong thing, here’s the tool you actually want”? Why is the only message it provides when you try to upgrade “Download the latest thing from the releases page” instead of either “just doing the upgrade” or, again, telling you that you have the wrong tool installed???

This is a theme we will return to throughout this blog post: Bazel itself clearly knows that something is wrong, and it clearly should (or at least could) either resolve the problem automatically or tell the user how to resolve it, but instead it does neither of these and expects the user to flounder through extremely arcane and indecipherable error messages. This is not good UX design.

Reason 2: Why are there all these BUILD files everywhere?

One of the ways that Bazel accomplishes its tasks is by building a definitive list of dependencies for every single package, library, or module for the system it is building. This is fine and necessary to ensure reproducible builds. The way it accomplishes this task is by putting a BUILD file in every single subdirectory of your project. Again, also fine.

Now, naturally, any time your build changes (by including a new dependency or source file), you have to update the BUILD file, and thankfully, bazel doesn’t force you to do this by hand. Instead, it provides a tool called gazelle5 to update your BUILD files. BUT—if you, say, happen to forget to run gazelle, Bazel just refuses to compile. It just will complain that “some symbol is undefined”. It doesn’t try to resolve it automatically (which would be the nicest approach); it also doesn’t tell you how to resolve it, which is extra confusing if you’re a newcomer, because clearly the symbol is defined, it’s right &$*#ing there, I’m looking at it in my text editor. All Bazel says, in the middle of a stack of other output6, is “symbol is undefined”.

Again, Bazel knows that something is wrong. It knows how to fix it. And yet, it does nothing, provides no helpful information to the user, and leaves you to scrounge around for feedback until you’ve fully internalized the fact that you just need to run gazelle after every single change to any file in the source tree, just in case.

(A weird aside: if you use various non-code files—e.g., text files for snapshot testing, those files also need to be contained in bazel’s BUILD files, even though they have nothing to do with the generated code. I think this makes sense if you think about it long enough, but it was still weird to me when I first encountered it).

Reason 3: What is up with this syntax?

When you run any bazel command, you have to specify what you’re operating on: e.g., bazel build //some/package. The thing following the command “looks like” a directory path, except it is not. Sometimes you include the beginning forward double-slash. Sometimes you do not. Sometimes you include a trailing ellipsis (...). Sometimes you do not. Sometimes the path goes after a double-hyphen (presumably to separate this from other command-line arguments, which is a fairly standard convention), sometimes it does not.

The point here is: interacting with Bazel requires you to learn a new syntax that looks and acts sort of like a directory structure, but is not actually a directory structure, and if you do it wrong, Bazel just complains about missing files (and/or symlinks, which is extra confusing, what symlinks are we even talking about?). It clearly knows that the command is structured incorrectly, but it doesn’t give you any indication for how to structure things actually correctly, you’re just left to flounder.

Reason 4: How do I even build something successfully?

The whole point—the raison d’être, as it were—of Bazel is to build things. So then, why is it so incredibly difficult to actually build something resembling an executable with the tool? The first time I tried to build things I had my paths structured wrong and it complained about missing symlinks (see Reason #3, also note that I was just following the command that was listed in the documentation, which, sigh. I get that documentation goes out of date, but come on).

Once I fixed my paths, it then failed with the following error:

curl: (22) The requested URL returned error: 401
WARNING: Download from https://some-url.com/whatever failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: no such package '@@some_stuff++whatever+blah_blah_blah_linux_amd64//': oauth failed:

(Note, again, this error is buried underneath a wall of other text).

But OK, clearly it’s trying to download “something” from “somewhere” and it’s either getting a 401 Unauthorized response OR a 404 Not Found response, and it’s not immediately clear which, but more to the point: this error was indicating that I hadn’t logged in to a private docker registry that was NOT hosted at some-url.com. In fact, nowhere in the entire error message does it say anything about Docker.

Also, what on earth is this nonsense?

@@some_stuff++whatever+blah_blah_blah_linux_amd64//

I have no idea what any of that means, except that it’s clearly some dependency that it’s trying to download. Bazel has to know where it’s trying to download from, and it has to know why it’s failing, but it doesn’t actually tell you any of that information, it just gives you an unrelated error message in arcane syntax and leaves you to flounder.

Anyways, I logged into the private registry and tried to build again, and was greeted with the following error:

ERROR: no such package '@@+http_io+whatever_v5//file` java.io.IOException: Error downloading [...] GET returned 404 Not Found
ERROR: /path/to/BUILD:67:42 //thing/im/building depends on @@+http_io+whatever_v5//file in repository @@+http_io+whatever_v5 which failed to fetch.  no such package

OK again so clearly again it’s depending on some package that… doesn’t exist? This makes no sense. Clearly it exists for other people, or everyone would be complaining that everything is broken. Why doesn’t it exist for me?

For quite a while I was stuck at this stage in the process, and resorted to just pushing my code to GitHub and letting CI run the build and the tests, because CI was obviously configured correctly and my system was not, and that was easier than trying to figure out and resolve whatever was broken for me locally. Man it sure would have been nice if Bazel had given me a clue!

Finally I got annoyed enough at the CI delays that I decided to revisit, and discovered (after asking some other folks for help) that Bazel was failing because I hadn’t logged into GitHub with the gh CLI app! WTF??? I thought Bazel was supposed to be hermetic? I mean, OK, look, I’m not naïve, sometimes there’s code in a private GitHub repo, and clearly you’re going to have to log in to access it. But again, Bazel doesn’t tell you what code it’s trying to download or why it failed (or even better, say “hey I need some creds can you please provide them”). It just spits out a page of errors in an arcane syntax that refers to packages and modules that you’ve never even heard of before, and expects you to figure it out.

Reason 5: That error handling, though.

While we’re on the subject, let’s talk about that output. Back in grad school, I cut my teeth on C++/Boost template errors, which were infamous for producing inhumane amounts of error messaging because you forgot a semi-colon. I got really, really good at parsing through C++ errors to figure out what the actual underlying issue was7. The error messages were atrocious, no doubt, but usually the right answer was “scroll to the top and resolve the first issue, and then work your way down”. Sometimes there were so many errors it would overflow your terminal scrollback, so I learned to always redirect compiler output to a file so I could get back to the top.

Fast forward a decade, and now most of my development is either in Rust or in Go. Rust has famously set the high waterline for “good error output”; it (generally speaking) distills the error down to the actual underlying cause, often includes a hyperlink to the official docs with more information, and (9 times out of 10) will give you the actual fix inline. Go has a much more barebones approach to compiler output, but honestly it’s still pretty good: it’s not going to hold your hand through how to fix it, but it will generally tell you what’s wrong in a single line. I’ve never seen pages of error output from the Go compiler, and sure, sometimes I have to Google the answer myself instead of clicking a link or taking the compiler’s recommendation, but it’s fine. I don’t hate it8.

The thing is, with all of these systems, it was easy to identify the actual error message itself. Even with C++, even if the compiler output was thousands of lines long, I always knew I could scroll to the top to figure out what broke.

Bazel, on the other hand, both produces pages and pages of error messages, and puts the actual error somewhere in the middle. I can know that there is a Go compiler error, and even have a rough idea of what the error probably is, but I still have to spend a significant amount of time looking for the actual message. Worse, the actual error message is not singled out or highlighted in any way, while all of the content both above and below is covered with ERROR labels and other scary-looking output. So my experience with Bazel thus far is, if I want to identify a problem, I need to find the least-obvious line in the entire output to determine what to fix.

Reason 6: Compile times are whack

My next complaint is about Bazel’s respect for your (as a developer’s) time. The compile times are through the roof! And I say that as a Rust dev, the language which has as its primary complaint “things take too long to compile”. In some ways I don’t think there’s a lot you can do about this; if you’re trying to make an entirely hermetic build with bit-for-bit reproducibility, that’s going to take more time than “just” running go build. But still. I’ve got stuff to do, I can’t wait 10 minutes for you to finish building my app!

To be fair to Bazel, most builds don’t take 10 minutes, because of Bazel’s local on-disk cache. Congratulations, you’ve identified my next complaint!

Reason 7: Ouch, you punched me in the disk space!

To try to alleviate some of the compile-time issues and need to download every single dependency on the planet for every build, Bazel maintains an absolutely immense local cache on your machine. One of my colleagues was bragging about their 1TB local Bazel cache. WTF??? Look I know we all grew up in an era where disk space was cheap, but it sure isn’t anymore9. And even if it was, seriously? A terabyte of data just to compile some nonsense? I get mad when Cargo doesn’t clean up after itself. I get mad because Docker just likes to vomit all its stuff all over your hard drive indefinitely10. A freaking terabyte???? My god, what have we done.

Reasons 8, 9, and 10: To be continued.

I am just on the very beginning of my Bazel journey, and have captured in this blog post all (or at least most) of my complaints with Bazel’s user experience. But I’m a fairly new Bazel user, and I’m sure that given my experience thus far, there are going to be many more reasons in the future that I run into with this software, so I’ve left space in this blog post to account for those as well.

And look, this is the Internet. I’m sure some of you will come along and say “Well if you just understood the system” or “This question is answered in the docs” or some other variant of “Git Gud, Noob”. My rejoinder to that is: I am a pretty smart person in general. I am able to reason about some pretty arcane shit. I like to work on complicated, hard-to-understand systems11. And I’m sure that, given enough time, I could “git gud” at bazel.

But here’s the thing: I don’t want to. All I want is to build some source code and produce a binary. I don’t want to have to learn an entirely new language, parse through pages and pages of errors that don’t actually even tell you what the problem is, spend hundreds of dollars on a new disk drive, and waste hours of my life waiting for something to build that used to “just work”.

I understand that Bazel solves some real problems, for some real organizations, and that if you have those problems that need to be solved, Bazel might be your best option12. I also want to be clear: I’m sure that whatever Bazel is doing in the backend is probably actually solving those problems moderately well! But I am extremely unconvinced, unless your name starts with a G and rhymes with Froogal, that you have any of the problems that Bazel solves bad enough to subject every single one of your developers at your organization to a build system that treats its users with such disdain13.

Anyways, just remember, the next time you’re reading through hundreds of lines of error messages just to discover that the real error isn’t mentioned anywhere, and you have to go beg one of your more-enlightened coworkers for help: we did this to ourselves. You’re welcome.

As always, thanks for reading 🤣

~drmorr

“Sure,” I said, before I’d gotten my morning caffeine. It’ll probably take about 15 minutes and then I can get on with my day, I thought to myself1. Five hours later, I finally understood what was going on. Two hours after that, I understood it well-enough to be able to explain it to my colleague2. A week or so later, I understood it well enough to write this blog post.

Spoiler warning: there’s no bug in Karpenter, and everything in the sequence logically follows from one step to the next, but (at least for me) the outcome was still surprising enough that I thought I’d write a post about it.

It all starts with a graph

So just to be clear, I’ve filed off all the serial numbers on this post to protect… I dunno, probably nobody, I don’t think there’s anything particularly sensitive here, but anyways. The issue started out with a graph that looks like this:

This graph is showing the unused CPUs in a cluster, aggregated by the nodes in the cluster. In other words, each line represents a different node, and the value at any point is the number of available (schedulable) CPUs on that node. It is a stacked graph so that we can get an idea of both the distribution of unused CPUs as well as the total unused CPUs in the cluster.

This graph is pretty hard to read! But, if you squint at it for long enough, two things become apparent: first, the “available CPU capacity” in the cluster is pretty constant: we more-or-less always have 5 CPUs available. And secondly, despite this, nodes in the cluster are churning constantly. This pattern in this graph repeated for well over 12 hours. What??? Why would Karpenter be doing this? Is this even Karpenter’s fault???

However, after checking the Karpenter logs, I was able to confirm that Karpenter was terminating these nodes because of node consolidation. But I’m still very confused: Karpenter is supposed to consolidate nodes to binpack the cluster better—in other words, after a node consolidation event, the “available capacity” in the cluster should decrease! But that’s not what we see: after a node consolidation event, the available capacity stays constant.

To understand what’s happening here, we need to know about two things: what’s running on this cluster, and how Karpenter’s node consolidation works. Let’s first talk about Karpenter node consolidation, because I hear a lot of lies subtly imprecise discussion about how it works3.

Who’s consolidating who?

The standard sales pitch4 for Karpenter goes like this: “Install this magic autoscaler and all your problems will go away!”

Sorry, that was snarky. Let’s try again. The standard sales pitch for Karpenter goes like this: “Install this magic autoscaler and it will make your cluster much, much cheaper!” That was still snarky5, but is closer to the truth. The question is, “How does Karpenter make things cheaper?” The answer is node consolidation.

At a very high level, Karpenter has two control loops: one is the “scale up” loop and the other is the “consolidation” loop (I wrote about analyzing the performance of these two control loops way back in the SimKube 1.0 days). The scale-up loop is designed to be fast: it responds to any Pending pods very quickly, and launches new nodes to get those pods scheduled ASAP. The consolidation loop is slower; it’s constantly on the hunt for the “optimal” configuration of pods and nodes, where “optimal” in this case means “cheapest”. In practice, what this means is that it is constantly trying to shrink the cluster (aka, bin-pack it) by moving pods off of “lightly used” nodes so that those nodes can be terminated6. In theory, this results in lower cost since you’re not paying AWS for as many compute resources.

Let’s dig into the consolidation behaviour in a bit more detail; if you read the Karpenter docs on this topic, you might be surprised to learn that there are very few knobs you can turn here! You can only set two parameters (on a per-node-pool basis): consolidationPolicy and consolidateAfter. The first parameter controls how Karpenter performs consolidation, and you can either set it to WhenEmpty or WhenEmptyOrUnderutilized7; the second parameter controls how long a node has to sit around before the consolidation mechanism kicks in.

In general, if you’re running Karpenter, you “probably” want to be using WhenEmptyOrUnderutilized; if not, you’re just waiting for pods to naturally cycle out of your cluster before any sort of bin-packing kicks in, and this is in general going to result in a lot of mostly-empty nodes sitting around in your cluster. So really the only parameter that has any meaningful choices is consolidateAfter. This knob helps reduce churn: set it low, and Karpenter will kick nodes out of the cluster extremely rapidly8; but this might be undesirable if you have a new workload coming in that could have used that node. Then you have to spin up a new node, and node provisioning likely takes you longer (and costs you more) than just keeping a “warm” node around in the cluster a bit longer. If you expect this sort of thing to happen regularly, you can set consolidateAfter to a higher value to tell Karpenter to keep nodes around9.

I’ve read through that Karpenter documentation page dozens of times at this point, and every time I’ve read through it, I’ve thought, “It sure is weird that there’s no way to control the utilization threshold for Karpenter’s consolidation behaviour!” In other words, we always talk about Karpenter consolidating “underutilized” nodes, which implies that there is some threshold at which Karpenter considers a node “not underutilized”, and I’ve always been a little surprised that threshold isn’t tunable. But it turns out there’s a very good reason why that threshold isn’t tunable: there is no such threshold.

Specifically: Karpenter will consider a node for consolidation regardless of how full it is. Doesn’t matter if there’s one pod or one hundred pods running on the node, as long as we’re outside the node’s consolidateAfter window, the node is eligible for consolidation. The only10 thing that matters, from Karpenter’s perspective is, “do all these pods fit somewhere else?” If the answer is yes, buh-bye, out you go!

The implication here is that Karpenter’s consolidation behaviour has nothing at all to do with the state of the node in question, and EVERYTHING to do with the state of the cluster around it. A node with a hundred pods in a fully loaded cluster? Won’t get consolidated. That same hundred-pod node in a cluster with lots of other empty nodes? Might get consolidated. Or to put it in even starker terms: Karpenter is using global state (what does the entire cluster look like?) to make a local decision (what should I do with this node?). Generally that is the inverse of what we want in distributed systems.

Now you see it, now you don’t

Now that we (or, at least, I) have a better understanding of Karpenter’s consolidation behaviour, let’s look back at the workloads running on this cluster, and specifically at the graph I shared at the beginning of this post. Does anything seem odd about this graph? I didn’t see it until I looked at the inverse of the graph. Instead of looking at the remaining available capacity in the cluster, let’s look at the committed (scheduled) CPU capacity in the cluster:

This graph is still kinda hard to read, but it’s a little bit easier. We can see that the used capacity in the cluster is more-or-less constant now, even as nodes are churning, but also—huh, look at those spikes11! I wonder what those are? It turns out that in this cluster, there are a number of long-lived pods running on the same node pool as an extremely short-lived Job that a) gets triggered at fairly regular intervals, and b) consumes quite a large amount of compute (relatively speaking). The other important thing to understand is that some of the long-lived pods in the cluster have a termination grace period set, which means that (after Karpenter has marked the node for consolidation) the node will stick around for some period of time before the remaining pods get shuffled off of it.

So now we have all the pieces to understand what’s happening; instead of squinting at stacked timeseries graphs, it will be a little clearer if we draw the nodes and their contents out explicitly. So, consider the following scenario: we have three nodes, A, B, and C with workloads distributed as follows:

At this point, there is no short-lived, resource-heavy pod present, and all of the other pods can fit on two nodes, so Karpenter decides to start consolidating node A:

Most of the pods on node A have been evicted and rescheduled onto node B; one of the pods on node A is still in a “terminating” state while its grace period expires. Now, shortly thereafter, the big, short-lived pod shows up (for the purposes of this example, it takes up an entire node, but it’s trivial to construct cases where it does not). It has no place to run12, so Karpenter launches a new node for it:

Soon thereafter, the short-lived pod finishes, and we now have three active nodes in the cluster again. Nodes B and D are both ineligible for consolidation, because they’ve both had scaling activities recently13; however, node C has just been sitting there for a while! Karpenter can consolidate it! So it does:

And, now we’re back at the beginning of the cycle. This will repeat indefinitely as long as this short-lived, resource-heavy pod continues to appear at regular intervals, and as long as nothing else breaks the cycle. Congratulations! You have introduced constant node churn into your cluster as a consequence of seemingly isolated and independently logical scaling rules.

So, how do you fix it?

Well now, that’s the real interesting14 question, isn’t it? How do you fix this? This is the real challenge with autoscaling: you can add in another rule to try to break the cycle, but how do you know that this new rule that you add isn’t going to have other unintended consequences down the line? Basically: you don’t. There is so much emergent behaviour when you try to scale things up and down, and it’s nigh-impossible to reason about all of it.

I know you all are hoping I have a good answer, but I actually don’t. It’s a little bit bonkers that Karpenter has decided, “Hey, I’ve got this fully loaded node, and this mostly empty node over here, but I’m not allowed to touch the mostly empty node, so I’m going to reschedule ALL OF THE PODS on this fully loaded node instead!” But how do you fix this? Here are some options:

1. Maybe Karpenter should actually have some kind of consolidation threshold for nodes? E.g., if node A is more than 75% full (or it has more than 42 pods as an absolute number), Karpenter won’t touch it. This would prevent nodes with a lot of pods from getting consolidated, but could also prevent the cluster from scaling down when desired.

2. Karpenter could consider nodes that have had recent scheduling activity as ineligible for other pods during its scheduling simulation15. This could also have significant impacts to its consolidation ability, and is also a significant departure from what kube-scheduler would actually do (edit to add: this has actually been done and is available in Karpenter 1.14)

3. Maybe Karpenter needs some kind of “global churn threshold”: it kind of has this, with its node pool disruption budgets, but this just takes into account nodes that are being terminated Right Now, and doesn’t look at historical churn rates.

4. Maybe Karpenter should consider historical usage when making consolidation decisions; in other words, it tries to predict that “oh, this short-lived but resource-intensive pod has been showing up extremely regularly, maybe I should try to leave some room for it.” This would be extremely cool, but predictive autoscaling behaviour is also extremely hard to do, and there’s even more cases where it can go wrong than a purely reactive autoscaler.

There are probably some other potential solutions as will, but which one is right? Maybe you should implement multiple options? But what if they conflict in weird ways with other scaling rules? How do you even know? Most of the time, when faced with an autoscaler problem like this, people make decisions based on extremely rigorous criteria such as “vibes” and “YOLO”.

But man, wouldn’t it be nice if there was a better way?

As always, thanks for reading :)

~drmorr

Six months and several millicores ago I built an AMI pipeline for ACRL. We needed a way to automate builds for the SimKube AMI and SimKube GitHub Action Runner AMI. Each piece of our pipeline was carefully chosen and to this day mostly defensible: Ansible, Packer, and GitHub Actions. Simple and boring. One of these helpful tools was about to ruin my week month.

The Seven Stages

Stage 0: Blame

It was a rainy Tuesday morning2. I logged in to AWS to find an EC2 instance had escaped containment. My AMI pipeline hit an error and unfortunately bypassed cleanup. Not the end of the world but certainly annoying and wasteful. The GitHub Actions log showed an error during the Ansible provisioner phase of my Packer build. The error type was new, an EOF. Which I Googled. The first result said that EOF stood for “End of File” and could be a connection issue. I immediately thought “Oh, some change in our Ansible repo is causing our connection to time out, no big deal.” The last commit to the Ansible repo was authored by so obviously I didn’t cause this problem.

Adding it to our issue tracker, I set it aside. I told myself I had other, more important, things to work on. This would be an easy fix I could get to later. Right?

Stage 1: Retrying

A few weeks later I’m finally getting back to it. Now I really need to sort this out. The AMIs must flow. I look at the last commit in the Ansible repo, the suspect one that shipped. There is nothing obviously wrong with it, just some CopyFail mitigation. I do the most optimistic thing possible: I rerun the pipeline. It’s probably worth noting that our AMI pipeline is initiated via a GitHub Action which clones our Ansible repo and configures Packer. Then, it runs our Packer build which provisions an EC2 instance and runs the Ansible provisioner against it. Finally, once the provisioner steps complete we snapshot the instance and copy it to the regions it needs to appear in3.

The rerun fails: EOF. I rerun it again this time omitting ’s changes by reverting to the previous commit. Same failure, EOF. No detail. I test it again with an even older branch and, yep, EOF again.

This is starting to get weird. That last commit I tried was the code used to produce and ship our last AMI artifact. It isn’t working now. Now I’m scared. A known-good version is now no longer good, so we aren’t chasing a simple bad commit.

Stage 2: Self-Recrimination

I’m looking at my creation with disdain and realize that I am Dr. Frankenstein and this CI pipeline is my monster. Lovingly crafted from parts of FOSS, abstractions I thought I understood. For what? To be trapped in an EOF prison of my own design. There are so many places this issue could be. I wanted to do anything else. I suspect every piece of the pipeline, including its creator, is to blame. Pinching the bridge of my nose I tell myself: “It’s not the end of the world. We’ve got a few culprits.” This issue could be Packer, Ansible, GitHub Actions, SSH, EC2 networking, the bastion, or a poltergeist. At this stage it’s dangerous to rule anything out. It has to be one of them.

Stage 3: Isolation

Debugging CI sucks. So I have an idea: let’s first make sure it’s a CI problem. Because if I can choose, I would rather solve a non-CI problem. I’m an optimist. The easiest isolation is running the Ansible playbook locally, it isn’t the exact same as configuring a remote, but it will rule out a lot of Ansible4 and it’s fast. I run the Ansible playbook locally and it works, no errors. I now know that the vast majority of our configuration management system is working. Now I can move a little closer to reality, so I create an EC2 instance and add it to a test inventory and repeat my Ansible playbook run. Now I’m configuring a remote host, so all of our common remote configurations will also be applied. This is where I expect the issue to arise. It does not. The playbook completes, I do some spot testing to confirm, and everything is as it should be.

The problem was not “Ansible can’t configure the instance.” The problem was “Ansible can’t configure the instance when Packer is orchestrating from CI”. I’m devastated because the issue only appears in CI which means repeated testing is going to be an absolute pain in the ass5.

Stage 4: Bargaining with SSH

If you’ve ever used Packer, or any image build tool, you know how slow the feedback loop can get. I’m devastated because my feedback loop just went from a few minutes to a meaningful fraction of an hour. At this point I know I’ve got something weird going on, and it doesn’t seem to be related to one of our Ansible roles.

So I get sidetracked and start furiously Googling: “Packer+EOF”, “Ansible+EOF”, “EOF+WTF?”. None of the GitHub issues and StackOverflow posts I am finding seem to align with the EOF error I am seeing. Lots of comments like “EOF means End of File, dummy,” as if that tells me everything I need to know. When I stop skimming and start reading in more detail it seems that EOF is often a symptom of connection-related issues. Somewhere in the communication between my GitHub runner that is executing the configuration and the EC2 instance that is being configured we are losing the plot. Ansible is waiting for a signal that it is never going to receive and the result is EOF.

It’s going so poorly at this point I ask Claude and it says this is a classic issue with SSH multiplexing. Spoiler: it was not SSH multiplexing.

I don’t believe Claude. But I try changing our SSH configuration anyway. I change SSH variables in Ansible and pass extra SSH args via Packer. Now I’m turning off multiplexing: EOF, increasing timeouts: EOF, and fiddling with keepalives: EOF. In frustration, I have to walk away.

I’m stuck in an SSH tunnel, and my canary just died.

Stage 5: Log Diving

Okay, it’s a new day now. We’re going back to basics. I turn Ansible verbosity way up -vvvv and set it to output to a logfile. I turn on PACKER_LOG too. The build now takes forever. The reason is that all these logs are being round-tripped over SSH and it’s slow as hell. The build takes so long my build times out at my 90-minute cutoff. I drop verbosity to -vvv; still timing out. I set the build timeout to two hours, I finally make it to my EOF with some enhanced logging. The logging doesn’t provide any more context. I can’t live like this.

While I’m at it I keep a builder EC2 alive after the provisioner fails to inspect it. The EC2 instance is fine6. It’s patiently waiting for the next instruction from Ansible. The logs don’t point to what is broken, but they do tell me one thing: the build isn’t failing because of a specific Ansible task. The issue is not with the target machine. It seems to be somewhere in my connection to it. I haven’t found a smoking gun. I’m starting to become SSH-skeptical.

Stage 6: Acceptance

It could be an SSH issue. We have some proxying going on. Our GitHub Action runner has to tunnel through a bastion to get to our EC2 instance. I start looking into how Packer handles connections, and it has its own SSH communicator. I experiment with Packer’s SSH settings and realize that I can remove another layer: by turning off the Packer proxy and using a standalone SSH agent. On a whim, I add webfactory/ssh-agent, configure the agent once early in the run, and set packer_proxy = false. Boom, it works the first time and we haven’t seen an EOF since.

I wish I could say I know exactly what I was doing wrong with our Packer SSH settings. I am pretty sure it was a “me” problem because lots of folks are using it without issue. Perhaps they have less proxying, fewer long-running Ansible tasks, or simply aren’t cursed by the Packer gods. But if it doesn’t spark joy, remove it. I find the standalone SSH agent much more comprehensible. It actually makes all the SSH management more straightforward and transparent.

Conclusion

Maybe you saw a little of your own CI frustrations in this post. Maybe you think I’m being silly and should just be using the ansible-local provisioner in Packer and cleaning up my AMI before it’s published. That also would have solved my problem, but one of the reasons we wanted to avoid the ansible-local provisioner was specifically to prevent unnecessary cleanup work for our public AMIs. Yes, executing Ansible over SSH is slow, but it’s been fast enough for our needs. We want a simple boring AMI pipeline. By adding a standalone SSH agent we managed to make it a little more boring. We are using Packer only for the piece we really need Packer for and that’s for the best.

I hope that my misguided meanderings trying to debug this issue and recounting it here for your enjoyment will save one person a little time. If you find this post in the year 2056 and are experiencing EOF errors in your Packer pipeline, I would recommend the following EOF pain-reduction plan:

1. Rule out the easy stuff: run a local Ansible test.

2. Test the build against an EC2 instance using a test inventory.

3. If that all worked fine, but you are still getting EOFs, I would recommend testing a standalone SSH agent and turning off the Packer proxy: packer_proxy = false

4. If you are still getting EOFs, I apologize. I have failed to solve your problem. Please send your complaints directly to .

Cheers,

Ian

P.S. set up a Tailscale tailnet for ACRL. So in the future I’ll be adding my GitHub Runner that executes the AMI pipeline to the ACRL tailnet. That means we won’t have to make the bastion hop to reach the EC2 instance. Exciting!

SimKube uses KWOK (Kubernetes WithOut Kubelet) to simulate large clusters efficiently. Instead of running thousands of workloads, KWOK lets us substitute simulated Nodes and Pods that behave enough like real ones to make large-scale replay practical.

Since there is no kubelet running on KWOK Nodes, KWOK is responsible for simulating much of the Pod lifecycle. The problem is that, until recently, Pods moved a little too quickly. At first glance, this seems easy to dismiss. If a Pod takes 50 milliseconds or five seconds to reach Running, it eventually gets there either way, right? Let’s take a closer look.

Real Pods spend time in Pending, images need to be pulled, and containers need to initialize. Those delays are real and vary quite a bit from cluster to cluster. You can imagine pulling a light image cached on a warm Node versus a heavy image pulled from a remote registry. Now compound those differences over thousands of workloads. If every simulated Pod skips these delays, replay behavior starts to drift slightly away from the source cluster. That difference might not matter for every workload, but it absolutely matters when you’re trying to faithfully replay cluster behavior.

In SimKube v2.7.0 we fixed this by introducing configurable image pull and container startup delays. Let’s talk about why and how.

Why Delay?

You might still be asking yourself who cares about these small delays? Well, I do, and I think you should too. So much so that I ran a small experiment this weekend. I created a kind cluster and installed the SimKube tracer1, which captures the event data needed for replay. I then applied a single deployment with 50 replicas and exported a trace. At the same time I applied the deployment, I started a small polling script that recorded how many Pods had reached Running every second. I set this up in a single script, a lazy man’s test harness, so I could repeat this when I messed it up and I did several times.

What the polling data showed was that all 50 Pods reached Running in about four seconds, with the first pods transitioning sometime between the two and three-second marks. With this baseline safely tucked away in a CSV that I won’t accidentally overwrite2, we can move into the fun stuff—simulation!

First, we need to capture the existing behavior in SimKube, so we gather the same polling data for the simulated run, using skctl run without any configured delays. We get about what we expect: Pods transition too quickly. In fact, all 50 Pods in our deployment had reached Running by the two-second mark and the first pods that hit Running between zero and one second3. That’s too fast. We’ve compressed a four-second rollout into roughly two seconds.

We can then repeat our experiment with our configured delays. If you look at figure 1, you can see that by introducing delays we largely corrected the Pod startup profile for our experimental workload. It’s not a perfect match, but the replay now behaves much more like the original workload.

In our simple test, configurable delays moved Pod startup time in replay from roughly two seconds to roughly four seconds, much closer to the behavior of the source cluster. So how did we go about making simulated Pod startup look more like the real thing?

The Design Goals

In a simulated cluster, we want timing to be as close to the source cluster as possible. Having Pods that jump to Running too quickly introduces drift we don’t want. Our goal was to add realistic Pod startup timing to SimKube replays. Not realistic in the abstract, but realistic for the cluster being modeled. It doesn’t take much imagination to come up with examples for why these delays vary. Nearly everyone has dealt with a weirdly large image, a flaky registry, high latency regions, or a container with a lengthy startup time. By making delays configurable we can let the user decide what the delays should look like. Everyone can be unhappy in their own way.

Teaching Pods to Wait

Implementation-wise, we got lucky. SimKube already uses KWOK Stages to move Pods through lifecycle transitions. The problem was that those transitions were happening almost immediately and were not yet configurable. To make startup timing more realistic, we first introduced delays and jitter between the stages to simulate image pulls and container startup. Our stages are based on the Pod General Stages in the KWOK documentation.

We introduced delays at two points in the simulated lifecycle. The first delay models image pull times. The second models container startup before the Pod reaches Running. Figure 2 is a simplified view of the transition timeline. SimKube now adds a new pod-create KWOK stage before the Ready transition, allowing us to model image-pull delay. The startup delay was configured in our existing pod-ready stage.

Advanced Delays for the Curious and Crazy

In SimKube v2.7.0, delays and jitter are fully configurable, meaning that when you run a simulation you can inject the jitter and delays specific to your production environment (in milliseconds) directly in your skctl run command:

skctl run my-simulation \
  --trace-path s3://my-bucket/my-trace \
  --image-pull-delay 1000 \
  --image-pull-jitter 1000 \
  --pod-startup-delay 1000 \
  --pod-startup-jitter 1000

These options are applied globally at the start of the simulation. For backwards compatibility, all delays and jitters default to zero so you can safely omit them if you need to reproduce or compare results with simulations from prior releases. We may revisit these defaults in a future release, but we’ll be sure to communicate any changes.

Behind the scenes, SimKube passes delay and jitter values to KWOK through Pod annotations, allowing a single set of stage definitions to be reused for all our simulated Pods. SimKube sets Pod annotations in its admission webhook the first time a Pod is seen.

Without further Delay

So why go to the trouble of adding a few seconds of delay to a simulation that already has a high level of fidelity? Because startup timing is real. Image pulls, registry latency, and container initialization all vary from cluster to cluster. By making delays configurable, SimKube can more closely model the cluster being replayed instead of assuming every cluster has similar delays.

So without further delay give our configurable delays a try in SimKube v2.7.0 and let us know what you think!

Cheers,

Ian

And look. I know AI slop is a thing, and it’s really tiring to wade through. But accusing other random people on the Internet of cheating (because let’s be real, that’s what you’re doing) isn’t going to solve the AI slop problem. All it’s going to do is make real people hurt, upset, and angry. I say this as a person with relatively thick skin, as someone who doesn’t much care what other randos on the Internet think about my writing, and (also) as someone who doesn’t use LLMs to write on this blog, like, at all. I’m pretty solidly in the “AI;DR” (Artificial Intelligence; Didn’t Read) camp, or in other words, “If you can’t be arsed to write it, I can’t be arsed to read it”.

But I also say this as someone who got one of his blog posts labeled as “hot garbage” by some random Redditor because they thought it was AI slop, and honestly? It kinda stung. My coping mechanism was to take that Redditor to task, both on Reddit, and also somewhat mercilessly on here. You can argue about whether that was helpful or not (probably not) but if you’re going to throw around false accusations about my writing you can bet I’m not going to take that lying down.

There are lots of arguments people make as to why “labelling things as AI slop is good, actually,” and I recognize that I’m not going to change anybody’s mind on this blog post, but I feel like deconstructing some of them anyways:

1. “I don’t want to read AI slop, and I don’t think other people want to read it either, so I’m doing a public service!” Ok, I’m sure you think your AI slop detector is the most refined slop detector and you never have any false positives, but I guarantee you that you’re wrong. One thing that LLMs are good at is pretending to be human, and regardless of whether they will ever be useful, they will definitely get better at pretending to be a human. So I know you think your list of tells (em-dashes, bullet points, “it’s not X, it’s Y”) is better than everyone else’s list of tells, but sooner or later you’re going to get one wrong, and when you do the person on the other end is going to be hurt and frustrated.

2. “Yea, you’re right, I’m going to get some things wrong, but this is a war against Big AI! There’s going to be some casualties!”1 OK, look, putting aside the violent rhetoric, which clearly doesn’t help anybody, the way you “win” your “war” against “Big AI” is by embracing the actual humanity in each of us. We need communities who can come together and support each other and elevate the actual humanness on the Internet. When you come in and start tossing “This is just AI slop hot garbage” bombs all over the place, you’re actually HELPING your perceived enemies, by dividing the communities you should be supporting.

3. “Some people don’t realize that LLMs are bad, and I’m just educating them. I’m doing a public service!” Same comment as (1). Your LLM detector isn’t as good as you think it is. Also this one has an extra layer of elitism and superiority: even if you “get it right” and the target of your critique used an LLM, all you’re doing is announcing that you’re better than them, under a very thin veneer of “just trying to help.”

4. “People just need to have thicker skin; if you can’t handle criticism on the Internet, you shouldn’t be posting stuff for people to read.” Ah yes, my favorite, the drive-by victim blame: “It’s your fault that other people are assholes.” Are we really still doing this in 2026?

5. “This is exactly why we need humans.json!” I am happy to support efforts to re-build the “small web” or use a “web of trust” or whatever else—I do think those things are interesting—but proposing them as a solution to LLM slop is looking for a technical solution to a human problem, which is always the wrong way to go about things. How exactly will this weird, niche technology that most people won’t even know about stop people from making false accusations about AI slop?

6. “Everyone should just put a banner at the top of their website saying that the content is AI-free!” This is a great combination of victim-blaming and naive solutions. For one: why should I have to modify my behaviour because other people are assholes? For two: if this becomes normalized (and I’m already seeing it in quite a few places), the “bad actors” are just going to start putting “This is human-generated” at the top of all their posts, and instead of throwing around accusations about AI slop, we’re just going to start throwing around accusations of “lying and AI slop”. How is that better?

Anyways, a while back I read The AI Con by Emily Bender and Alex Hanna, because I’d heard a lot of good things about it. Unfortunately, I came away from that book profoundly disappointed. One of the key takeaways from that book was “make fun of people who use AI”, and, excuse me, but what???? This is the definition of “punching down”, and it leads to exactly the situation we’re in today, where Internet randos get accused by other Internet randos of doing things they didn’t do, with real impacts to the mental health and physical well-being of everyone involved2.

And look, I get it. AI sucks. The hype is exhausting. There are real, serious, extremely troubling implications with the technology. The rhetoric from the “other side” is over-the-top, overblown, and seems to “demand a response”. But if your response is to shame and make fun of internet bystanders, you can GTFO.

~drmorr

This is a quick story about a bug in SimKube where I knew what the bug was from the beginning and it still took me several hours to actually troubleshoot. In the end, the problem was… exactly what I suspected it was at the beginning, but it manifests in such a weird way that I couldn’t piece it together.

This is a story in two parts1 about mutating webhooks in Kubernetes.

What’s a webhook, precious?

OK, so, Kubernetes lets you extend the system in two ways2: the first is through Custom Resources, namely, you can define your own “Kubernetes objects” that look, feel, and behave just like all the built-in Kubernetes objects (Deployments, DaemonSets, etc). The second way is by injecting random bits of code into the pipeline to inspect or modify these resources at runtime. These random bits of code are called webhooks.

There are two types of webhooks3: validating admission webhooks and mutating admission webhooks. More-or-less, these do exactly what they sound like: they either check that Kubernetes resources or actions meet some policy, or they modify Kubernetes resources before they’re created. The reason they’re called webhooks is that you interact with them over the web the cluster’s network, which I dearly hope is not something that you’ve exposed to the open Internet, but, you know, you do you.

There are two methods to create a webhook4: YAML and JSON. For example, here is the webhook specification that SimKube uses (with some fields omitted for brevity):

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sk-testing-mutatepods
webhooks:
  - name: create-update.mutatepods.simkube.io
    clientConfig:
      service:
        name: sk-testing-driver-svc
        namespace: simkube
        port: 8888
    rules:
      - apiGroups:
          - ""
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - pods
          - pods/status
    failurePolicy: Fail
    matchConditions:
      - expression: object.metadata.namespace.startsWith('virtual-')
        name: virtual-namespaces-only
    matchPolicy: Equivalent

There are two key things to note5: the first is the clientConfig block; this contains a service reference, which tells Kubernetes what the endpoint of the webhook network request should be. In this case, it’s the SimKube driver, which is actually running our simulations. The second key thing to note is the rules: this tells Kubernetes under which circumstances the webhook should be invoked. For SimKube, we want our webhook to be invoked whenever a pod object is created or updated. The third key thing to note is the failure policy, which instructs Kubernetes that the pod creation or update operation should be rejected if the webhook doesn’t work for whatever reason (previously, the failure policy was set to “Open”, which was one of the inciting factors in ACRL’s first public postmortem). The last key thing to note are the match conditions; these are CEL expressions which allow you to narrow the targets of the webhook even further; here we’re instructing Kubernetes to only call the webhook for pods that are created in a namespace with the virtual- prefix.

What happens when a webhook breaks?

As I mentioned a few weeks ago, I’ve been extending SimKube to work better with bare pods (aka a pod that doesn’t have some other owning Kubernetes resource). There are two weird things about bare pods6: if the node they are running on disappears, the pod itself is deleted by the Kubernetes Garbage Collector, and because they don’t have an owner, there is nothing to reschedule them somewhere else. This can lead to inaccurate simulation results if you’re operating with an autoscaler like Karpenter, so I wanted to add a feature to SimKube to reschedule bare pods that were interrupted before they could complete. Fortunately, Kubernetes makes this easy; the two steps7 I took were to a) register a DELETE operation for the webhook, and b) update the driver code to check if the deleted pod had finished running, and recreate it if not.

Once I had written the code to do this, I loaded it up into my cluster to test it out; before the change, if I deleted a node with running pods on it, the node (and the pods) disappeared for good. After the change… deleting the node hung forever, and the pod never went away.

“Aha!” I said to myself. “I have seen this before; the mutating webhook is failing, and the logs for the failure are going to be in a weird place, because that’s how Kubernetes Mutating Webhooks work!” I then spent several hours debugging before confirming that this was, indeed, what was happening. I’ll spare you the details, but at the end of the journey, I found myself looking at this log line from KWOK (as an aside, I recently set up VictoriaLogs on my Kubernetes cluster, and it made this entire process so much easier: 10 out of 10, would read logs in the VM UI again).

failed to apply stage:
failed to delete pod bare-pod-1781282509: pods "bare-pod-1781282509" is forbidden: expression
'object.metadata.namespace.startsWith('virtual-')' resulted in error: no such key: metadata

There are two confusing bits here8: first, the CEL expression above is clearly coming from my webhook, and yet my webhook was never actually even getting called; and second, the error was actually coming from KWOK, which was the last place that I was expecting to see an error. So even though I predicted exactly this scenario at the beginning, I ended up being so confused that it took me a while to understand what was happening.

The short version is: if the configuration for a webhook is broken in some way, the error gets reported to whatever component caused the webhook to get invoked. In this case, Karpenter was calling the Eviction API, which in turn was causing KWOK to make a deletion request to the Kubernetes API server, which meant that KWOK was the component that saw the error. (As I mentioned above, I was expecting the delete request to come from the Kubernetes Garbage Collector, which I naively thought was in controller-manager, so that’s where I spent most of my time looking).

What was the error? Well, for DELETE actions, the body of the webhook request populates an oldObject field instead of an object field to indicate that the object used to be there and now it’s not. Which meant that there was no metadata field on object. The solution was to duplicate the webhook for DELETE requests, but change the match condition to the following:

- expression: oldObject.metadata.namespace.startsWith('virtual-')

As soon as I made that change, everything started working. So anyways, all this is just to say, if you’re writing a webhook and things are behaving in weird ways, there are two things you need to do9: search your logs for error messages that seem like they are coming from really confusing places, and then think really hard about how you broke your webhook configuration so as to cause that particular log line to show up in that particular place.

As always, thanks for reading!

~drmorr

Happy June, everybody! In this post, we’re going to take a quick detour from our strategic plan to do a recap of the second clinic project that ACRL has sponsored with Harvey Mudd College (HMC). Just as a quick reminder, one of ACRL’s goals is to promote and facilitate public, open-source research and development, and the HMC clinic program is a really fantastic way to facilitate this. At HMC, every graduating senior has a requirement to participate in either a thesis project (which is a 1-1 guided research project with a faculty advisor, very similar to the experience you might get in graduate school), or a clinic project, where a team of seniors work together with a faculty advisor and sponsoring company on a project of interest to the sponsor.

Both projects that ACRL has sponsored have been SimKube-related (of course); last year’s project tackled a data-generation project, specifically trying to address the question of “How can we bootstrap a small amount of SimKube trace data into a large amount of data?” One potential use case for this would be as training data for an autonomous Kubernetes AI agent. This year’s project tackled a complementary question, namely: “Given that we have a whole bunch of training data, how can we actually build an interface or environment for an agent to learn from it?”

Answers to both of these questions are necessary (but probably not sufficient!) for developing some machine learning model that can actually operate (or even diagnose!) complex distributed systems like Kubernetes.

The training arena theory

In the AI/ML space, there are lots of “arenas” or “gymnasiums” that people use as a sandbox for agents to perform learning and evaluation tasks. The idea is that you provide your reinforcement learning (RL) agent a carefully-controlled environment and a goal, and through many, many training iterations, the agent eventually learns how to accomplish the goal; and if you’re exceptionally lucky, maybe it can also generalize what it learns. There are a number of popular arenas for various video games, including MineDojo for MineCraft and MiniHack for NetHack1.

And, since obviously Kubernetes is just a giant video game, this concept translates naturally.

So this was the project that I tasked the clinic team with: use SimKube to build a training arena for Kubernetes RL. Just like year, this was a very open-ended “research-oriented” project; there aren’t any right answers, and I was very up-front with the team that if we got to the end of the project and the outcome was “this is a dumb idea and here’s why,” that would have been a success in my book.

Fortunately that’s not what happened!

To go big, you gotta start small

Distributed systems generally (and Kubernetes specifically) are enormously complex2, and it turns out that machine learning isn’t easy either3. I was able to provide a lot of expertise and guidance on Kubernetes and distributed systems, but I do not have the ML expertise that was needed for this project4; fortunately, the faculty advisor that we were working with is an expert in reinforcement learning and machine learning, and was able to provide the needed expertise for this side of the project.

The first challenge that the team tackled was how to limit the scope of the project to something tractable in a 9-month period. There was a lot of excitement on the team about actually training an AI agent for Kubernetes, but this was very quickly ruled out as “too ambitious”, both from a “time” perspective and also from a “we’ve got no GPUs and no money to buy them” perspective. Instead, after a period of intense literature review and research/brainstorming, the team narrowed in on a much more tightly-scoped project: given an (extremely) small action space and a well-defined goal, can we build a set of APIs that someone else could use to train a Kubernetes agent on top of SimKube?

The team spent the bulk of the fall semester fleshing out this concept; the three main questions they needed to answer were “What are the inputs to the agent?”, “What does the agent output?”, and “How does the agent know it’s doing a good job?” To keep the project tractable, the team settled on the following answers:

1. The input to the agent—e.g., what it can observe about the Kubernetes cluster—is just “are the desired pods healthy?” and “what are the resource requests and replica counts for each pod?” This choice of inputs naturally restricted us to “autoscaling-like” behaviours, which not coincidentally, is also what SimKube is really good at!

2. Once the agent has collected its input and made a decision, it outputs a Simkube trace file. The modifications that the agent is allowed to make to the trace is extremely small; it can adjust pod replica counts and CPU/memory resource requests.

3. The agent has “succeeded” at its task when all of the pods in the cluster are healthy (aka, they have entered the “Running” phase at the end of the simulation).

This is obviously a hugely simplified version of the actual problem, but I think it was a great place to start: it was simple enough to reason about in a short period of time, but also complex enough that we were able to uncover some potential problems with the approach. And, it’s obviously extensible: if you want to make the problem harder, you can either expand the observation space, the action space, or the target metric.

By the end of the first semester, the team had implemented an extremely basic POC of the Kubernetes training arena based on SimKube that was able to stitch all of these pieces together. And we’d uncovered the biggest problem we would encounter along the way: time!

Time, time, time, see what’s become of me

It turns out that most reinforcement learning algorithms require hundreds of thousands of training iterations before you’re able to get something “good” out of them. But SimKube runs all of its simulations in real-time! That means if your trace file lasts for an hour, your simulation has to run for an hour5. This is.... kindof a problem. If you need 100,000 training epochs before your RL agent is able to effectively operate your Kubernetes cluster, you need to wait 100,000 hours, which is… 11 years. I’m sure in 11 years we’ll still be using Kubernetes, right? Also I hope you never need to re-train your agent.

Now, it turns out that there are some techniques in the RL community to deal with training against processes that run in real-time, and the most common one is just “run a bunch of training iterations in parallel”. So this is the approach the team took: they extended SimArena to operate in a distributed manner across multiple different machines. Every so often, the arena manager will checkpoint all of the running processes and synthesize the weights for the RL agent together. Along the way, the team got to learn all about the joys of AWS: spinning up EC2 instances, managing S3 buckets whose names are globally unique, and figuring out IAM roles. The clinic team also got to be the first users of ACRL’s SimKube AMIs, which was very exciting and validating.

So now we’ve solved the time problem: if you want your training run to take one day instead of 11 years, you just need 4200 EC2 instances! That shouldn’t be a problem for any of the big AI companies, I’m sure6.

Do you even lift, bro?

The final pieces of the project all came together within the last couple months; as the original goal of the project was to create an arena that anybody could plug their ML agent into, the team built two different hooks into SimArena.

The first was an interface based off OpenAI’s Gymnasium; if you’re unfamiliar (like I was), Gymnasium provides a standard interface for RL agents to interact with… well, just about anything. If you can make your system implement the Gymnasium API, then anybody who builds an RL agent can just plug it in and let it run. The second hook into SimArena that the team built was an MCP server for SimArena. MCP stands for “Model Context Protocol”7, and was an API developed by Anthropic8 for allowing LLMs to interact with arbitrary systems.

The last bit of work the team did was to train a very simple RL agent based on deep Q-learning, which is a standard RL learning algorithm used in a lot of different settings. I’m happy to report that not only did SimArena work, but the agent learned to make the Kubernetes pods healthy!

I’m extremely happy with the work that the team did this year, and really excited to see the results that they were able to put together; but don’t just take my word for it. The SimArena code is on GitHub, and you can also read the team’s final report for more details about their work.

And lastly, I’d just like to say that I’m really excited to have had the opportunity to work with these great students and to do another clinic project at HMC, it’s been an exceptionally valuable experience for me and (I hope) also for the people I’ve gotten to work with. A huge thank you to the team and the faculty and staff at Mudd that have made this so successful!

As always, thanks for reading.

~drmorr

It is kind of wild to me that ACRL was founded 3 years ago; in June of 2023 I stepped out on this crazy adventure and I had no idea where it was going to go or how long it would last, but here we are, three years later, and ACRL is still alive. I feel like “company years” are kind of like “dog years”, which means I’m pretty sure ACRL is now old enough to vote1. It also just so happens to be my birthday today, which I don’t think we need to discuss any further, except to say that “drmorr years” are like “inverse dog years”, which means I am also just now old enough to vote.

I’ve heard from a number of folks that it takes, on average, about five years for a company to go from “I don’t know if this will succeed at all” to something that is somewhat sustainable. Which means that ACRL is about halfway there, statistically speaking! I have been thinking a lot about what that means over the last few months, and one of the conclusions that I reached is that, while I am doing pretty well2, I would like to be doing “pretty well faster”. The traditional thing that tech companies do when they reach this stage is to go raise a bunch of VC money, but that door is closed to me, given my core tenet of “absolutely no way in hell”. So instead, I ended up securing a private investor, who also happens to be me.

One of the goals of this blog (in additional to starting flamewars about Rust vs Go and rambling endlessly about SimKube) is to document my process for bootstrapping a company, so I thought I would spend this birthday post talking a little bit about why we decided that “right now” is the right time for such an investment, what we’re planning to do with the funds, and how we’re going to know if it’s successful.

Supporting the next phase in ACRL’s journey

When I started ACRL, I had a small amount of “seed money” that I took into the business. I knew, intellectually, that “you have to spend money to make money”, and I also knew that there’s just a lot of financial overhead involved in running a business. The rough goal was, if the seed money ran out before I’d gotten people to pay me, I was giving up and going to get a real job.

As evidenced by the fact that I’m still here, that seed money didn’t run out. Between the work that I did at Yelp and the work that I’ve been doing at Astronomer, I’ve approximately managed to 10x the initial seed money for ACRL over the last three years3. It should be noted that I’m not attempting to brag here: this is a remarkable achievement for a small business and it needs to be acknowledged that there was (and continues to be) an extremely large amount of both luck and privilege involved in this success. Nevertheless, it is a success, and I’m proud of where we’ve gotten.

Unfortunately, we’ve somewhat hit the limit with my current business model; with the exception of this blog4, all of our work is supported via consulting hours5, where Ian and I embed as expert consultants in client teams to help with Kubernetes. We are at a point where clients are paying us to use SimKube for them, which is—honestly—an amazing feeling, but there is a limit to how many clients we can support in this model, and that limit is, basically, two. Maybe three if we really wanted to burn ourselves out. It’s been clear to me for a while6 that if I want to scale beyond this, I need to change my business model.

So that’s the goal: we are trying to move a step up the “business” ladder from “selling our expertise” to “selling a product that has been developed using our expertise”. And that’s why we’ve got this investment right now: it’s going to support this next phase in our journey7. The goal (which I fully recognize is ambitious and not fully in my control) is to achieve something like a similar return on investment to what we got on ACRL’s seed money.

Let’s write down some go-tay-jars

Astute readers may remember a post I made a while back about OKRs. So, I’d just like to clarify that the things I’m going to talk about in this section are goals. This goals are artisanal, organic, home-grown, non-GMO, and contain absolutely nothing rhyming with “Bokay Arrr” or “Jay Pee Eyes”.

Anyways, with that out of the way, it’s one thing to say that we’re going to approximately 10x our investment over the next few years, and another thing to actually do it. We need to have some target that we’re aiming for besides “I really hope someone gives us money!” So what are these targets? I’ve identified two:

1. Have a product to demo at KubeCon: in the course of my conversations with potential clients and other interested parties over the last couple years, it’s been made extremely clear to me that SimKube (as it currently stands) is not a product, it’s a tool. It is (in my opinion) an extremely cool tool, but it’s also a tool that can probably only be used by myself and Ian. It’s easy to explain the high-level concept, but as soon as someone wants to go deeper, it’s easy to get lost in the weeds. There’s just something about showing a CLI interface and a terrible Grafana dashboard that makes people’s eyes glaze over; so if we’re going to move from “selling our expertise” to “selling a product” we need an actual, real, demo that we can show to people.

2. By this time next year, we should have approximately doubled our ARR: obviously if we’re going to 10x our investment over three years, we need to be making some progress on that this year. Just like when ACRL started, it’s going to take some time to kickstart this process, so we’re not going to get all of that 10x all at once. But approximately doubling our current ARR seems like an ambitious, but still achievable, goal. As a part of that, I’ve been spending a lot of time learning how to be a salesperson—and it’s kinda fun! I really did not expect to like this part of running a business8, nor did I expect to be particularly good at it, so it’s been a pleasant surprise to learn that I do kindof like it and I am kindof good at it. It’s definitely an untrained muscle for me though, so a big part of being able to hit this goal is going to depend on how quickly I can level up my sales skills9.

So how do we get there?

A couple of weeks ago, I met up with Ian and a few other “trusted” advisors in Portland to talk about our strategy for achieving these goals. It was an extremely valuable (and extremely intense) few days, with a lot of good conversation and good food10. A lot of the conversation revolved around refining ACRL’s target market: what types of companies we want to be talking to, and who we need to be talking to within those companies. The rest of the conversation was concerned with “what we’re trying to sell.”

You’ll be hearing a lot more about the strategy in the upcoming weeks and months leading up to KubeCon, but the high level theme is “Using SimKube to solve autoscaling.” Autoscaling is (still) an area that many companies struggle with, and justifiably so: it turns out to be really hard to do well11. But the general consensus from both our Portland strategy session, and conversations that I’ve had with many other people in the industry, is that ACRL is well-positioned to tackle this problem. I’ve been doing some variant of “distributed systems autoscaling” for over ten years at this point, and as one of my friends pointed out, SimKube is (at least) the third time I’ve tilted at the “distributed systems” windmill: so we have a really great combination of deep expertise, deep excitement and passion for the subject area, and lots of hard problems that still need to be solved.

Like I said: we’ll be talking a lot more about this strategy for the remainder of the year and I don’t want to give away all the magic right now, but I do want to close with a teaser: at KubeCon last year, we ran “Can U Fit In the Kube”, an epic guerilla marketing scheme, which was a ton of fun and generated a lot of laughs and good vibes. We’re going to do something a little bit different at KubeCon this year: ACRL is going to host the world’s first-ever ✨️Autoscaling Grand Prix✨️. Just like everything we do, it’s going to be fun, a little bit snarky, a little bit tongue-in-cheek, and a lot of autoscaling goodness.

We are actively looking for sponsors and partners for the Autoscaling Grand Prix, so if that’s you—or you work at a company that might be interested in this—we would love to talk to you.

I’m so excited that ACRL has made it to its three-year birthday, and I’m extremely excited and hopeful for what’s to come; thanks to all of you for coming along with us on this crazy journey. I’m so grateful for your support.

As always, thanks for reading.

~drmorr

I’ve been doing some work with SimKube for a client recently and have run into a bunch of issues with SimKube fixed regarding its handling of bare Kubernetes pods. Bare pods are kindof an unusual special case for Kubernetes so I thought it might be interesting and/or instructive to see some of the problems I’ve been having and what the fixes have looked like. But before I dive into the details, let me give a quick reminder of what we’re all doing here1:

A Quick Reminder Of What We’re All Doing Here ™️

SimKube is a “record-and-replay” simulation environment for Kubernetes that we’ve been building for the last couple of years; it works by recording a “trace” of events that occurred in a production Kubernetes environment, and then replaying those same events in a simulated environment. In the simulator, we use KWOK to mock out the application pods and the compute nodes they’re running on, while still running all of the real control plane components. This lets us simulate extremely large Kubernetes clusters on your local dev machine2.

Under “normal” operation, the trace file collects changes over time to “higher-level” Kubernetes resources, like Deployments, CronJobs, StatefulSets, or any custom resources that you’ve installed. It then uses real controllers to manage those objects and create the actual (simulated) pod objects that they own. This way you can be sure that the controllers are running in the same way that they might in a “real” environment3.

However, in some cases, you might not want to (or might not be able to) watch the higher-level resources, but you still want to understand some properties of your production cluster (like autoscaling or scheduling behaviour). In this case, you can configure SimKube to watch “bare pods”—that is, the raw pod objects themselves, independent of their owning resources. In principle this will work just as well in SimKube; however, because it’s a bit of an edge case, it hasn’t been as well tested or supported during development, so we’ve encountered some problems in the last few weeks. What kind of problems? I’m so glad you asked!

Bare pods problem #1: Pod Spec Template Paths

The “replay” component of SimKube is called the SimKube driver, which is a pod that sits in your simulation Kubernetes environments; it is responsible for a) replaying all of the events in the collected trace file, and b) intercepting the controller-created pod objects to re-target them to the fake KWOK nodes. In order to apply the correct node selectors, labels, and annotations to the simulated pods, the driver needs to know what these pods “look like”4. All of the core Kubernetes resources (Deployments, etc) and most third-party/custom resources do this via a “pod template”, where users actually embed the pod YAML into the higher-level resource and the owning controller will use that template to “stamp out” copies of the pods it should create. The SimKube driver modifies the contents of this pod template to make sure they run in the right place, which means it needs to know “where” in the higher-level manifest that template is defined. SimKube does this through a podSpectemplatePaths field defined in its configuration5, where the path is represented in JSON pointer notation. For example, the podSpecTemplatePath for a Deployment is

/spec/template

whereas for a CronJob, it is

/spec/jobTemplate/spec/template

But what should the podSpecTemplatePath for a bare pod be? It’s kindof a weird question because there’s no other controller that is “stamping out” copies of that pod, it’s just… the pod. But it turns out not to matter, because the SimKube driver can still do all the modifications it needs in order to run the sim. So, what is it?

If you answered "/", you would be incorrect, but you’d be wrong in good company because both Ian and I thought that’s what it should be at first. However, it turns out that SimKube prepends the podSpecTemplatePath value to a JSON pointer suffix in several places, and using "/" for bare pods results in a double-slash, which is incorrect and caused everything to crash:

//metadata/labels  # This is not correct

The fix for this was extremely simple.

Bare pods problem #2: The pod’s already been scheduled

The second “bare pods” problem we ran into was around scheduling; the normal flow of operations in Kubernetes is

1. Something (usually a human operator) creates a resource (like a Deployment).

2. The Kubernetes controller that owns that resource (e.g., controller-manager does some stuff that eventually results in pods getting created.

3. The Kubernetes scheduler tries to assign that pod to a node, or report that no node is available.

4. If the pod was scheduled to a node, the kubelet starts the containers in the pod and kicks off the application code.

The part we’re interested in here is in step 3; see, in the pod specification, there is a nodeName field which indicates what node the pod has been assigned to. However, this field is immutable; once the pod object has been created, you can’t change it later. Instead, you have to use the bind API to bind the pod to the node (I wrote an entire blog post about this wayyyyyyyy long ago in the early days of ACRL).

Again, this is all fine and dandy normally, but if your trace file contains bare pods, it is likely that these pods have already been scheduled to some node in the cluster that the trace came from, which means that the nodeName field will be set to some node that doesn’t even exist in your simulation environment! This clearly will cause problems; the fix here is again extremely simple.

Bare pods problem #3: My pods are running forever!

My initial thinking when tracking bare pods is that they work just like any other Kubernetes resource: we track when they’re created and we track when they go away, and everything else will work as normal. Pods are weird, though, because they can finish without being deleted. This is normal in the case of Jobs and CronJobs; it turns out to also be normal in the case of tracking bare pods.

Fortunately, SimKube has already built in primitives to handle this for Jobs and CronJobs, using the lifecycle annotation capabilities of KWOK. Basically, in addition to watching changes to “higher-level”6 Kubernetes resources, the SimKube tracer also can be configured to watch for pod lifecycle events—e.g., when the pod containers start and finish. It records all this information inside the trace so these pod lifecycles can be replayed. In the sim, whenever we detect matching pod lifecycle data in the trace, we set the following annotations on the simulated pods:

simkube.kwok.io/stage-complete=true
simkube.kwok.io/stage-complete-time="2026-06-01 11:00:00"

We then create a KWOK stage such that when the indicated completion time has passed, KWOK will move the pod from the “Running” phase to the “Succeeded” phase. Cool! We should just be able to use this for bare pods, right?

Turns out, yes you can! Say it with me now: “The fix is extremely simple.”

HAHAHAHAHAHAHA I tricked you good. The fix, while being relatively small, isn’t simple.

There are two problems that you run into: the first is that, in order to limit the amount of data in a trace file, we only record the pod lifecycle data for pods that are owned by a resource that the tracer is tracking. For bare pods, there is no such owner! It turns out that we can just set the pod to be its own owner inside the trace file; it’s a little weird, it’s arguable whether this is correct or not, but it does work without needing to do a bunch of special-casing.

The second problem is that you need to know whether the pod finished running before the simulation started or not. Remember: the pod object can still exist in etcd, long after all of the containers in the pod have completed running. If you don’t know when the containers stopped relative to the start of the simulation, it’s going to throw off all your analysis78910.

Unfortunately there’s no good way to track this without adding some kind of new field into the trace file, which I was reticent to do. It’s not impossible, and I made some changes with SimKube 2.0 so that I can make these changes in a backwards-compatible way, but I still wanted to avoid it if possible. Then I came up with a ~dirty hack~ genius solution: instead of modifying the trace file format, I can instead just modify the Kubernetes resources in the trace. Now the tracer inserts two annotations on every object it tracks indicating the original creation time and deletion time of the resource.

Most of the time, this information is just ignored; however, if the resource in the trace is a bare pod, the simulation driver will see that these values are set on the pod, and it can then use that to calculate the remaining lifecycle time for that pod object, relative to the starting timestamp on the simulation.

I have no idea if that solution is going to bite me in the future, but for now it seems to work OK.

Also, we’re writing some tests to check for regressions in the bare pod behaviour, cuz those things are *#%&ing weird, man.

~drmorr

Ok, so, suppose you’ve written a Kubernetes controller, you did it in Rust1, and then you realized that “Hey, when this thing breaks it’s hard to understand what’s going on”2. So, you decide to emit some Kubernetes events that give a little more context as to what your controller is doing and what steps it’s taken. You run all your tests3 and expect them to all pass4, because you didn’t change any functionality, you’re just emitting some events, this is basically just like adding some log lines, right? And all the tests fail. Because you’re not just adding log lines, you’re making a network request to the Kubernetes apiserver, and in your unit tests there’s no apiserver listening. What do?????

OK so this post is not actually hypothetical, as you maybe figured out by now. This is exactly what I tried to do with SimKube recently and I spent WAY TOO LONG trying to figure out the ABSOLUTE BEST way to do this and still make all my tests pass. This post also touches on a particular pet peeve of mine around programming practices, which is namely: “DON’T MAKE ME MAKE MY PRODUCTION CODE WORSE JUST TO MAKE IT EASIER TO TEST, COMMA JERK”5. So I figured somebody ought to benefit from all the time I wasted on this, so here it is: all the ways to mock a network call in Rust.

The setup

The kube-rs Rust library provides a built-in interface for writing controllers; the main thing you have to provide is a reconcile function which gets called whenever the controller needs to “do something”. The signature for reconcile looks like this:

pub async fn reconcile<T: kube::Resource>(
    object: Arc<T>, global_ctx: Arc<GlobalContext>) -> Result<Action, Error>

The first argument is the Kubernetes object that changed, and the second argument is a user-defined “context” that includes additional “stuff” that the reconciler needs to be able to operate successfully. For this post we’re concerned with that context argument, which looks like this:

struct GlobalContext {
    client: kube::Client,
    recorder: SkEventRecorder,
}

The first field is the client we use to talk to the Kubernetes API server (we don’t want to recreate that on every reconcile for performance reasons). The second field is our event recorder, and you don’t want to recreate that on every reconcile because it loses its memory of “what events I’ve already sent”, so instead of nice aggregated events (“this event happened 57 times in the last 4 seconds”), you get 57 separate events in 4 seconds.

The event recorder struct is just a wrapper around kube::runtime::events::Recorder from the kube-rs library, which includes a few helper functions for generating and sending the events. This is a simplified version of the real thing:

use kube::runtime::events;

struct SkEventRecorder {
    recorder: events::Recorder,
}

impl SkEventRecorder {
    pub fn new() -> SkEventRecorder { /* initialize a new recorder */ }

    pub async fn send_event_A(&self) -> Result<(), Error> {
        let event_a = events::Event::new( /* make an event */ );
        Ok(self.recorder.publish(event_a).await?)
    }
    pub async fn send_event_B(&self) -> Result<(), Error> {
        let event_b = events::Event::new( /* make an event */ );
        Ok(self.recorder.publish(event_a).await?)
    }
}

And there you can see the network calls we’re making: every publish sends the event off to the apiserver and waits for a response. How do we make this code testable? Let me count the ways.

Method 1: Boxed Trait

This first approach is the most common, and it uses interfaces and dependency injection to provide a real event recorder in your production code and a fake event recorder for your tests. We’re going to change our code to look like this:

#[async_trait]
pub trait EventRecordable {
    pub async fn send_event_A(&self) -> Result<(), Error>;
    pub async fn send_event_B(&self) -> Result<(), Error>;
}

struct SkEventRecorder {
    recorder: events::Recorder,
}

impl SkEventRecorder {
    pub fn new() -> SkEventRecorder { /* initialize a new recorder */ }
}

impl EventRecordable for SkEventRecorder {
    pub async fn send_event_A(&self) -> Result<(), Error> {
        let event_a = events::Event::new( /* make an event */ );
        Ok(self.recorder.publish(event_a).await?)
    }
    pub async fn send_event_B(&self) -> Result<(), Error> {
        let event_b = events::Event::new( /* make an event */ );
        Ok(self.recorder.publish(event_a).await?)
    }
}

struct GlobalContext {
    client: kube::Client,
    recorder: Arc<dyn EventRecordable>,
}

This pattern is familiar to you if you’ve ever written any Go code; literally everything in Go implements an interface simply because it’s the only way to make anything testable. Given the above, you can then create a MockSkEventRecorder object in your testing code that implements the EventRecordable trait6, and pass that in to the GlobalContext object in your tests.

There is actually a slight simplification we can make to this code; the only thing that needs to be mocked is the recorder itself, we can use the same send_event_X methods for both implementations:

#[async_trait]
trait EventRecordable {
    async fn send_event(&self, event: &events::Event) -> Result<(), Error>;
}

struct EventRecorder {
    recorder: events::Recorder,
}

impl EventRecordable for EventRecorder {
    async fn send_event(&self, event: &events::Event) -> Result<(), Error> {
        Ok(self.recorder.publish(event_a).await?)
    }
}

struct SkEventRecorder {
    recorder: Arc<impl EventRecordable>,
}

impl SkEventRecorder {
    pub fn new() -> SkEventRecorder { /* initialize a new recorder */ }

    #[cfg(test)]
    pub fn new_with_mock() -> SkEventRecorder { 
        /* initialize a new recorder with a mock sender */ 
    }

    pub async fn send_event_A(&self) -> Result<(), Error> {
        let event_a = events::Event::new( /* make an event */ );
        Ok(self.recorder.send_event(event_a).await?)
    }
    pub async fn send_event_B(&self) -> Result<(), Error> {
        let event_b = events::Event::new( /* make an event */ );
        Ok(self.recorder.send_event(event_a).await?)
    }
}

struct GlobalContext {
    client: kube::Client,
    recorder: SkEventRecorder,
}

This second variant is slightly nicer in that it reduces the number of abstractions that a dev reading the reconcile function has to look at, but it’s functionally equivalent to the first version. This pattern is deeply frustrating to me for three reasons:

1. You have to use the async_trait crate, because you can’t have async traits in standard Rust. It’s fine, the crate is fine, it works well, it’s literally just an extra import and a proc macro annotation, but it annoys me that I have to use it.

2. I’ve now added a bunch of extra boilerplate in my code that isn’t really necessary. There’s no reason to use dependency injection here aside from making it easier to test. Someone else who comes along to read the code has an additional layer of abstraction that they have to parse through to understand what’s going on, and I’d like to have fewer abstraction layers, not more.

3. Most damning, this code now has a performance impact on what’s running in production! We have two pointer indirections where before we had zero: the Rust compiler doesn’t know how big an impl EventRecordable is, so you must create it on the heap, and because this is highly asynchronous code, you have to use an Arc instead of a Box, so now we have a pointer lookup and a reference counter to care about. Not only that, we’re now using dynamic dispatch (essentially a vtable, aka another pointer lookup) to figure out where the function definitions are for the event recorder.

Now maybe if you’re writing Go code which already does a bunch of vaguely non-performant things, you don’t care, but this is Rust, dammit! You’re supposed to prematurely hyper-optimize everything! Isn’t there a better way?

Method 2: Generics

We can address concern number 3 by converting from dynamic dispatch to static dispatch, which gets rid of the performance impact. The way you do that in Rust is with generic type parameters, and it looks like this:

#[async_trait]
pub trait EventRecordable {
    async fn send_event(&self, event: &events::Event) -> Result<(), Error>;
}

struct EventRecorder {
    recorder: events::Recorder,
}

impl EventRecordable for EventRecorder {
    async fn send_event(&self, event: &events::Event) -> Result<(), Error> {
        Ok(self.recorder.publish(event_a).await?)
    }
}

struct SkEventRecorder<T: EventRecordable> {
    recorder: T,
}

impl<T: EventRecordable> SkEventRecorder<T> {
    pub fn new() -> SkEventRecorder { /* initialize a new recorder */ }

    pub async fn send_event_A(&self) -> Result<(), Error> {
        let event_a = events::Event::new( /* make an event */ );
        Ok(self.recorder.send_event(event_a).await?)
    }
    pub async fn send_event_B(&self) -> Result<(), Error> {
        let event_b = events::Event::new( /* make an event */ );
        Ok(self.recorder.send_event(event_a).await?)
    }
}

struct GlobalContext<T: EventRecordable> {
    client: kube::Client,
    recorder: SkEventRecorder<T>,
}

This code has transformed a runtime cost into a compile-time cost, which is great! The Rust compiler will “monomorphize”7 this into the right thing when we build our binary or run the tests; no heap allocation, no vtable lookup. This is great! The only problem? This is ugly as $#&%. That stupid template parameter has wormed its way all the way up into my top-level GlobalContext struct! Now someone who comes along to read this code has a lot of stuff they have to parse through to figure out what’s going on. This is unacceptable! What else is there?

Method 3: Compile-time feature flags (take 1)

You know what, this whole trait thing is pretty terrible, what if we just… didn’t do that?

struct SkEventRecorder {
    recorder: events::Recorder,
}

impl SkEventRecorder {
    pub fn new() -> SkEventRecorder { /* initialize a new recorder */ }
}

#[cfg(not(test))]
impl SkEventRecorder {
    pub async fn send_event_A(&self) -> Result<(), Error> {
        let event_a = events::Event::new( /* make an event */ );
        Ok(self.recorder.publish(event_a).await?)
    }
    pub async fn send_event_B(&self) -> Result<(), Error> {
        let event_b = events::Event::new( /* make an event */ );
        Ok(self.recorder.publish(event_a).await?)
    }
}

#[cfg(test)]
impl SkEventRecorder {
    pub async fn send_event_A(&self) -> Result<(), Error> { Ok(()) }
    pub async fn send_event_B(&self) -> Result<(), Error> { Ok(()) }
}

struct GlobalContext {
    client: kube::Client,
    recorder: SkEventRecorder,
}

This is kinda cute. We’re using Rust compile-time directives to say “If you’re running this under test, just always return immediately, but if you’re a real binary send an event”8. There are two problems here: one is that if, at some point in the future I decide I need more fine-grained tests for the event recorder, I can’t without jumping through a whole bunch’o hoops. The second is that in the real code, the event recorder and the context are in different crates, and the test configuration parameter doesn’t get passed through to crate dependencies—which makes sense, but means I now have to introduce a new feature into my code and juggle that in various Cargo.tomls. Gross.

Method 4: Compile-time feature flags (take 2)

We can actually combine the two approaches above, and get some godawful frankentrait code:

#[async_trait]
pub trait EventRecordable {
    async fn send_event(&self, event: &events::Event) -> Result<(), Error>;
}

struct EventRecorder {
    recorder: events::Recorder,
}

impl EventRecordable for EventRecorder {
    async fn send_event(&self, event: &events::Event) -> Result<(), Error> {
        Ok(self.recorder.publish(event_a).await?)
    }
}

struct SkEventRecorder {
    #[cfg(not(test))]
    recorder: EventRecorder,

    #[cfg(test)]
    recorder: MockEventRecorder
}

impl SkEventRecorder {
    pub fn new() -> SkEventRecorder { /* initialize a new recorder */ }

    pub async fn send_event_A(&self) -> Result<(), Error> {
        let event_a = events::Event::new( /* make an event */ );
        Ok(self.recorder.send_event(event_a).await?)
    }
    pub async fn send_event_B(&self) -> Result<(), Error> {
        let event_b = events::Event::new( /* make an event */ );
        Ok(self.recorder.send_event(event_a).await?)
    }
}

struct GlobalContext {
    client: kube::Client,
    recorder: SkEventRecorder,
}

This has all the drawbacks of approach 3, with the added benefit that it looks really really dumb.

Method 5: The actor framework

Ok, so this is all dumb and frustrating, let’s take a step back and figure out what we’re actually trying to do here9. We want to: a) send events, b) over a network, c) and test it. Maybe the problem is that we haven’t made our code convoluted enough!

I’m not going to write it all out here, but the other “standard” approach to solving this problem is using something called the actor model. The basic idea is that the bits of code in your program are “actors” and the only way that actors can communicate with each other is by sending messages, typically over some type of queue. So, in this model, we would re-write our reconcile function to just submit events to a queue, and then we would have some other bit of code on the other end that is asynchronously listening to that queue, and forwarding those events along to the Kubernetes apiserver over the network connection, which you can imagine is just another type of queue10.

And actually, the actor pattern is a pretty good one! It has a lot of use cases, especially for more complicated interactions, but remember: we’re just trying to write some fancy log lines here! Come on, what are we even doing, I’m not spinning up a whole queue and separate async reconcile loop to handle the log lines from my primary reconcile loop. This is insanity!

Method 6: Fake the apiserver

Ok, the core problem that we’re running into here is that all of these approaches require changing your production code from something simple and straightforward (“do a thing, record that you did the thing, do the next thing”) into an unholy abomination just to make it easier to test. What if we just… didn’t do any of that? Instead, we could leave our code the same and just spin up a fake Kubernetes apiserver in our test harness to listen and respond to events. Then we can test real interactions with a real apiserver, with only the very tiny problem that we’ve now had to create an entire Kubernetes apiserver so that our unit tests pass, and we’ve introduced a network call into our tests, and networks are famously incredibly robust and never drop packets and absolutely will not create flaky tests all willy-nilly, nosiree.

Method 7: There is no method 7

So that’s it! That’s all the ways I know of to mock out your Rust code for the purposes of testing. None of them are satisfying. All of them feel gross in various ways. And yet, I don’t think there’s a better way to do it? The problem with statically-typed, compiled languages like Rust (and to a lesser extent, Go), is that you have to tell the compiler that you have a Real Type that is going to do Real Type Things at compile time, and the unfortunately reality is that makes testing harder11.

What approach did I end up taking? Well, after venting about all this for a while to Claude, the robot finally told me: “What are you doing, you dumbass12? The cost of a network request is orders of magnitude higher than the cost of a couple of pointer lookups. Just stick it in an Arc and call it a day.”

So that’s what I did. I’m kindof lowkey upset that the stochastic labubu was right about this, but, well, I guess we can’t win them all.

Anyways, thanks for reading! Tune in next time for more SimKube stuff.

~drmorr

Last month I had the pleasure of going to SRECon 2026 Americas in Seattle; I realized that I posted the transcript of my talk but I didn’t actually do a recap of the conference itself, so I’m going to use this post to do so. In short: it was a wonderful conference, and I’m definitely hoping to attend again next year!

The highlights

A loooooong time ago in a past life, I had the opportunity to go to LISA1, which was another Usenix conference focused on sysadmin/devops-type content. Unfortunately, LISA stopped being a thing; the last LISA was in 2021 as a virtual event, and the last in-person LISA was in 2019 in Portland. I had several folks at SRECon this year tell me that SRECon has become kindof the “spiritual successor” to LISA, and I can definitely say that feels true based on my experience! Both LISA and SRECon were/are smaller conferences2 with a really strong focus on high-quality talks and giving lots of space for people to meet and interact3. While some of the talks made me grumpy with how much Ayyy Eyye was in them, there wasn’t a single talk that was bad. All the presenters that I saw gave very highly polished presentations about really interesting topics.

But, as I think is the case for a lot of folks, the highlight of the conference was the hallway track. There’s a lot of space and expectation that you can just walk up to somebody and start a conversation, and then they will see somebody else passing by that they know professionally and be like “Hey so-and-so, we’re talking about X, wanna join us,” and suddenly you’re in a conversation with 10 really smart people you’ve never met before. AND, the thing that’s cool about smaller conferences is that it’s highly likely you will see those same people again later, so you can go back to them and say “Hey, I was thinking about this thing you said yesterday…” and start the whole process over again, and then a total stranger shoves a clipboard in your face and says, “Hey you seem like you have a lot to say, do you want to be on a podcast, we’re recording it in that room, sign up for a time slot here!” and then all of sudden you’re recording a ten-minute podcast episode with some really influential folks from Google4. It’s a really refreshing change from something like KubeCon, where the conference is so big and broad that it can be hard to find “your crowd”, so to speak.

The other thing that SRECon did that I’ve not seen anywhere else is, in addition to two tracks of normal conference talks, every timeslot had a “discussion session” on a particular topic. These sessions typically had one or two facilitators, who might have some light rules/guidelines/suggestions of things to talk about, but by-and-large it was a safe space5 to have an hour-and-a-half focused conversation with a group of ten or so strangers on a dedicated topic. I went to a really interesting discussion on mental health in software engineering, and part of a discussion around AI and SRE/devops6.

The talks

Like I said above, all of the talks I went to were really high quality, but I’ll just put a few of my favorites here (also, the SRECon 2026 YouTube playlist is available, so you can watch all of them online!)

• The WTF Problem: Developer Experience as a Reliability Property: This talk by Nicole Forsgren at Google7, was the second plenary talk of the conference, and I loved the premise: your developer tooling is impacting your ability to quickly respond to incidents. Nicole started off with a series of questions: How many have been paged for something your tooling should have caught? How many have a runbook that you don’t trust? How many have context-switched out of an incident and into “how do my debugging tools work?” (Nearly) every single person in the room raised their hand for every question. She then segued into a new thing we can measure, which she called “Mean Time to WTF (MTTW)”—basically, how long does it take for an engineer on your team to go from “I got paged” to “I understand what is happening here?” Much of the friction that impacts MTTW is about developer tooling that doesn’t work or is out of date, and if you invest in improving in that you will improve your reliability posture. I thought this was a great talk, and it really reinforced my decision at ACRL to “act like a big company” with respect to tooling, automation, and the like: it would be “easier” to just click buttons on the AWS console to get the resources and things that I need, but by spending time getting the automation right now when we’re small means (hopefully) we have less pain in the future as we grow.

• Infinity is not a strategy: right-sizing the cloud: This talk was by Praval Panwar from Microsoft. The talk was largely a discussion about the tensions between cost and reliability, which any of my long-time readers will know is something that I harp on a lot. You can have 100% reliability by scaling to infinity, but that strategy is neither a) feasible or b) cost-effective. Praval spent the remainder of the talk discussing strategies you can use to manage this tension: things like oversubscription, load forecasting, load-shedding, error budgets, etc. etc. etc. I really liked in this talk that he got into some of the math and statistics behind these techniques. At its core, risk is about “quantifying uncertainty”, which sometimes I think devops people forget means “doing math”. Also he mentioned mixed-integer programming, which is near and dear to my PhD research, so I was just extra happy to be in this talk.

• Keeping a Hypervisor Fleet Up to Date with Minimal Customer Disruption: This talk, by Atalay Kutlay at Akamai, was really fun because (again) it’s a really cool application of mathematical optimization to distributed systems. Atalay described how they use mixed-integer programming8 to keep their fleet of hypervisors (read: virtual machines) up to date. The problem, more or less, is whenever you need to update a VM, you have to a) kick all the workloads off, and b) reschedule them somewhere else, before you can update the VM. You want to do this in a “fair” manner, for some definition of fair, and it turns out that you can just write this all down in a big linear program and solve it to get your maintenance schedule! Super cool work, and I continue to be convinced that we could be doing a lot more of this type of optimization in distributed systems than we are doing.

Those were probably my top three talks at the conference, but there were just so many good talks that I feel compelled to put down a list of “honorable mentions”:

• From ESXi to Kubernetes: Modernizing 1,400 Edge Locations with Open Source: this talk was a really nice “user story” about converting a giant grocery chain in Canada to Kubernetes.

• The Case of the Misnamed Cities: CAST Analysis of a Google Maps Incident: this talk was a cool post-incident analysis of a Google Maps issue from a few years ago, and showed compared a systems-theoretic approach to postmortems to a more traditional “root-cause analysis”.

• The Gashlycrumb Tinies of AI Networking You Must Know (or Languish!): this talk was absolutely delightful for the darkly gothic vibes, definitely worth the watch even if you don’t know or care about AI stuff.

• How Security Incidents Are Different … and How They’re Exactly the Same: this was a deep dive into the lifecycle of a security incident by two former DataDog security engineers who would like me to stress that everything in this talk is STRICTLY HYPOTHETICAL and IS NOT FROM A REAL SECURITY INCIDENT. Anyways, tldr, security incidents are just like regular incidents except with more lawyers involved.

AI? In my SRECon?? It’s more likely than you think!

Look, it wouldn’t be a blog post by me if I didn’t include some degree of complaining. And of course, given the current tech zeitgeist, that complaining is (of course) about Ayyyy Eyyyee. Every single talk (including mine, sadly) at least mentioned AI, and many of the talks were fully bought in on “AI all the things all the time”. The first plenary talk was about how we were are all going to survive as SREs in this new magical AI-ified world, and just… SIGH. I am… Just. So. Tired. This discourse is exhausting and it’s everywhere, even in the conferences that I love.

That said, the discourse about AI at this conference was… somewhat… more measured than your standard AI bro on LinkedIn. The first plenary talk (and a number of other talks) spent quite a bit of time talking about Jevon’s Paradox, which more-or-less says “the easier something is to produce, the more expertise you need in that field, not less.” The obvious application here is code generation: it’s now become trivially easy to generate hundreds of thousands of lines of sloppy code. So even if9 gets better than your typical “expert software engineer”, and is able to significantly reduce the error rate in generated code compared to a human, that error rate is never going to go to zero, which means the absolute number of errors in code is going to increase because there will just be so much code out there.

Is this a hopeful view of reality? Eh. Not really, but, I mean, I think it’s a more realistic view of reality. I’m not worried about my job, as I’ve said many times before, I get paid a lot of money because I am an expert in the field. Do I hate that we’ve collectively turned our job description into “software plumbers”, aka, “manage all the shit that AI spews out”? Yes, yes I do. Am I going to continue forcing my way upstream against that trend as best as I can? Yes, yes I am. Am I confident that in a year or two ACRL is going to suddenly have a lot of business opportunities? Yes, yes I am.

But do I wish I could go to a conference and hear a little less about it? Sure do, Jan.

As always, thanks for reading!

~drmorr

We just released SimKube 2.5.0, which as I said on Mastodon, feels like a big milestone for mainly “I like round numbers” reasons; but I think this is actually a pretty major release, as it’s the first SimKube release to include support for a bespoke DSL that I’m calling SKEL, or the SimKube Expression Language. SKEL opens up a lot of really interesting possibilities for SimKube, so I wanted to spend some time in this post talking about it!

Why do we need SKEL?

As one of my friends likes to say, WPAYS? This is an extremely dumb acronym that stands for “What Problem Are You Solving?”, which is nevertheless an important question to ask in many situations. In this case, the Problem I’m Solving™️ is that SimKube trace files (which, as you may recall from previous hot garbage like Anatomy of a Trace, form the heart of the record-and-replay functionality in SimKube) are really hard to work with.

The main issue at play is that trace files are saved in the msgpack format, which is a binary file format that is similar, but not completely identical to, JSON. The decision to use msgpack was made extremely early on in the life of SimKube; I was looking for a file format that would be relatively compact but also easy to work with1, and the tagline on the msgpack website is “It’s like JSON, but fast and small,” and that sounded great to me. And, to be fair, it is both of those! The size difference between a SimKube trace file and the equivalent JSON text file is about 70%2! The main problem is that, unlike JSON and jq, there are no really ubiquitous tools for manipulating msgpack files. The best thing I’ve found is msgpack-tools, which lets you convert between msgpack and JSON; and, up until recently, that has been how I’ve worked with my trace files: convert the trace to JSON, make some changes, and convert back to msgpack.

This approach is fine, except that a) it’s not very user-friendly, and b) the msgpack schema and the JSON schema are very similar, but they are not identical. One of the big differences is that msgpack allows integer-valued map keys and JSON does not. And, in an unfortunate-but-somewhat-painful-to-fix design decision, SimKube trace files use integer map keys to track pod lifecycle events: that is, information like “this pod ran for this many seconds before terminating with this error code”. Because pod names are not a stable identifier, we use a hash of the pod spec to identify these lifecycle events, and that hash appears as a map key in the trace.

This is a problem for msgpack-tools: it can write out a JSON-like text file that includes these integer map keys, but it is completely unable to convert back to the msgpack format from this JSON-like text file because it first tries to parse the text file as JSON. This means that my “write to pseudo-JSON, make some changes, write back to msgpack” workflow is now entirely broken.

If it’s so hard, how does anybody else ever work with msgpack???

The main way that msgpack appears to be used in other applications is a serialization format for over-the-wire communication: meaning that you’ve got two programs on either end of a network connection that need to talk to each other. So, while msgpack doesn’t have a lot of CLI tools to support its use, it does have libraries to support serializing and deserializing msgpack for basically every programming language on the planet. So one option that I considered for solving the “trace modification problem” is “just make people write code”. We could provide a “SimKube trace modification library” that handles some standard sorts of operations, but otherwise allows users to write code to change the trace how they see fit using a msgpack library of their choice. This has the advantage of being extremely flexible, because users can do whatever they want to the trace, but also a bit of a usability nightmare: users now have to understand the internal trace file format, and if that format ever changes, it’s gonna break a ton of scripts3.

So instead, I opted for a narrower approach that still gives users flexibility, but doesn’t require them to understand the full internal workings of the trace file, and gives me the ability to change things about the trace file format in the future without breaking things for existing users: a domain-specific language. This approach, while requiring a bit more up-front work on my side, also means that I don’t have to provide “simkube libraries” for every programming language under the sun; I can continue to use Rust for everything, and users just have to learn an entirely new language all on their own! That seems like a fair tradeoff, right? Right???

Designing SKEL

I’ve not written a full-fledged DSL before, so this was also just an exciting opportunity for me to dip my toes in a new area of computer science. Before I started this whole project, I was dimly aware that there were things like “abstract syntax trees” and “lexers” and “parsers” and a whole bunch of other nonsense. But before diving into any of those details, I decided to just settle on a syntax for SKEL. I wanted the language to be relatively simple and intuitive, but I also knew that I had a wide variety of operations that it needed to be able to support. I tried to make sure that the syntax could handle all of the transformations that I knew I needed, as well as being relatively extensible for the future.

To keep the language simple and tractable to implement, I set a few requirements for myself:

1. Statements should be executed independently of each other; the only coupling between statements is how the impacts the underlying trace file4

2. There is no flow control (no loops, no conditionals, no gotos): every SKEL file is executed one line at a time in order

3. Since we’re focused on modifying trace files, each statement needs two parts: a selector, which specifies what bits of the trace file to operate on, and an action, which says what to do with those bits.

I went through a few different options, including more of a SQL-like syntax, but I finally settled on a functional approach based on the third criteria above. Every statement in SKEL has the following syntax5:

function(selector, action);

A SKEL file then just looks like a sequence of these statements. The supported functions are pretty self-explanatory; right now we just support two, remove and modify; the remove function un-sets a field in the matching objects, and the modify function changes the value of a field in the matching objects. All of the magic is what happens inside these function calls.

SKEL selectors

If you recall from earlier posts on this blog, a SimKube trace is just a timeline of events: each event has a timestamp and a list of Kubernetes resources to apply or delete. So naturally, there are two types of selectors that SKEL supports: timestamp selectors and resource selectors. Timestamp selectors use the magic variable @t, and you can write any sort of relative or absolute query using this variable; @t > 5m means “all the events after the 5-minute mark in the trace file”, for example.

Resource selectors are a bit more complicated, but just slightly; a SKEL resource selector uses jq-style “dot-notation” to dig into the internals of a Kubernetes resource: metadata.labels."simkube.dev/whatever" == "foo"6 selects all of the resources in the trace that have a simkube.dev/whatever=foo label assigned to them. And of course, you can combine these two selectors: @t > 5m && metadata.labels."simkube.dev/whatever" == "foo" means “select all resources in the trace that were changed after 5 minutes, and have the appropriate label applied to them”.

The epiphany that I had while designing the syntax was that SKEL selectors define a set of events; the right way to think about writing SKEL commands is to decide what set of events you are trying to target, and then narrow in on that set using the && operator. But, sometimes, in order to narrow in on the set of events, you need to be able to reference other fields in the resource. For example, you might want to target all resources that have a node selector matching the value of a particular label on that resource. This is where SKEL variables come into play!

$x := metadata.labels."simkube.dev/whatever" | exists($x)
    && spec.template.spec.nodeSelector.myNodeSelector == $x

Here, we are defining a variable $x7 that points to the value of the simkube.dev/whatever label; every statement in a SKEL selector must be a boolean, so in this case we just assert that this label is actually set on the object, and discard it otherwise. Then, the second statement narrows the scope to only target resources whose myNodeSelector matches the value of the simkube.dev/whatever label applied to that resource. This is very cool, in my opinion!

There’s one last feature that SKEL selectors support, which are wildcards. Some Kubernetes fields (like the containers field in the PodSpec) are arrays; we can target individual elements of those arrays using standard bracket notation (e.g., spec.template.spec.containers[2]), but in some cases we may want to target every element of an array; we can do that using the * wildcard. For example, spec.template.spec.containers[*].image == "ubuntu:latest" targets every container using the ubuntu:latest Docker image.

You can also mix-and-match variables and wildcards:

$x := spec.template.spec.containers[*] | $x.image == "ubuntu:latest"

defines $x as the set of full container specifications whose image field is ubuntu:latest. Super cool! I love how a (relatively) small set of selection operations has given rise to a very rich ecosystem of capabilities.

SKEL actions

This section is comparatively shorter than the selectors section; that’s because, once you’ve defined a set of events and resources to take action on, actually performing those actions is pretty straightforward. For example,

remove(metadata.labels."simkube.dev/whatever" == "foo", spec.template.spec.nodeSelector);

just removes the nodeSelector field from every resource with the simkube.dev/whatever=foo label applied to it. Similarly, the modify command would allow you to change the value of the nodeSelector instead of removing it. Of course, variables and wildcards also work inside the “action” side of the command, which lets you build up extremely targeted operations to your trace file as well.

Implementing SKEL

I’ll try to keep this section brief, but for all you programming language theory nerds out there8 I’ll talk a bit about the implementation details of SKEL. The core library that I’m using is called pest, which continues in the grand tradition of “cool Rust projects with really questionable names”9. Pest implements a Parsing Expression Grammar library in Rust. To use it, you define the DSL grammar in a .pest file, and Pest will automagically convert the grammar into a set of Rust enums representing all of the grammar’s rules. So, for example, one of the PEG rules in my pest file looks like the following10:

conditional_test = { resource_path ~ comparator ~ (val | resource_path) }

This defines a grammar rule called conditional_test (accessed in the Rust code as Rule::conditional_test), and a conditional_test consists of a resource_path followed by a comparator, and either an atomic value (like a string or an integer) or another resource_path11.

One of the interesting things about Pest is that the library itself asserts which rules are valid in which positions, so it is idiomatic in Pest code to sprinkle .unwrap()s and unreachable!()s everywhere based on where you are in the parsed expression. This takes a bit of getting used to.

Now if you know anything about designing programming languages, you’re probably expecting that the next words out of my mouth are going to be “abstract syntax tree”. But, you’d be wrong. An abstract syntax tree is a data structure that, more-or-less12, lets you unambiguously specify order of operations: in other words, all those pesky rules you learned in grade school about whether you add things before you multiply things. However, for SKEL, this turns out to be entirely unnecessary: there is one unambiguous way to parse a SKEL file, and that’s from left-to-right. I no longer have an abstract syntax tree, I have more of an… abstract syntax path. This turns out to be very handy. It means that all of the logic for SKEL is contained in about 2000 lines of code, much of which is pretty straightforward “convert this Rule into this data structure”.

Anyways, once I’ve loaded everything into my ASP, I can just traverse through all the statements in the file and apply them to the input trace one-by-one. The mechanics of this are pretty straightforward, the most complicated bits were just about keeping track of the variable state so that I can correctly know which variables are defined at which point in time, and to what values they point.

There was one open question about the order in which statements should be applied: should I apply each statement to the entire trace file before moving on to the next one? Or should I loop through the trace file once, and apply every statement in order to each event? In the interest of “keeping statements and effects local”, I elected for the former; in other words, once an event in the trace has been processed by all SKEL statements, it will never be touched again. It “feels like” this will be easier to reason about in the long run, but I don’t actually know for sure.

Deleting code feels good man

I’ll just wrap this post up by saying that SKEL is actually the second attempt I’ve made to solve this problem; about a year ago, I wrote a bunch of code to “validate” a trace file, and confirm that it won’t cause any common problems in a simulation (example of a common problem: a Pod needs a Secret injected as an environment variable, but the Secret isn’t defined in the simulation environment, which probably you do not actually want, and the value of the Secret doesn’t actually matter anyways because there’s no code running in the Pod to use the Secret). One of the features in this validator allowed users to auto-apply fixes to the generated trace; unfortunately, for every class of problems, there are many different types of fixes that you could apply. In the Secret example above, for instance, you could either remove the secret from the pod specification, or you could create the secret with a placeholder value at the beginning of the simulation. And who knows? Maybe neither of these solutions is what you actually want to do.

The logic to keep track of these potential fixes, and which once were applicable at which times to what events, was some of the ugliest, most complicated code in the entire codebase. I had visions of expanding the feature, and eventually making it so that you could even have user-editable “fixes”, but the code was so gross already that I just kindof abandoned it. It worked! But it wasn’t pretty and I didn’t want to add anything else to it.

However, with the advent of SKEL, I was finally able to just rip all that out and simplify a lot of things: now, instead of auto-fixing issues, the SimKube validation code just prints out a sample SKEL file that will address the issues that were detected in the trace; users can look at that and decide whether those are the correct fixes, or if they’d like something different. Then they can modify it how they’d like before applying the transformations and running a simulation. Even with all the complexity of the DSL and everything, this is a much better approach! And I’m very happy about it.

Everything we’ve talked about is now available in SimKube 2.5.0 and there’s a bunch more details in the documentation as well, but this has been a really fun new addition to SimKube, and I think it’s going to open up a lot of possibilities in the future!

I hope you enjoyed reading this as much as I enjoyed building it, and as always—thanks for reading.

~drmorr

I’ll explain what each of these AMIs are and how we build them in due course, but first off, let’s address an important question:

Two AMIs, in THIS economy?

We know it’s crazy, who even has the action minutes to raise AMIs these days; we sure don’t. But we had a problem, or maybe an opportunity. SimKube just keeps getting better and better but configuring it can be, frankly, difficult. Building high-fidelity simulation environments requires installing and configuring a long list of tools: kind, KWOK, kubectl, Docker, prometheus, and SimKube, to name a few. So spinning up a ready-to-go SimKube environment takes some doing.

Internally, we have a configuration management repository called isengard2. It is ~48k lines of pure Ansible bliss. We use it to automate the deployment of repeatable simulation environments. It occurred to us that users of SimKube probably don’t want step one of using it to be “here’s ~48k lines of Ansible, good luck!”. It turns out there is a better way: a custom SimKube AMI.

Why not a Docker image like a normal person?

That’s a fair question. We did evaluate using a Docker image because one of our primary goals is a fast, one-click startup.

The challenge is that SimKube relies on kind, which spins up Kubernetes nodes as Docker containers. Initializing and configuring the kind cluster requires access to a live Docker daemon. During docker build, there is no Docker daemon available inside the build environment, which means we can’t just “run all the setup steps in our Dockerfile” and ship the result.

We also looked at snapshotting a running Docker environment, but that’s complicated for a different set of reasons. So after a long side quest that included Vagrant and QEMU, we realized what we actually need isn’t a container image but a prebuilt machine image that preserves the state of our configured simulation cluster. Since we primarily work in AWS, an AMI fits naturally.

Baking the AMIs

Fortunately, baking AMIs is a fairly straightforward task that our ancestors have been doing for thousands of years. We can reuse a lot of what we have already built in our configuration management system (which I will remind you is lots and lots of Ansible). We use Packer to bake our AMIs, so the first step is selecting a base image which our custom AMI will be built on top of. We chose Ubuntu 24.04 LTS for its stability, compatibility with our tooling, and long term security patching.

Using Packer we can initiate an automated build via a GitHub Action. For configuration, Packer includes a range of provisioners—including one for Ansible—so we are able to leverage our existing configuration library in isengard. The GitHub Action itself is fairly simple: it clones the repo and runs packer. This helps keep our Packer configuration sparse and maintainable. We only need to configure a handful of things: the Ansible playbook to run, the region of our builder, our base AMI, regions to copy the finished AMI to, and any cleanup activities or supplemental provisioners.

Our AMI pipeline is triggered by a weekly cron trigger, or by a manual build via a dispatch trigger3. After some bake time, we end up with a custom AMI image backed by the snapshot created during the build. That AMI is then listed on the Marketplace.

AMI patching

Shipping a public AMI means we own patching it. At a minimum, Ubuntu is going to ship security patches (we really want those) and there will be patches for other software in our stack. Instead of patching in place, we treat each AMI build as an immutable artifact tied to the configuration management repository (isengard) git hash used to produce it. Every build is traceable and reproducible.

Every time our pipeline runs, it starts fresh with a clean Ubuntu image and pulls down the latest patches so each AMI is fully up to date at build time. The result is a simple, deterministic build process that is easy to maintain. AMIs are short-lived, they won’t drift over time, and there is no ambiguity about which code produced which image.

The tradeoff is that older AMIs are never patched. If you launch an older version, you get exactly what existed at build time. For our use case, this ends up being a feature since we value reproducibility. This comes in handy for debugging thorny issues in our AMIs, especially those that manage to bypass our validation tests. We can fire up an AWS EC2 instance and watch one of our services get clobbered in real time by some bad code that I definitely didn’t write.

For the most part our AMI pipeline quietly churns out new images. To keep our account from piling up with tons of old AMIs4, we use a Packer post-processing block to deprecate old AMIs and clean up their snapshots automatically.

You said TWO AMIs

I did say that! We have two versions of our AMI. The first is the free SimKube AMI with everything needed to run SimKube including a running kind cluster and management tools. This is our free-to-use simulation environment. All the user needs to do is launch it in AWS EC2 and get right to running simulations—though you will need a trace from the cluster you are simulating.

The second AMI is our SimKube GitHub Action Runner it includes everything in the SimKube AMI but also has some extra configuration applied. We use an iterative build process, so this version is literally built on top of the base SimKube AMI5.

This cuts down on build time and allows us to patch the GitHub Action Runner software independently of the base AMI if we wish. The additional configuration in this version is the GitHub runner software and a systemd wrapper that manages it. We use this version to run SimKube in CI pipelines (via GitHub Actions). Effectively, this AMI is primed to register itself with a GitHub repo as a custom action runner when it receives the information contained in our User Data script.

A world of opportunities

Our SimKube AMI is a step forward in making SimKube approachable and easy to use. Instead of spending a few hours setting up a simulation environment, you can grab the SimKube AMI off the AWS Marketplace and have SimKube up and running in a couple of minutes. You will need to grab a trace from your production cluster, but the environment for running those simulations is available at the click of a button or at the end of a AWS CLI command.

We want to continue to extend Kubernetes simulation into CI pipelines using our GitHub runner AMI. The vision is an engineer, maybe you, checks in some change to your cluster. Then, SimKube CI simulates it based on your production cluster and sends you back metrics you can use to evaluate your change before it hits production.

Today, ACRL is already running small simulations in CI in the SimKube repo. We have developed custom GitHub Actions, available in our simkube-ci-action repo, to make launching runners backed by SimKube AMIs as easy as adding a few lines in your GitHub Actions workflow.

So maybe you find SimKube interesting but setting it up has been too much of a hassle. Or perhaps you are already running SimKube locally but want to run a dozen simultaneous simulations in AWS. The AMIs are there for you, and the SimKube AMI is free-to-use–though you still have to pay AWS for the compute (sorry).

If you want to learn more, we’ve added a new SimKube in the Cloud section to the documentation that walks through how they work and how to get started.

So get out there and simulate some trouble… before it makes it to prod!

Cheers,

Ian

Last week I had the privilege of giving a talk at SRECon Americas in Seattle; the recording of the talk is available on the Usenix YouTube channel, but I thought for this post I’d do something a bit different and write an (approximate) transcript of the talk for those of you who don’t want to wait and/or prefer reading things to watching videos. The talk is, of course, about SimKube: I hope you enjoy! I’ll update this with a link to the talk once it’s posted.

Thanks all for coming; the title of my talk today is “Stop Reading Changelogs: Safer Kubernetes Upgrades with Simulation”.

On Pi Day three years ago, March 14, 2023, Reddit suffered a 100-pi-minute long outage, a 314-minute outage. In the extremely detailed and well-written postmortem, it was revealed that one of the primary causes of this outage was a Kubernetes version upgrade from Kubernetes 1.23 to 1.24. Specifically, in this upgrade, Kubernetes changed one of the labels that was applied to its control plane nodes, and Calico, the service mesh component used by Reddit at the time, was using a label selector that pointed to the old label for the node instead of the new label. This caused Reddit to go hard down for around 5 hours.

Read more

OK, as promised last week, this post is a follow-on to my previous write-up about running a single-node Kubernetes cluster at ACRL. After calling that post “hot garbage”, the redditor who commented on the post went on to say that he was hoping for more details and less high-level pish-tosh1. So, random redditor, this 4000-word manuscript is for you! I hope you brought your coffee.

Project Goals: Quick Recap

Before we dive into all the arrows in the architecture diagram, I just wanted to quickly review what the goals of this project are:

1. Run a Kubernetes cluster in the cloud,

2. as cheaply as possible,

3. and schedule a workload on it. Lastly,

4. it should be one command to completely destroy and recreate our entire stack if we need to.

To recap what I discussed in the previous post, we’re running k3s on a single AWS EC2 spot instance. We’re not using EKS, a) because it’s expensive, and b) because we’re supposed to Kubernetes experts over here and not just some random hack jobs, so we should probably have some expertise running Kubernetes. We’re running on a spot instance a) because it’s cheap, and b) because I wanted to know if it could be done. If you’re following along at home and you’re not running on a spot instance, you can probably do away with a lot of this complexity; but on the flip side, running on spot serves as a forcing function to build the automation and tooling to easily recreate your entire state from scratch, which is in my experience an extremely underrated superpower.

In my previous post, we got to the point of “having a running k3s node”, but we didn’t actually get to the point of “running a workload on it”, partly because I ran out of time, and partly because I thought that once I had the node up and running, running the pods would be easy. It’s just YAML, right2???

Anyways, in the last post we also didn’t define what workload we’re going to be running on our Kubernetes cluster. I have quite a few things I eventually want to run, but my first goal was just to run a simple XMPP server. I’ve wanted a good internal “chat” solution for a while, I really don’t want to use Slack or Discord, and it turns out that XMPP has been living a healthy thriving life for the past 15 years, even if most people don’t realize it. So after a bit of research, I decided that I was going to get Prosody running inside my Kubernetes cluster. It’s small and lightweight enough that “configuring Prosody” wouldn’t take too long, but it would also exercise a bunch of features that I’m going to want later: specifically, ingress, certificates, and persistent data.

I also also didn’t explicitly discuss the fourth goal in my last post, but this is really important to me. I’m a very big proponent of the “GitOps” pattern3. If you can check in your desired infrastructure state into version control, and it’s easy to recreate and/or get back to a “known good configuration”, that makes everything so much easier down the line. So for me at least, it’s worth it to spend some more time up front doing things “right” to (hopefully) make my life a little easier in the future4.

So with that said, let’s go into the nitty-gritty details.

Revisiting Middle-earth

In my last blog post I explained that I have two repos named after famous locations from Lord of the Rings: Moria handles my “infrastructure as code” and Isengard is my server configuration management repo. Since that post, I’ve added a third git repo, Mirkwood, which contains all of my Kubernetes manifests5. You can see in the above diagram how each of these repos interact with components in AWS, shown by the colored solid arrows (green is moria, blue is isengard, orange is mirkwood, and the dashed lines indicate physical hardware connections or network connections); feel free to refer back to that diagram as we continue.

Moria

My last post did a pretty good job of covering the basic AWS infrastructure setup, so I’ll just quickly recap it here. In Moria, we use Pulumi to create the following AWS resources:

1. A single bare EC2 node as an SSH proxy/bastion host/NAT gateway into my VPC.

2. A single-node AutoScaling Group (ASG) which contains the k3s server node, registered as a spot instance.

3. A persistent EBS volume which is auto-attached to the k3s server node on first boot (via a systemd service), which serves as the data volume for k3s.

4. An internal zone in Route53 which I can use to store a DNS A record pointing to the k3s server.

5. An AWS Lambda function, run as an ASG Lifecycle Hook which updates the A record whenever the k3s server is terminated and recreated.

Since the first post, I’ve had to add a number of other AWS resources as well into Moria to support various tooling further down the stack. XMPP requires TLS these days, which means I need a certificate; I considered whether to use the CA that I set up earlier in the year for this purpose, but those certificates are for a very different purpose than what I want now. I also considered whether to create a second CA for “internal services” but that also didn’t feel like a good approach. Fortunately, Let’s Encrypt is a thing; I’ve been using it for a long time for my personal website, and it has good integrations with Kubernetes, so this felt like the right approach.

Certificates need to be signed for a particular domain, however, and Let’s Encrypt needs to know that you own the domain that it’s signing certificates for. But my Kubernetes cluster is inside a private VPC that Let’s Encrypt knows nothing about! So this means I need to use Moria to add

6. Two more hosted zones67 into my infrastructure; a public zone that Let’s Encrypt can use to verify ownership, and a private zone that will host the A records for my XMPP server. Note that we don’t actually configure those A records with Moria, however, because the IP address of Prosody can change.

I also didn’t talk about this at all in the last post8, but Moria also manages

7. All of the AWS IAM (Identity and Access Management) permissions for this configuration.

8. An encryption key in AWS KMS (Key Management Service) to help with secure secret storage.

IAM permissions are some of the most arcane and convoluted things I have ever had to work with, but essentially every component above requires a corresponding IAM policy that enables it to perform its task. The proper, most secure way of managing these permissions involves things like short-lived temporary credentials and OIDC and JWT and a whole bunch of other ugly acronyms, but frankly that stuff gives me heartburn every time I look at it, so for right now I just have a bunch of “service account” IAM users with static credentials configured in Moria. This is marginally less-secure, but I make it easy in Moria to rotate the credentials, and try to keep “what they can access” scoped as narrowly as possible. Someday in the future when I can afford to hire an infrastructure security engineer, I’ll make their first task to rip out all the service account users and replace it with the OIDC thing9.

I’ll lastly mention one note about the single-node spot instance: previously I had configured my ASG to launch a t3a.medium, which uses an AMD processor, has 2 CPUs, and 4GB of RAM. My expectation was that, since this is a (relatively) small instance type, it would have pretty high availability, since AWS can probably bin-pack that more-or-less anywhere. That expectation was extremely incorrect; the k3s server node was getting disrupted between 5 and 10 times a day. Since then, I’ve expanded to allowing any of t3a.medium, t3.medium, t3a.large, or t3.large spot instance types10, and using the “price capacity optimized” allocation strategy: this essentially tells AWS “pick an instance type that is relatively cheap, but also is relatively unlikely to get interrupted”. Since I made that change, I’ve been running a t3.medium spot instance for the last 3 days with no interruptions!

Isengard

In the last post, I gave an extremely high-level overview of my Isengard (Ansible) configuration management setup. Just as Pulumi or Terraform let you define AWS resources as code, Ansible lets you define software resources as code. In Ansible, you write all of your software configuration using a combination of YAML files and Jinja templates. These configuration steps are bundles into “roles” which are then rolled up into “playbooks”. The idea is that a server can have multiple roles, each of which has its own software and configuration installed, and the playbook is what tells Ansible which roles should be applied to which hosts.

In Isengard, we have three playbooks:

1. The SimKube playbook: this is used to quickly and repeatably create SimKube environments. It’s also what’s powering our SimKube AMI and Github Action Runners, which I’ve been teasing in several posts, but will be getting its own blog post very soon now. So I won’t say anything further about this.

2. The bastion playbook: this playbook manages the “entry point” into our private VPC.

3. The k3s server node playbook, (also used to generate the AMI that packer builds for the k3s root volume).

In the previous post, I mentioned that I was using SSH tunneling through this bastion host, but that has since changed. I configured Tailscale, and I have no other words for it beyond “freaking magical.” I understand (somewhat, at a high level) how Tailscale works, but the experience has been so unbelievably good that I wish I’d done this years ago. Everything Just Works; I now have my bastion host configured to forward both DNS lookups into my private VPC, as well as to expose the internal routing tables from my VPC to any other client in the tailnet. It really is magical, there’s just no other way to describe it.

I am also running the fck_nat NAT gateway on my bastion node, to allow everything inside the VPC to talk to The Internet :tm:. AWS does actually provide a native/built-in solution for this, but it’s frankly hard to describe it as anything other than price gouging. A single NAT gateway instance costs $30/month just to exist, and then they charge you on top of that for all of the traffic that goes through the NAT! For comparison, I’m running fck_nat on a t4g.nano, which works just fine and costs me $3/month.

On the k3s server node, I am obviously using Ansible to install k3s, kubectl, and other supporting tools. I also configure the k3s systemd service here. This is what my k3s config looks like:

---
disable-helm-controller: true
disable-network-policy: true
tls-san: "k3s-server-0.uswest2.acrl.dev"
secrets-encryption: true

I disable helm because, ew gross. I disable the network policy controller because I don’t care about network policies, and that controller expects a stable IP address which we obviously don’t have. The tls-san block gives a stable domain name to issue the Kubernetes certificates for11, and the secrets encryption line enables AES at-rest encryption for Kubernetes Secret objects.

For a brief period in time I also specified a stable Kubernetes node name in this config, but that ended up causing more problems than it solved: whenever the node was interrupted/restarted, it would remember all of the state of the node from before it was interrupted, which led to some extremely weird “stale state” issues. More on this later.

For what are probably “premature optimization” reasons, this config file is not baked in the k3s AMI, but instead is generated as an ExecStartPre script in systemd; the script looks up the tls-san name from a tag on the EC2 instance, which allows me to dynamically detect the hostname and potentially run more of these clusters in the future.

There is also an ExecStartPost k3s systemd hook which is configured by Isengard that does the following:

1. Applies the EBS CSI Driver “not-ready” taint to the node (if this doesn’t mean anything to you yet, don’t worry, just keep reading).

2. Cleans up any Node resources that are left over from when the spot instance was restarted.

The latter point is necessary because Kubernetes doesn’t actually ever delete nodes that are stored in its database, it just lists them as NotReady in perpetuity. This isn’t really a problem, except that it looks ugly and takes up space. If we were not using k3s, we would configure the cloud-controller-manager to monitor the state of our AWS nodes and delete the ones from the Kubernetes database that don’t exist anymore; but k3s ships with its own, slimmed-down cloud controller manager that has some other nice features, and I didn’t want to disable that. So instead I just stuffed the cleanup into this post-run hook.

Mirkwood

Whew! We’re almost done here! We’ve got one last git repo to cover before we can have a working messaging app! Mirkwood contains all of my Kubernetes manifests, and it uses kustomize to provide the configuration management tooling12.

There are (currently) five applications running on my cluster:

1. The AWS EBS CSI driver

2. The AWS node termination handler

3. cert-manager

4. The external-dns operator

5. And, lastly, Prosody itself, aka, the thing we’ve been trying to run this whole damn time.

The EBS CSI driver watches for Kubernetes Persistent Volume Claims (PVCs) and handle the creation, deletion, mounting, and unmounting of those block devices into Kubernetes pods. This is necessary so that my XMPP/Prosody pod can have persistent data across pod restarts. The EBS CSI driver must be running before any pods that request PVCs, so the CSI driver pod tolerates the taint that we created as part of the k3s configuration, and once it’s running and healthy it removes the taint so that other pods can schedule.

The node termination handler watches for spot interruption warnings (which AWS provides with a 2-minute time window) and triggers pod and node cleanup ahead of the interruption. This is necessary so that the EBS CSI driver can actually cleanly unmount the EBS volume before all the hardware is rudely yanked out from underneath it13.

Cert-manager, as we’ve discussed, is responsible for talking to Let’s Encrypt and getting certificates for my XMPP server. It’s operating using the DNS-01 challenge mode, wherein the Let’s Encrypt issuer creates a challenge, and then cert-manager creates a TXT record in my public acrl.dev hosted zone to prove that yes, in fact, I do own the zone. Cert-manager then stores the certificate as a secret, which gets injected into the Prosody pod. It also automagically handles getting new certificates ahead of the expiry date, which is pretty slick14! I’m a big fan of cert-manger.

Lastly, external-dns handles setting the DNS entry for my Prosody pod in my private acrl.dev hosted zone. The details here are a little subtle: we create a Kubernetes LoadBalancer Service15, which normally would be handled by the cloud controller manager to create a network load balancer resource in AWS. But that stuff’s expensive! We don’t wanna run no network load balancer here. So instead, we use the builtin ServiceLB controller in k3s, which more-or-less just exposes the right ports on the host where the pod is running via iptables rules. The external-dns controller just looks up the IP address of the Prosody Service and writes that as an A record into Route53.

And now we (finally!) have everything we need in order to run our single Kubernetes pod. The Prosody pod requests a 5GB persistent volume to store all of its data, grabs a certificate from cert-manager, uses a ConfigMap to inject all the Prosody configuration, and runs as a single-pod Deployment. And, it all works!

Putting all the pieces together

There’s two bits that I didn’t cover in the above descriptions, which are a) automation and tooling, and b) secrets management. While the details are slightly different for each of my three repos, the high-level bits are the same. For automation and tooling, we use just, which I have increasingly fallen in love with over the last few years. I have justfiles in (almost) every repo, and any time there’s a complicated command that I need to remember, I drop it into the justfile. So if I needed to recreate this entire setup tomorrow from scratch it would be three commands16:

~moria > just apply
~isengard > just apply
~mirkwood > just apply

Not quite the one command I started off with in my initial goals, but honestly it’s pretty good imo. I also have GitHub Actions configured in each of these repos17 to automatically run the above just commands whenever code is changed (or, in the case of moria and isengard, on a periodic schedule to rebuild the AMIs and update the running hosts).

On the secrets management front, each of the four systems (moria, isengard, mirkwood, and k3s) have slightly different methods for dealing with them. Any secret data is stored encrypted at rest, and the user interface is more-or-less the same for each, it’s just the backend that changes. Moria uses Pulumi secrets, which are stored encrypted (using local credentials) in the Pulumi state file in S3. Isengard uses Ansible secrets, which are stored encrypted (using local credentials) in the Git repo itself. Mirkwood uses SOPS to store sensitive data in the Git repo itself, this time encrypted the AWS KMS key that we set up with Moria18. To make each of these three systems easier to work with, I have just targets configured to read or write encrypted data.

So that’s it! After several months of nights-and-weekends work, I now have a usable Kubernetes cluster that is running a single XMPP server in a somewhat reliable fashion, all for about $50/month of AWS bills. Is it cheaper than Slack? No. Is it a better user experience than Slack? Also no. Did I have fun? Lmao, nope, this was annoying as all hell to get configured. Was all the time I spent on this worthwhile, when I could have been using it to make SimKube better? Well… I think you know where this is going.

Anyways, hopefully this 4000-word guide (of sorts) might be useful to someone else down the line who is seeking to embark on a similar endeavour. Next time we’ll be returning to our regularly-scheduled SimKube content!

As always, thanks for reading.

~drmorr