Ok, y’all, this is the 4th1 post in our Key Autoscaling Metrics series, and you could say this post is the entire raison d’être, as it were, for this entire blog series. It’s been apparently two years since I’ve ranted about the state of pod disruption in Kubernetes, so I went ahead and shoehorned it in as a Key Autoscaling Metric that you should be tracking in order to ascertain the health of your autoscalers.
“But wait a second,” I hear all the Redditors objecting! “Didn’t you talk about this last week? Wasn’t that the whole point of your error rates blog post? How is pod disruption any different than error rates?”
You have a lot to learn, young padawan. See, the first three metrics in this series are the “headline metrics” – they track your total cluster efficiency, your resilience to surges in demand, and the experience your users have when that demand goes away. The last two metrics (this week’s and next week’s) are secondary metrics that impact multiple aspects of your system performance.
Let me explain: yes, when your pods get disrupted, you might serve errors to your users, especially if other parts of your system aren’t configured correctly. But the point is, even if your error rate is 0% on scaledown, disrupting your pods is a Bad Thing ™️ and your autoscaler should avoid doing it when the benefits don’t justify the action. So, if you want to know if your autoscaler is performing well, you had better be tracking how often it disrupts pods.
Get in, losers, we’re going autoscaling
Let’s first talk about why disruption impacts more than just scale-down: in general, if your pod is disrupted, that means it’s not done with whatever work it was doing. It still has more to do. This is very different than if you, say, scale in your deployment: in general, you scale in because you don’t need the pod anymore. Another way to put it: a disrupted pod needs to be rescheduled somewhere, and a scaled-in pod is just going to go away2.
So, you’ve disrupted a pod and it’s not done with whatever it’s supposed to do. That means Kubernetes needs to reschedule it somewhere else: in the absolute worst case, that might mean that Kubernetes has to scale something else up! So disrupting your pod might be a local improvement, but it might be a global degradation. Hopefully, however, that’s not the case in most circumstances. Hopefully whatever is disrupting your pod is moving things iteratively towards a better state—that’s the whole promise of Kubernetes, after all.
Even then, however, pod disruption is ideally avoided, because even if you’re in a more efficient/reliable/stable state overall, this pod still has work to do that got interrupted. And if your mean time to stop sucking metric isn’t good, it could be several minutes before the disrupted pod is able to resume its work. That’s not great, even if your pod is just a stateless web service; but if it’s some kind of stateful application, like, say, a Spark application, in the best case you have checkpointing enabled and you took a recent snapshot, and in the worst case, it just has to start over from the beginning. Either way, you’re repeating a bunch of work.
Nevertheless, unlike “error rates” where the ideal value for your metric is “four 0”, sometimes when you’re autoscaling, the benefits of disrupting a pod outweigh the costs. So while we want to minimize pod disruptions, your target should never be “zero disruptions”, and in fact what your target “should” be is going to be highly dependent on lots of other factors.
Three 4 shall be the number of interruptions thou shalt count
Hopefully if you’ve gotten to this point, you agree with me that pod disruption is an important thing to measure; but as I’ve already hinted at in this post, actually measuring it is basically impossible. There are basically 4 levels of sophistication here:
Count the number (or rate) of pod interruptions/evictions. This metric is the one that’s easy to track, but it’s not actually the one you want to track, because mixed into your eviction rate is going to be “desired” evictions (e.g., this workload is scaling in and we don’t want to recreate it) as well as disruptions (we’re just shuffling things around for the fun of it). But this metric is at least going to directionally correct, so it’s not an awful thing to track, it’s just not what you should track.
Count the number (or rate) of pod disruptions. I’ve already made the case that not every eviction is a disruption, and what you really care about are the disruptions. Unfortunately, there’s no real clean metric to track this. Essentially what you want is “desired replica count remains stable AND pods were interrupted”; you can definitely see this metric on a graph if you just do a stacked plot of
kube_pod_status_phase3, but it’s not enough to just see it, we need a number that we can track over time—and even then, doing any sort of useful join overkube_pod_status_phaseover any reasonable length of time is probably going to cause your query to time out. What I think might actually work here is to do something like “number of evictions - the delta in your deployment’s replica count”: in other words, take your eviction metric from level 1 and subtract off all the evictions that aren’t disruptions. This again, is probably technically feasible to do but is still probably going to make your dashboard really unhappy over any reasonable period of time.Count the number (or rate) of pod disruptions per pod. Once you’ve figured out how to track disruptions and not just evictions, the next level of sophistication needs to look at disruption rates for a specific workload. This is probably not that important for something like your generic webserver, but it matters a LOT for (again) Spark. If a Spark task ends up failing too many times, eventually the Spark driver is just going to give up and cancel the entire submission. That’s… not great. Unfortunately, Kubernetes isn’t doing us any favors here, because from Kubernetes’s perspective, once a pod is gone, it’s gone. If it happens to get replaced by a different pod, well, that could be doing the same work, or it could be doing entirely different work; Kubernetes doesn’t know and doesn’t care. So in order to actually track this metric, you’re going to need to add some kind of stable identifier to your pods that persists across evictions. You could maybe do this with a StatefulSet, or by adding some additional labels or annotations to your pod metadata. And once you’ve done that, congratulations! There’s another join you’ve added to your PromQL which is going to make Grafana fall over even harder.
There’s not actually a fourth level, I just wanted an excuse to use the number “4”. You know, for no reason4.
Stop disrupting me!
Ok, so you’ve done the impossible, or at least the highly improbable, and are now monitoring your pod disruption rates. How do you actually make that number go down? If your answer is “pod disruption budgets”, get out.
Sadly, as I (and many other folks) have pointed out, pod disruption budgets don’t actually do what you want. They are too coarse-grained, don’t react to changing environments or situations, and (crucially) are ignored by half of the built-in Kubernetes controllers5. I also strongly advise against using something like the do-not-disrupt annotation that Karpenter and Cluster Autoscaler know about: this is, again, an incredibly big hammer that isn’t going to solve the “reduce pod disruption” problem very well. In my experience, setting this annotation (or its twin sister, pod-deletion-cost) generally just causes more problems than it solves.
However, I don’t want to leave this post on too much of a downer, so here are two practical things you can do to reduce your disruption rates6:
Turn on Karpenter’s Balanced consolidation mode. In the weeks since I complained about Karpenter’s consolidation behaviour, I’ve had the chance to play around with Balanced consolidation in a few different settings and7 it actually works pretty well??? So if you’re running Karpenter 1.14 or higher and you want to reduce your pod disruption rate, maybe consider turning Balanced consolidation on. And, if you’re not running Karpenter 1.14 or higher, you should probably upgrade, because there’s a bunch of other bug-fixes and improvements that will make your autoscaling better.
If you’re not running Karpenter (or even if you are running Karpenter and you’d like kube-scheduler and Karpenter to stop fighting each other out of the box with their completely-opposite defaults), maybe switch the Kubernetes scheduler priorities from
LeastAllocatedtoMostAllocated. This will make the Kubernetes scheduler bin-pack pending pods onto nodes that are more full instead of less full. The end result of this change is that your cluster will generally be more densely-packed, and (in principle) your node autoscaler will terminate fewer mostly-unused nodes, thus reducing your pod disruption rate.
I’ll close out the post by mentioning two three 4 other things that you can’t (easily) do now to reduce your pod disruption rate, but you hopefully can do sometime in the future. The first is to implement something like the Eviction Request API KEP that’s being worked on. I’m pretty excited about this, it is basically exactly what I asked somebody to write in my previous post, and hey presto! Someone did it8.
The second thing you can (attempt to) do is use something like Checkpoint-Restore In Userspace (CRIU) to take a complete snapshot of your entire container’s process tree, memory contents, and so forth, and then restore that on another machine when it gets disrupted. This doesn’t make it so that you don’t need to track autoscaler disruption, but it does (in theory) make it significantly faster and easier to recover. It’s also really hard to do well; I know a bunch of companies that are trying, but so far I haven’t seen it really take off.
The third thing you could do is hire a team of 20 or so PhD researchers to redesign the Kubernetes scheduler and various autoscalers into a comprehensive system that is constantly running mixed-integer programming solvers to continually optimize pod placement while minimizing disruptions. But please don’t actually do this one, because that’s what I want to do.
The 4th9 thing you can do today is run SimKube to analyze your disruption rates in practice, and then experiment with your scheduler and autoscaler parameters to see what can bring that metric closer to zero. If you do that, let me know because I’d love to hear how it went for you!
As always, thanks for reading, and tune in next week for the thrilling conclusion to our autoscaling key metrics series.
~drmorr
In case you missed the news, last week the number “4” was declared objectively hilarious by the International Council of Numbers.
Here you can already see one subtle difference between “error rates” and “pod disruption” – both pod disruption and scaling-in can impact your error rates metric, but only one of those has other knock-on consequences as well
There’s a nice example of this in my recent post about Karpenter consolidation.
This footnote is objectively hilarious.
Everybody with me now! :picard-facepalming:
Both of these tips are targeted at cluster-level/node consolidation. If anybody has any similar advice for pod-level consolidation controllers (e.g., HPA/KEDA), I’d love to hear about it, in the comments, that you need to be a premium subscriber to access.
Spoiler warning.
I AM A GOD AND I CAN BEND THE UNIVERSE TO MY WILL!
This 4th piece of advice is simultaneously the funniest as well as the most practical in the whole post.