Kubernetes Cluster Autoscaler Tuning for Variable Workload Patterns

Q: What is Scale-Up Tuning?

The most impactful setting is --scan-interval, which controls how often the autoscaler checks for unschedulable pods. The default is 10 seconds. That seems fast, but the total time from "pod can't be scheduled" to "new node is ready" includes the scan interval, the cloud provider's API call to create the instance, the instance boot time, and.

Default Autoscaler Settings Will Disappoint You

The Kubernetes Cluster Autoscaler does one thing: it adds nodes when pods can't be scheduled and removes nodes when they're underutilized. Sounds simple. In practice, the default configuration causes either slow scale-ups that leave pods pending for minutes during traffic spikes, or aggressive scale-downs that evict pods unnecessarily during normal workload fluctuation.

I've tuned the Cluster Autoscaler across clusters handling everything from steady-state API traffic to batch ML training jobs with wildly variable resource demands. The right configuration depends entirely on your workload pattern, and the defaults assume a workload pattern that probably isn't yours.

Scale-Up Tuning

The most impactful setting is --scan-interval, which controls how often the autoscaler checks for unschedulable pods. The default is 10 seconds. That seems fast, but the total time from "pod can't be scheduled" to "new node is ready" includes the scan interval, the cloud provider's API call to create the instance, the instance boot time, and kubelet registration. On AWS with EKS, I've measured this end-to-end at 3-5 minutes with default settings.

You can't make the cloud provider spin up instances faster, but you can reduce the autoscaler's reaction time and make smarter decisions about which node group to scale:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
spec:
  template:
    spec:
      containers:
      - name: cluster-autoscaler
        command:
        - ./cluster-autoscaler
        - --scan-interval=10s
        - --scale-down-delay-after-add=10m
        - --scale-down-delay-after-delete=1m
        - --scale-down-unneeded-time=5m
        - --max-graceful-termination-sec=600
        - --balance-similar-node-groups=true
        - --expander=least-waste
        - --skip-nodes-with-local-storage=false
        - --skip-nodes-with-system-pods=false

Expander Strategy Selection

The --expander flag determines which node group the autoscaler scales when multiple groups could satisfy the pending pods. The options matter more than most people realize.

random (default) — picks a random eligible node group. Simple but can lead to unbalanced groups and suboptimal instance type selection.

least-waste — picks the node group that would have the least idle resources after the pending pods are scheduled. This is my default recommendation for mixed workloads. If a pending pod needs 2 CPU and you have a node group with 4-CPU instances and one with 16-CPU instances, it'll choose the 4-CPU group to minimize waste.

priority — lets you define an explicit priority order for node groups via a ConfigMap. Useful when you want to prefer spot instances over on-demand, or prefer a specific instance family:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    50:
      - .*spot.*
    30:
      - .*ondemand-compute.*
    10:
      - .*ondemand-general.*

This tells the autoscaler to try spot instance groups first, then on-demand compute-optimized, then general purpose. The priority expander has saved us roughly 40% on node costs compared to the random expander, because it consistently prefers cheaper instance types.

Scale-Down Tuning

Scale-down is where most teams get burned. The autoscaler marks a node for scale-down when its resource utilization (combined requests of all pods divided by the node's allocatable capacity) falls below a threshold. The default threshold is 50%.

That 50% default is aggressive. A node with two pods requesting 200m CPU each on a 4-CPU node shows 10% utilization and gets marked for removal. The autoscaler will try to move those pods to other nodes. If it succeeds, the node gets terminated. But if the pods have PodDisruptionBudgets, local storage, or are part of a StatefulSet, the eviction might fail or take longer than expected.

I set --scale-down-utilization-threshold=0.65 on most clusters. This means nodes don't get removed until they're less than 35% utilized, which reduces churn from normal workload fluctuation. Combined with --scale-down-unneeded-time=10m (node must be underutilized for 10 minutes before removal), this prevents the "add a node, remove a node, add a node" oscillation that wastes money on instance launch charges.

Protecting Workloads During Scale-Down

PodDisruptionBudgets (PDBs) are your protection against aggressive scale-down. Every production deployment should have one:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: api-server

This guarantees that at most one api-server pod can be disrupted at any time. If the autoscaler tries to drain a node, it respects the PDB and won't evict a pod if doing so would violate the budget. Without PDBs, the autoscaler can drain all your pods off a node simultaneously during scale-down.

For workloads that should never be evicted by the autoscaler, annotate the pod:

metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Node Group Architecture for Variable Workloads

A single node group is almost never sufficient for production. I typically set up three to four groups per cluster:

A general-purpose group (m5.xlarge or equivalent) handles steady-state workloads — API servers, web frontends, background workers. This group has a minimum size of 3 for high availability and scales up based on demand. It uses on-demand instances because these workloads can't tolerate spot interruptions.

A compute-optimized spot group (c5.2xlarge or equivalent) handles batch processing, CI/CD runners, and any workload that can tolerate interruptions. Setting the minimum to 0 means you're not paying for these nodes when there's no batch work.

A memory-optimized group (r5.xlarge or equivalent) handles caches, in-memory databases, and analytics workloads. These typically run on-demand because losing a cache node to a spot interruption causes a thundering herd of cold cache misses.

Use node affinity and taints to direct workloads to the appropriate group:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-group
            operator: In
            values:
            - compute-spot
  tolerations:
  - key: "spot"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

Overprovisioning for Faster Scale-Up

Even with optimized settings, adding a new node takes minutes. For workloads that need to scale in seconds, use the pause pod overprovisioning trick. Create a low-priority deployment that occupies space on existing nodes:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -1
globalDefault: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
spec:
  replicas: 3
  selector:
    matchLabels:
      app: overprovisioning
  template:
    metadata:
      labels:
        app: overprovisioning
    spec:
      priorityClassName: overprovisioning
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          requests:
            cpu: "1"
            memory: 2Gi

When a real workload needs to scale, its pods (with normal priority) preempt the pause pods. The pause pods become unschedulable, triggering the autoscaler to add a new node. But the real workload is already running on the space the pause pods freed up — it didn't have to wait for the new node. The cost is maintaining three extra CPUs and 6Gi of memory as a buffer, which is typically cheaper than the business impact of multi-minute scale-up delays.

Handling Spot Instance Interruptions

If you're using spot instances with the Cluster Autoscaler (and you should be for cost optimization), spot interruptions add complexity to scale-down behavior. AWS gives you a 2-minute warning before terminating a spot instance. The Cluster Autoscaler doesn't handle spot interruptions - it handles scale-down of underutilized nodes. Spot interruptions are handled by a separate component.

The AWS Node Termination Handler (NTH) watches for spot interruption notices and cordons/drains the node before the instance is terminated. Install it alongside the Cluster Autoscaler. NTH cordons the node (preventing new pods from being scheduled) and drains existing pods (respecting PDBs) within the 2-minute window. The Cluster Autoscaler then sees the drained node and adds a replacement if there are unschedulable pods.

Monitoring Autoscaler Decisions

The Cluster Autoscaler exposes Prometheus metrics that are essential for understanding its behavior. Track cluster_autoscaler_unschedulable_pods_count for pods triggering scale-up, cluster_autoscaler_function_duration_seconds for timing, and cluster_autoscaler_scaled_up_nodes_total for event counts.

I build a Grafana dashboard around these metrics. The most useful panel shows the correlation between unschedulable pods and scale-up events - if there's a persistent gap where pods are unschedulable but no scale-up occurs, it usually means the autoscaler can't find a node group with sufficient capacity, or it's hitting the maximum size limit for all node groups.

Set alerts on unschedulable pods sustained for more than 5 minutes. That catches situations where the autoscaler is stuck - maybe the cloud provider is out of capacity in your region, maybe your maximum node group size is too low, or maybe there's a taint/toleration mismatch preventing pods from being scheduled on available nodes.

Also watch the autoscaler's own logs. When it decides not to scale down a node, it logs the reason. Common reasons include pod with local storage, pod with PDB that would be violated, and node utilization above threshold. These logs are invaluable when debugging why nodes aren't being reclaimed.

Karpenter as an Alternative

AWS Karpenter takes a different approach to autoscaling. Instead of managing predefined node groups with fixed instance types, Karpenter provisions individual nodes based on the exact requirements of pending pods. If you have a pod requesting 4 CPU and 16Gi memory, Karpenter finds the cheapest instance type that satisfies those requirements.

In my testing, Karpenter reduced node provisioning time from 3-5 minutes (Cluster Autoscaler) to 60-90 seconds. It also reduced overall node costs by 20-30% because it picks the most cost-effective instance type for each workload rather than using a predefined set. The trade-off is that Karpenter is AWS-specific (though there are efforts to support other clouds), and it's a newer project with a smaller operational knowledge base.