Why Cloud Bills Surprise Everyone
The most common infrastructure failure mode isn't a crash or an outage. It's the cost anomaly email that arrives on the first of the month showing a 40% increase nobody expected. Capacity planning for cloud-native applications is fundamentally different from traditional infrastructure planning because resources are elastic -- and so are the bills.
I've been on both sides of this. The team that over-provisions "just in case" and wastes $200K annually on idle capacity. The team that right-sizes aggressively and hits a traffic spike that degrades service for two hours during their biggest sale. Neither approach works in isolation. Good capacity planning sits between them.
Demand Forecasting
Start with historical data. If you don't have at least three months of metrics, you're guessing. Prometheus, CloudWatch, or whatever you use for metrics needs to retain enough data to capture weekly and monthly patterns.
# Extract historical resource usage from Prometheus
import requests
from datetime import datetime, timedelta
def get_usage_history(metric, service, days=90):
end = datetime.now()
start = end - timedelta(days=days)
query = f'avg_over_time({metric}{{service="{service}"}}[1h])'
resp = requests.get('http://prometheus:9090/api/v1/query_range', params={
'query': query,
'start': start.isoformat() + 'Z',
'end': end.isoformat() + 'Z',
'step': '3600'
})
data = resp.json()['data']['result']
return [(float(ts), float(val)) for ts, val in data[0]['values']]
cpu_history = get_usage_history('container_cpu_usage_seconds_total:rate5m', 'api')
memory_history = get_usage_history('container_memory_working_set_bytes', 'api')Pattern Recognition
Most services have predictable patterns. Business applications peak during working hours and drop overnight. E-commerce has weekly cycles (higher on weekends) and seasonal spikes (holidays, sales events). Batch processing creates periodic bursts.
Identify these patterns before projecting future capacity. A service growing 10% month-over-month with a 3x daily peak needs different planning than a flat-usage batch job growing at the same rate.
import numpy as np
from scipy import signal
def decompose_usage(timeseries, period_hours=168):
values = np.array([v for _, v in timeseries])
# Separate trend from cyclical pattern
trend = np.convolve(values, np.ones(period_hours)/period_hours, mode='same')
seasonal = values - trend
# Find peak-to-trough ratio
peaks = signal.find_peaks(seasonal, distance=period_hours//2)[0]
troughs = signal.find_peaks(-seasonal, distance=period_hours//2)[0]
if len(peaks) > 0 and len(troughs) > 0:
peak_avg = np.mean(seasonal[peaks])
trough_avg = np.mean(seasonal[troughs])
burst_ratio = peak_avg / max(abs(trough_avg), 1)
else:
burst_ratio = 1.0
# Growth rate
if len(trend) > period_hours:
weekly_growth = (trend[-1] - trend[-period_hours]) / max(trend[-period_hours], 1)
else:
weekly_growth = 0
return {
'trend': trend,
'seasonal': seasonal,
'burst_ratio': burst_ratio,
'weekly_growth': weekly_growth
}Resource Modeling
Translate business growth projections into resource requirements. The key relationship is between throughput (requests per second, messages processed, records written) and resource consumption (CPU, memory, network, storage).
Load Testing for Baselines
You can't model what you haven't measured. Run load tests that establish the relationship between traffic and resource usage for each service.
# k6 load test with resource correlation
import http from 'k6/http';
import { check } from 'k6';
export const options = {
stages: [
{ duration: '5m', target: 100 }, // warm up
{ duration: '10m', target: 500 }, // normal load
{ duration: '10m', target: 1000 }, // peak load
{ duration: '10m', target: 2000 }, // stress test
{ duration: '5m', target: 0 }, // cool down
],
thresholds: {
http_req_duration: ['p(99)<500'],
http_req_failed: ['rate<0.01'],
},
};
export default function() {
const res = http.get('https://api.internal/healthz');
check(res, {
'status 200': (r) => r.status === 200,
'latency ok': (r) => r.timings.duration < 500,
});
}Correlate the load test stages with Prometheus metrics to build a resource model: at 500 RPS, the API service uses 2.1 CPU cores and 3.8GB memory. At 1000 RPS, it uses 4.0 cores and 4.2GB. Memory growth is sublinear (mostly connection pools and caches), CPU growth is roughly linear.
Capacity Planning Models
The Simple Model
For most services, a linear projection with a buffer works:
def simple_capacity_plan(current_usage, growth_rate_monthly,
burst_multiplier, headroom=0.3, months=6):
plan = []
for month in range(1, months + 1):
projected_base = current_usage * (1 + growth_rate_monthly) ** month
projected_peak = projected_base * burst_multiplier
required = projected_peak * (1 + headroom)
plan.append({
'month': month,
'base': round(projected_base, 2),
'peak': round(projected_peak, 2),
'provisioned': round(required, 2)
})
return plan
# API service: 4 CPU cores baseline, 8% monthly growth, 2.5x daily peak
cpu_plan = simple_capacity_plan(4.0, 0.08, 2.5, headroom=0.3, months=6)
# Month 6: base=6.35, peak=15.87, provision=20.63 coresThe 30% headroom accounts for unexpected spikes, garbage collection pauses, kernel overhead, and the fact that your growth estimate is probably wrong. In my experience, 20% headroom is too tight for production services, and 50% is wasteful unless you have extreme SLA requirements.
The Tiered Model
Cloud pricing has tiers and breakpoints. Reserved instances are cheaper than on-demand. Spot instances are cheaper still but can be interrupted. A good capacity plan uses all three:
def tiered_capacity_plan(base_usage, peak_usage,
reserved_discount=0.4, spot_discount=0.7):
# Reserve for minimum sustained load (P10 usage)
reserved = base_usage * 0.7
# On-demand for normal variation (P10 to P90)
on_demand = base_usage - reserved
# Spot/preemptible for peak headroom (P90 to P100 + buffer)
spot = (peak_usage * 1.2) - base_usage
on_demand_rate = 1.0 # normalized
reserved_cost = reserved * (1 - reserved_discount)
on_demand_cost = on_demand * on_demand_rate
spot_cost = spot * (1 - spot_discount)
total = reserved_cost + on_demand_cost + spot_cost
naive_cost = peak_usage * 1.2 * on_demand_rate
savings = (naive_cost - total) / naive_cost
return {
'reserved': round(reserved, 1),
'on_demand': round(on_demand, 1),
'spot': round(spot, 1),
'total_cost_ratio': round(total, 2),
'savings_vs_on_demand': f"{savings:.0%}"
}Autoscaling Configuration
Capacity planning doesn't end with provisioning. Autoscaling fills the gap between your plan and reality.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-server
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
minReplicas: 4 # reserved baseline
maxReplicas: 20 # peak + headroom
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Percent
value: 50
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 25
periodSeconds: 120
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 65The 65% CPU target leaves room for the autoscaler's reaction time. If you target 80%, traffic spikes hit full capacity before new pods are ready. The asymmetric scale-down (slower than scale-up) prevents thrashing during variable traffic.
Review Cadence
Capacity plans rot. Review monthly: compare projected vs. actual usage, update growth rates, adjust reserved instance commitments, and recalibrate the burst multiplier. The plan from three months ago is a historical document, not current guidance.
Set alerts for when actual usage exceeds planned capacity at any tier. When the alert fires, it's not an emergency -- it's the capacity plan telling you it needs an update.