Breaking Things on Purpose
Your team says the system is resilient. Your architecture diagrams show redundancy at every layer. Your runbooks cover common failure scenarios. But nobody has actually tested what happens when an entire availability zone drops, or when the database failover takes 90 seconds instead of the documented 30, or when the message queue silently drops 5% of messages under load.
Chaos engineering fills that gap. It's not about breaking things for fun. It's a disciplined practice of injecting controlled failures to discover weaknesses before production incidents find them for you. Litmus and Gremlin are two tools that make this practical for Kubernetes environments.
Litmus: Open Source, Kubernetes-Native
Litmus runs chaos experiments as Kubernetes custom resources. The ChaosHub provides a library of pre-built experiments, and you define workflows that chain multiple experiments together.
# Install Litmus
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm
helm install litmus litmuschaos/litmus \
--namespace litmus --create-namespace \
--set portal.frontend.service.type=ClusterIPPod-Level Chaos
Start simple. Kill pods and verify that your deployment recovers within the expected timeframe.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: api-pod-kill
namespace: production
spec:
appinfo:
appns: production
applabel: "app=api-server"
appkind: deployment
engineState: active
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: "60"
- name: CHAOS_INTERVAL
value: "10"
- name: FORCE
value: "false"
- name: PODS_AFFECTED_PERC
value: "50"The experiment kills 50% of API server pods every 10 seconds for 60 seconds. Your Kubernetes deployment should replace them. If it doesn't -- or if the replacement takes longer than your readiness probe timeout -- you've found a problem before users did.
Network Chaos
Pod kills are the easy test. Network chaos reveals deeper problems. Latency injection catches timeout misconfiguration, and packet loss exposes retry logic that doesn't actually work.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: network-latency-test
spec:
appinfo:
appns: production
applabel: "app=api-server"
experiments:
- name: pod-network-latency
spec:
components:
env:
- name: NETWORK_LATENCY
value: "300"
- name: JITTER
value: "100"
- name: DESTINATION_IPS
value: "10.0.0.0/8"
- name: TOTAL_CHAOS_DURATION
value: "120"Adding 300ms +/- 100ms of latency to all internal traffic for two minutes. We ran this on a payment service and discovered that the checkout flow timed out at 500ms total, but the service made three sequential calls to internal APIs. Under normal conditions, each call took 50ms. With 300ms added, the chain exceeded the timeout and customers saw checkout failures. The fix was parallelizing those calls -- something nobody would have prioritized without the chaos test proving it was a real risk.
Gremlin: Enterprise Chaos Platform
Gremlin is commercial, polished, and broader than Kubernetes. It runs chaos on VMs, containers, bare metal, and cloud services. The UI makes it accessible to teams that aren't comfortable writing YAML experiment definitions.
Attack Types
Gremlin categorizes attacks into three groups: resource (CPU, memory, disk, IO), network (latency, packet loss, DNS, blackhole), and state (process kill, time travel, shutdown).
# Gremlin CLI: CPU stress test
gremlin attack cpu \
--length 120 \
--percent 90 \
--cores 0 \
--target-tags "service=api,env=staging"
# Gremlin CLI: DNS failure
gremlin attack dns \
--length 60 \
--target-tags "service=api,env=staging" \
--domains "database.internal,cache.internal"The DNS attack is particularly revealing. We used it to test a microservice that cached DNS results. The service handled DNS failure gracefully during the attack, but when DNS recovered, the stale cache entries persisted for hours because the TTL was set to the default of the DNS client library -- not the upstream TTL. Without the chaos test, we'd never have noticed until a real DNS issue caused prolonged stale routing.
Scenarios and Game Days
Gremlin's scenario builder chains attacks into sequences that simulate realistic failure cascades. A "zone failure" scenario might combine network blackhole for a specific IP range, increased latency on remaining connections, and elevated CPU from the surge of requests hitting surviving instances.
# Multi-step scenario via Gremlin API
{
"name": "Zone B Failure Simulation",
"steps": [
{
"attacks": [{
"type": "blackhole",
"args": {"hostnames": ["10.0.2.0/24"], "length": 300},
"target": {"tags": {"zone": "us-east-1b"}}
}],
"delay": 0
},
{
"attacks": [{
"type": "latency",
"args": {"ms": 200, "length": 300},
"target": {"tags": {"zone": "us-east-1a"}}
}],
"delay": 30
},
{
"attacks": [{
"type": "cpu",
"args": {"percent": 80, "length": 300},
"target": {"tags": {"zone": "us-east-1a"}}
}],
"delay": 60
}
]
}Building a Chaos Practice
Most teams fail at chaos engineering not because the tools are wrong but because they skip the discipline around them.
Start with Steady State
Before breaking anything, define what "working" looks like in measurable terms. Error rate below 0.1%. P99 latency under 500ms. Order processing throughput above 100/minute. Without this baseline, you can't objectively determine whether the system survived the experiment.
Blast Radius Control
Run experiments in staging first. When you move to production, start with a single pod, then a percentage, then a full node, then a zone. Never skip levels just because staging looked fine -- production traffic patterns are different.
# Progressive blast radius
experiments:
- phase: 1
target: single_pod
duration: 60s
abort_condition: "error_rate > 1%"
- phase: 2
target: 25%_of_pods
duration: 120s
abort_condition: "error_rate > 0.5%"
- phase: 3
target: entire_node
duration: 180s
abort_condition: "error_rate > 0.1%"Automated Abort
Every experiment needs a kill switch. Both Litmus and Gremlin support automatic abort conditions. If the error rate exceeds your threshold during the experiment, the chaos stops immediately and the system returns to steady state.
Common Findings
After running chaos experiments across 20+ services, these are the most frequent discoveries:
Timeouts configured wrong: services set 30-second timeouts but their callers have 10-second timeouts. The upstream gives up before the downstream even knows there's a problem.
Circuit breakers never tested: teams install Hystrix or resilience4j, configure circuit breakers, and never verify they actually trip under load. We found circuit breakers that were configured but never wired into the actual HTTP client.
Health checks that lie: readiness probes return 200 when the application starts, even before it's connected to its database. Kubernetes routes traffic to pods that aren't actually ready to serve requests.
Retry storms: service A retries failed calls to service B three times. Service B retries calls to service C three times. A single failure in C generates 9 requests. With more layers, this multiplies exponentially and turns a minor issue into a cascading outage.
Don't treat these as theoretical risks. Run the experiments and find out which ones apply to your systems. The answer is usually "more than you'd expect."