Incident Response Automation

The 3 AM Problem Nobody Wants

I've found that Your phone buzzes at 3 AM. Database connection pool exhausted. You open your laptop, check dashboards, find the culprit, restart the affected service, verify recovery, update the incident channel, and go back to sleep. Forty-five minutes gone. The exact same scenario happened two weeks ago with the exact same fix.

That's the automation gap. Not the novel outages -- those need human judgment. It's the repetitive ones where the diagnosis and fix follow the same steps every time. PagerDuty's runbook automation closes that gap by executing predefined response procedures when specific alert conditions trigger.

Runbook Automation Architecture

PagerDuty's automation framework connects to your infrastructure through runners -- lightweight agents installed in your environment that execute automation actions without exposing internal systems to the internet.

# Runner installation
curl -sL https://runner.pagerduty.com/install.sh | \
 RUNNER_TOKEN=pdrt_xxxx \
 RUNNER_NAME=prod-us-east \
 bash

# Runner config at /etc/pd-runner/config.yml
runner:
 name: prod-us-east
 token: pdrt_xxxx
 labels:
 environment: production
 region: us-east-1
 allowed_actions:
 - restart_service
 - scale_deployment
 - drain_node
 - flush_cache

Diagnostic Runbooks

Start with diagnostics, not remediation. The simplest automation gathers context before a human even opens their laptop. When an alert fires, the runbook pulls relevant data and attaches it to the incident.

name: database_connection_diagnostic
trigger:
 service: api-production
 alert_contains: "connection pool exhausted"
steps:
 - name: check_active_connections
 action: run_script
 script: |
 psql -h $DB_HOST -U monitor -c \
 "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
 - name: check_slow_queries
 action: run_script
 script: |
 psql -h $DB_HOST -U monitor -c \
 "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
 FROM pg_stat_activity
 WHERE state != 'idle'
 ORDER BY duration DESC LIMIT 10;"
 - name: check_service_replicas
 action: run_script
 script: |
 kubectl get pods -l app=api -o wide
 kubectl top pods -l app=api
 - name: attach_to_incident
 action: add_note
 content: "Auto-diagnostic results attached"

When the on-call engineer picks up the incident, they already have active connection counts, slow queries, and pod status. The 10-minute diagnostic phase is eliminated.

Graduated Automation

Don't jump straight to fully autonomous remediation. Build trust gradually through three levels.

We covered a related topic in Multi-Cloud Networking: Interconnecting AWS, GCP, and Azure .

Level 1: Diagnostic Only

Automation gathers information, humans decide and act. Run this for 2-4 weeks to validate that diagnostics are accurate and useful. Track how often the gathered data actually helps resolve the incident faster.

Level 2: Suggest and Confirm

Automation diagnoses, proposes a specific action, and waits for human approval. The on-call engineer gets a notification: "Database connections exhausted. 47 idle connections from api-worker-3. Recommended: restart api-worker-3. Approve?"

name: connection_pool_remediation
trigger:
 service: api-production
 alert_contains: "connection pool"
steps:
 - name: diagnose
 action: run_script
 script: |
 psql -h $DB_HOST -U monitor -c \
 "SELECT application_name, count(*)
 FROM pg_stat_activity WHERE state='idle'
 GROUP BY application_name ORDER BY count DESC LIMIT 5;"
 - name: identify_culprit
 action: parse_output
 pattern: "^(\\S+)\\s+(\\d+)"
 store_as: idle_connections
 - name: request_approval
 action: create_status_update
 message: |
 Highest idle connections: {{ idle_connections[0] }}
 Recommended action: restart {{ idle_connections[0].name }}
 - name: wait_for_approval
 action: await_responder_action
 timeout: 300
 - name: restart_service
 action: run_script
 script: |
 kubectl rollout restart deployment/{{ idle_connections[0].name }}
 requires_approval: true

Level 3: Autonomous with Guardrails

Automation diagnoses and acts within strict boundaries. Only graduate to this level after Level 2 has been running successfully for at least a month with zero false positive remediations.

The guardrails are essential. Without them, autonomous remediation becomes autonomous destruction.

guardrails:
 max_actions_per_hour: 3
 max_actions_per_incident: 1
 cooldown_after_action: 600
 allowed_hours:
 start: "06:00"
 end: "22:00"
 timezone: "America/New_York"
 blocked_during:
 - deploy_in_progress
 - maintenance_window
 escalate_if:
 - action_failed
 - repeated_trigger_within: 1800
 - blast_radius: high

Escalation Policy Integration

Automation should integrate with, not bypass, your escalation policies. When autonomous remediation succeeds, still notify the on-call engineer with what happened and why. When it fails, escalate with full diagnostic context attached.

on_success:
 - add_incident_note: "Auto-remediated: restarted {{ service }}"
 - set_severity: info
 - schedule_postmortem: false
 - notify_oncall: low_urgency

on_failure:
 - add_incident_note: "Auto-remediation failed: {{ error }}"
 - escalate: true
 - set_severity: high
 - attach_diagnostics: true

Measuring Automation Effectiveness

Track four metrics to know if your automation investment is paying off.

Mean time to diagnose (MTTD): how long from alert to understanding the problem. Diagnostic runbooks should cut this by 60-80%.

Mean time to remediate (MTTR): how long from diagnosis to resolution. Automated remediation targets a 90% reduction for known failure modes.

Automation success rate: what percentage of automated actions actually fix the problem without human intervention. Below 85%, the automation creates more confusion than it resolves.

False positive remediation rate: how often automation "fixes" something that wasn't broken or applies the wrong fix. Above 5%, dial back to Level 2 and retrain the trigger conditions.

For a related perspective, see Log Aggregation Architecture: Loki vs Elasticsearch for Cost.

Common Automation Candidates

After building automation for dozens of services, these patterns cover roughly 70% of automatable incidents:

Connection pool exhaustion: diagnose idle connections, restart the offending service. Simple, high-frequency, low-risk.

Disk space alerts: identify large files or old logs, rotate or compress. Check if a known batch job is running before acting.

Certificate expiration: trigger renewal workflow, validate new certificate, reload the service. No human judgment needed for standard domain certificates.

Pod crash loops: gather logs from the last three restarts, check recent deployments, automatically rollback if deployed within the last hour.

Rate limiting triggers: scale up the affected service, verify throughput recovery, scale back down after traffic normalizes. Use HPA metrics to validate.

What Not to Automate

Data corruption incidents. Security breaches. Cascading failures affecting multiple services. Novel failure modes you haven't seen before. Anything where the wrong automated action makes the situation worse than waiting for a human. The goal is eliminating toil, not replacing judgment. If an incident requires understanding business context or making a tradeoff between availability and data integrity, keep a human in the loop.

Incident Response Automation with PagerDuty and Runbook Integration