Incident Response Automation with PagerDuty and Runbook Integration

The 3 AM Problem Nobody Wants

Your phone buzzes at 3 AM. Database connection pool exhausted. You open your laptop, check dashboards, find the culprit, restart the affected service, verify recovery, update the incident channel, and go back to sleep. Forty-five minutes gone. The exact same scenario happened two weeks ago with the exact same fix.

That's the automation gap. Not the novel outages -- those need human judgment. It's the repetitive ones where the diagnosis and fix follow the same steps every time. PagerDuty's runbook automation closes that gap by executing predefined response procedures when specific alert conditions trigger.

Runbook Automation Architecture

PagerDuty's automation framework connects to your infrastructure through runners -- lightweight agents installed in your environment that execute automation actions without exposing internal systems to the internet.

# Runner installation
curl -sL https://runner.pagerduty.com/install.sh | \
  RUNNER_TOKEN=pdrt_xxxx \
  RUNNER_NAME=prod-us-east \
  bash

# Runner config at /etc/pd-runner/config.yml
runner:
  name: prod-us-east
  token: pdrt_xxxx
  labels:
    environment: production
    region: us-east-1
  allowed_actions:
    - restart_service
    - scale_deployment
    - drain_node
    - flush_cache

Diagnostic Runbooks

Start with diagnostics, not remediation. The simplest automation gathers context before a human even opens their laptop. When an alert fires, the runbook pulls relevant data and attaches it to the incident.

name: database_connection_diagnostic
trigger:
  service: api-production
  alert_contains: "connection pool exhausted"
steps:
  - name: check_active_connections
    action: run_script
    script: |
      psql -h $DB_HOST -U monitor -c \
        "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
  - name: check_slow_queries
    action: run_script
    script: |
      psql -h $DB_HOST -U monitor -c \
        "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
         FROM pg_stat_activity
         WHERE state != 'idle'
         ORDER BY duration DESC LIMIT 10;"
  - name: check_service_replicas
    action: run_script
    script: |
      kubectl get pods -l app=api -o wide
      kubectl top pods -l app=api
  - name: attach_to_incident
    action: add_note
    content: "Auto-diagnostic results attached"

When the on-call engineer picks up the incident, they already have active connection counts, slow queries, and pod status. The 10-minute diagnostic phase is eliminated.

Graduated Automation

Don't jump straight to fully autonomous remediation. Build trust gradually through three levels.

Level 1: Diagnostic Only

Automation gathers information, humans decide and act. Run this for 2-4 weeks to validate that diagnostics are accurate and useful. Track how often the gathered data actually helps resolve the incident faster.

Level 2: Suggest and Confirm

Automation diagnoses, proposes a specific action, and waits for human approval. The on-call engineer gets a notification: "Database connections exhausted. 47 idle connections from api-worker-3. Recommended: restart api-worker-3. Approve?"

name: connection_pool_remediation
trigger:
  service: api-production
  alert_contains: "connection pool"
steps:
  - name: diagnose
    action: run_script
    script: |
      psql -h $DB_HOST -U monitor -c \
        "SELECT application_name, count(*)
         FROM pg_stat_activity WHERE state='idle'
         GROUP BY application_name ORDER BY count DESC LIMIT 5;"
  - name: identify_culprit
    action: parse_output
    pattern: "^(\\S+)\\s+(\\d+)"
    store_as: idle_connections
  - name: request_approval
    action: create_status_update
    message: |
      Highest idle connections: {{ idle_connections[0] }}
      Recommended action: restart {{ idle_connections[0].name }}
  - name: wait_for_approval
    action: await_responder_action
    timeout: 300
  - name: restart_service
    action: run_script
    script: |
      kubectl rollout restart deployment/{{ idle_connections[0].name }}
    requires_approval: true

Level 3: Autonomous with Guardrails

Automation diagnoses and acts within strict boundaries. Only graduate to this level after Level 2 has been running successfully for at least a month with zero false positive remediations.

The guardrails are essential. Without them, autonomous remediation becomes autonomous destruction.

guardrails:
  max_actions_per_hour: 3
  max_actions_per_incident: 1
  cooldown_after_action: 600
  allowed_hours:
    start: "06:00"
    end: "22:00"
    timezone: "America/New_York"
  blocked_during:
    - deploy_in_progress
    - maintenance_window
  escalate_if:
    - action_failed
    - repeated_trigger_within: 1800
    - blast_radius: high

Escalation Policy Integration

Automation should integrate with, not bypass, your escalation policies. When autonomous remediation succeeds, still notify the on-call engineer with what happened and why. When it fails, escalate with full diagnostic context attached.

on_success:
  - add_incident_note: "Auto-remediated: restarted {{ service }}"
  - set_severity: info
  - schedule_postmortem: false
  - notify_oncall: low_urgency

on_failure:
  - add_incident_note: "Auto-remediation failed: {{ error }}"
  - escalate: true
  - set_severity: high
  - attach_diagnostics: true

Measuring Automation Effectiveness

Track four metrics to know if your automation investment is paying off.

Mean time to diagnose (MTTD): how long from alert to understanding the problem. Diagnostic runbooks should cut this by 60-80%.

Mean time to remediate (MTTR): how long from diagnosis to resolution. Automated remediation targets a 90% reduction for known failure modes.

Automation success rate: what percentage of automated actions actually fix the problem without human intervention. Below 85%, the automation creates more confusion than it resolves.

False positive remediation rate: how often automation "fixes" something that wasn't broken or applies the wrong fix. Above 5%, dial back to Level 2 and retrain the trigger conditions.

Common Automation Candidates

After building automation for dozens of services, these patterns cover roughly 70% of automatable incidents:

Connection pool exhaustion: diagnose idle connections, restart the offending service. Simple, high-frequency, low-risk.

Disk space alerts: identify large files or old logs, rotate or compress. Check if a known batch job is running before acting.

Certificate expiration: trigger renewal workflow, validate new certificate, reload the service. No human judgment needed for standard domain certificates.

Pod crash loops: gather logs from the last three restarts, check recent deployments, automatically rollback if deployed within the last hour.

Rate limiting triggers: scale up the affected service, verify throughput recovery, scale back down after traffic normalizes. Use HPA metrics to validate.

What Not to Automate

Data corruption incidents. Security breaches. Cascading failures affecting multiple services. Novel failure modes you haven't seen before. Anything where the wrong automated action makes the situation worse than waiting for a human. The goal is eliminating toil, not replacing judgment. If an incident requires understanding business context or making a tradeoff between availability and data integrity, keep a human in the loop.