Infrastructure Drift Detection and Remediation Strategies

Drift Is a Symptom, Not the Disease

Infrastructure drift — when the actual state of your cloud resources doesn't match your Infrastructure as Code — isn't a technical problem. It's an organizational one. Every piece of drift represents someone who bypassed the IaC workflow. Maybe they were in a hurry. Maybe the IaC pipeline was slow. Maybe the change was "just a quick console tweak" that was supposed to be temporary and then wasn't.

Detecting drift is the easy part. The hard part is building systems and processes that prevent it without slowing down the teams that need to make urgent changes.

Detection Methods Compared

Terraform's built-in drift detection runs during terraform plan. It compares the state file against the actual cloud resources and reports differences. This works but has a significant limitation: it only checks resources Terraform knows about. If someone creates a resource manually that isn't in your Terraform code, terraform plan won't see it.

AWS Config Rules detect drift at the AWS API level. They compare resource configurations against rules you define and flag non-compliant resources regardless of how they were created. This catches both IaC drift (a manually modified security group) and shadow infrastructure (a manually created EC2 instance that no Terraform code manages).

CloudFormation has its own drift detection that works similarly to Terraform's — it checks resources managed by CloudFormation stacks against their template definitions. If you're using StackSets, you can run drift detection across all member accounts from the admin account.

Third-party tools like Driftctl (now part of Snyk) scan your cloud account and compare everything against your Terraform state, identifying both modified resources and unmanaged resources. I've found this the most comprehensive approach for organizations that want a complete picture:

# Scan for drift against current Terraform state
driftctl scan --from tfstate://path/to/terraform.tfstate

# Output shows:
# Found 145 resources (132 managed, 8 unmanaged, 5 changed)
# Unmanaged:
#   aws_security_group: sg-0abc123 (created manually in console)
#   aws_iam_role: emergency-access-role (created by break-glass script)
# Changed:
#   aws_instance.api_server: instance_type changed m5.large -> m5.xlarge

Automated Detection Pipeline

Running drift detection manually is like running security scans manually — it happens less often than it should, and the results get stale immediately. We automate drift detection with a pipeline that runs every 6 hours:

# drift-detection.yaml (simplified)
name: Drift Detection
on:
  schedule:
    - cron: '0 */6 * * *'

jobs:
  detect:
    runs-on: self-hosted
    strategy:
      matrix:
        workspace: [networking-prod, compute-prod, data-prod]
    steps:
      - uses: actions/checkout@v4
      - name: Terraform Plan (drift check)
        run: |
          cd infrastructure/${{ matrix.workspace }}
          terraform init -backend-config=prod.hcl
          terraform plan -detailed-exitcode -out=plan.tfplan 2>&1 | tee plan.txt
          EXIT_CODE=$?
          if [ $EXIT_CODE -eq 2 ]; then
            echo "DRIFT_DETECTED=true" >> $GITHUB_ENV
          fi
      - name: Notify on drift
        if: env.DRIFT_DETECTED == 'true'
        run: |
          # Post to Slack with the diff summary
          python3 scripts/notify-drift.py             --workspace ${{ matrix.workspace }}             --plan-file plan.txt

The -detailed-exitcode flag is key. Exit code 0 means no changes. Exit code 2 means changes detected (drift). Exit code 1 means an error. Without this flag, all three cases return 0, and you can't distinguish drift from "everything's fine."

Drift Severity Classification

Not all drift is equally urgent. A changed tag is low severity — it doesn't affect functionality. A modified security group rule is critical — it might represent an unauthorized access change. Our notification pipeline classifies drift by resource type and attribute:

Critical: security groups, IAM policies, KMS keys, network ACLs, route tables. These get an immediate PagerDuty alert.

High: instance types, database parameters, load balancer configurations. These get a Slack notification to the owning team.

Low: tags, descriptions, cosmetic changes. These get logged and included in a weekly report.

The classification prevents alert fatigue. When everything triggers the same alarm, engineers stop investigating drift reports. When only critical security changes page the on-call engineer, those pages get taken seriously.

Remediation Strategies

Once you've detected drift, you have three options: revert to IaC, adopt the change into IaC, or ignore it. Each has its place.

Reverting means running terraform apply to force the actual state back to what the code declares. This is the right call for unauthorized changes and accidental modifications. But be careful — if the drift was a hotfix (someone scaled up an instance during an incident), reverting will undo the fix. Always check with the relevant team before reverting critical-severity drift.

Adopting means updating your Terraform code to match the actual state. This is the right call when someone made a legitimate change through the console (scaling up during an incident, for example) and you need to persist it. Update the code, run plan to verify the diff is zero, and merge.

Ignoring means you've decided the drift is acceptable and documenting why. We use Terraform's lifecycle { ignore_changes } block for attributes that are intentionally managed outside of Terraform — like auto-scaling group desired counts that a scaling policy adjusts dynamically:

resource "aws_autoscaling_group" "web" {
  # ... other config ...
  desired_capacity = 3  # Base capacity; actual managed by scaling policy

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

Prevention Is Better Than Detection

SCPs (Service Control Policies) can prevent console changes to resources managed by IaC. This is the nuclear option — it means engineers literally cannot modify tagged IaC-managed resources through the console. It works, but it requires an emergency break-glass procedure for incidents where the IaC pipeline is down and engineers need to make immediate changes.

A softer approach: read-only console access for production accounts, with write access gated behind an approval process that includes "have you updated the Terraform code first?" as a checkbox. This doesn't prevent drift, but it creates a social norm that makes manual changes the exception rather than the path of least resistance.

The teams I've worked with that have the least drift share two characteristics: their IaC pipeline is fast (plan in under 2 minutes, apply in under 5), and their PR review process for infrastructure changes is responsive (reviewed within 30 minutes during business hours). When the "right way" is fast and easy, people use it. When it takes 45 minutes to get a security group change approved and applied through Terraform, people open the console.

Drift Detection at Scale: Multi-Account Patterns

Running drift detection across 50+ AWS accounts introduces challenges that single-account setups don't face. Each account potentially has its own Terraform state, its own backend configuration, and its own set of engineers making changes. The detection pipeline needs to authenticate to each account, locate the right state files, and aggregate results into a single report.

We solved this with a central drift detection service that runs as a scheduled ECS task. It iterates through all accounts in our AWS Organization, assumes a read-only role in each, runs terraform plan against the stored state, and collects the results. The whole sweep completes in about 20 minutes for 45 accounts with an average of 150 resources each.

# drift_scanner.py (simplified)
import boto3
import subprocess
import json

org_client = boto3.client('organizations')
sts_client = boto3.client('sts')

accounts = org_client.list_accounts()['Accounts']
drift_report = []

for account in accounts:
    # Assume role in target account
    role_arn = f"arn:aws:iam::{account['Id']}:role/TerraformDriftReader"
    creds = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName="drift-scan"
    )['Credentials']

    # Set credentials for terraform
    env = {
        'AWS_ACCESS_KEY_ID': creds['AccessKeyId'],
        'AWS_SECRET_ACCESS_KEY': creds['SecretAccessKey'],
        'AWS_SESSION_TOKEN': creds['SessionToken'],
    }

    # Run terraform plan with -detailed-exitcode
    result = subprocess.run(
        ['terraform', 'plan', '-detailed-exitcode', '-no-color'],
        capture_output=True, text=True, env={**env, 'PATH': '/usr/bin'},
        cwd=f'/workspace/{account["Id"]}'
    )

    if result.returncode == 2:
        drift_report.append({
            'account': account['Id'],
            'name': account['Name'],
            'plan_output': result.stdout[-2000:]  # Last 2000 chars
        })

The read-only role is important. The drift detection process should never have permission to modify resources. A bug in the detection script should produce a bad report, not accidentally terraform apply against a production account.

Drift Metrics and Trending

We publish drift metrics to CloudWatch: total resources monitored, resources with drift, accounts with drift, and drift resolution time (how long between detection and fix). These metrics feed into a dashboard that engineering leadership reviews monthly.

The trending data reveals patterns. Our drift rate dropped from 8% to under 2% over six months — not because we got better at detecting drift, but because we made the IaC pipeline faster and reduced the friction of making changes through code. The correlation was clear: when plan-to-apply time exceeded 10 minutes, drift increased. When we cut it to under 5 minutes, drift decreased. Engineers take the path of least resistance. Make the right path the easy path.