Testing Infrastructure Before It Breaks Production
Most infrastructure teams don't test their Terraform code. They write it, run plan, eyeball the output, and apply. This works until it doesn't — and "doesn't" usually means a production outage caused by a security group rule that looked right in the plan but opened port 22 to the internet.
Terratest and Checkov attack this problem from opposite angles. Terratest validates that deployed infrastructure actually works. Checkov validates that your code follows security and compliance policies without deploying anything. You need both.
Checkov: Policy-as-Code Without the Deploy
Checkov is a static analysis tool that scans Terraform files (and CloudFormation, Kubernetes manifests, Dockerfiles, and more) against a library of built-in policies. It runs in seconds, doesn't need cloud credentials, and catches the 80% of security mistakes that are predictable patterns.
# Run Checkov against your Terraform directory
checkov -d ./infrastructure/ --framework terraform
# Output looks like:
# Passed checks: 47, Failed checks: 3, Skipped checks: 0
#
# Check: CKV_AWS_24: "Ensure no security groups allow ingress from 0.0.0.0/0 to port 22"
# FAILED for resource: aws_security_group.web
# File: /security.tf:15-30The built-in checks cover common mistakes: public S3 buckets, unencrypted databases, overly permissive IAM policies, missing logging configurations. I've found the AWS checks to be the most comprehensive, with GCP and Azure trailing slightly in coverage.
Custom Checkov Policies
The built-in checks won't know about your organization's specific requirements. Maybe all resources need a "cost-center" tag, or all databases must use a specific KMS key, or subnets in production must never have public IP auto-assignment enabled. Custom policies handle these.
Python-based policies offer the most flexibility:
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
from checkov.common.models.enums import CheckCategories, CheckResult
class RequireCostCenterTag(BaseResourceCheck):
def __init__(self):
name = "Ensure all resources have a cost-center tag"
id = "CUSTOM_001"
supported_resources = ["aws_instance", "aws_rds_instance",
"aws_s3_bucket", "aws_lambda_function"]
categories = [CheckCategories.CONVENTION]
super().__init__(name=name, id=id,
categories=categories,
supported_resources=supported_resources)
def scan_resource_conf(self, conf):
tags = conf.get("tags", [{}])
if isinstance(tags, list):
tags = tags[0]
if "cost-center" in tags:
return CheckResult.PASSED
return CheckResult.FAILED
check = RequireCostCenterTag()We maintain about 15 custom policies. They've caught tagging violations, naming convention deviations, and one memorable case where an engineer accidentally configured a production RDS instance with the development instance class (db.t3.micro instead of db.r5.xlarge). That would've been a rough Monday morning.
Terratest: Integration Testing That Deploys Real Resources
Checkov tells you the code follows policies. Terratest tells you the infrastructure actually works. It's a Go library that executes terraform apply, runs assertions against the deployed resources, and then runs terraform destroy to clean up.
package test
import (
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/gruntwork-io/terratest/modules/http-helper"
"github.com/stretchr/testify/assert"
)
func TestVpcDeployment(t *testing.T) {
t.Parallel()
terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
TerraformDir: "../modules/vpc",
Vars: map[string]interface{}{
"cidr_block": "10.99.0.0/16",
"subnet_count": 3,
"environment": "test",
},
})
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
vpcId := terraform.Output(t, terraformOptions, "vpc_id")
assert.NotEmpty(t, vpcId)
subnetIds := terraform.OutputList(t, terraformOptions, "subnet_ids")
assert.Equal(t, 3, len(subnetIds))
}The defer terraform.Destroy is critical. Without it, a test failure leaves orphaned resources running in your AWS account, costing money until someone notices. I've found it helpful to also run a nightly cleanup job that terminates any resources tagged with "terratest" that are older than 4 hours — a safety net for the cases where even the destroy fails.
Testing Network Connectivity
The most valuable Terratest assertions go beyond "does the resource exist" to "does the infrastructure actually function." For a VPC module, that means testing that instances in private subnets can reach the internet through a NAT gateway, that security groups correctly allow and block traffic, and that DNS resolution works.
func TestPrivateSubnetNatConnectivity(t *testing.T) {
// After deploying, SSH into a test instance in the private subnet
// and verify it can reach the internet
publicIp := terraform.Output(t, terraformOptions, "bastion_ip")
// Test outbound connectivity from private subnet via NAT
result := ssh.CheckSshCommand(t, bastionHost,
"curl -s -o /dev/null -w '%{http_code}' https://httpbin.org/get")
assert.Equal(t, "200", result)
}These tests take time — usually 8-15 minutes for a VPC module with connectivity checks. They're not something you run on every commit. We run Checkov on every PR, and Terratest nightly plus on merges to main. The cost works out to about $30/month in AWS resources for a team running tests across three modules.
CI Pipeline Integration
Wiring both tools into your CI pipeline creates a two-layer safety net. Checkov runs first (fast, no cloud credentials needed) and fails the PR if policy violations exist. Terratest runs on merge to main or nightly, catching functional issues that static analysis can't detect.
# .github/workflows/infra-test.yml
jobs:
static-analysis:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Checkov
uses: bridgecrewio/checkov-action@master
with:
directory: ./infrastructure
framework: terraform
output_format: sarif
soft_fail: false
integration-test:
needs: static-analysis
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.21'
- name: Run Terratest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.TEST_AWS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.TEST_AWS_SECRET }}
run: |
cd tests
go test -v -timeout 30m ./...The SARIF output from Checkov integrates with GitHub's security tab, so policy violations show up as code scanning alerts alongside your application security findings. It's a small detail that makes a big difference for security teams who want one dashboard for all policy compliance.
One lesson we learned the hard way: use a dedicated AWS account for Terratest. Don't run infrastructure integration tests in your development account. An accidental VPC CIDR overlap between a test deployment and a developer's experimental cluster caused a routing conflict that took down the development environment for half a day.
Handling Checkov Suppressions
Sometimes a policy violation is intentional. A public S3 bucket hosting a static website is genuinely public by design. A security group allowing inbound traffic from 0.0.0.0/0 on port 443 is correct for a public load balancer. Checkov needs to know when to stop complaining.
Inline suppressions use comments in the Terraform code:
resource "aws_s3_bucket" "website" {
bucket = "company-public-website"
#checkov:skip=CKV_AWS_18: "This bucket hosts a public website - intentionally public"
#checkov:skip=CKV_AWS_19: "Public website content doesn't need encryption"
}
resource "aws_s3_bucket_public_access_block" "website" {
bucket = aws_s3_bucket.website.id
block_public_acls = false
block_public_policy = false
ignore_public_acls = false
restrict_public_buckets = false
}The comment after the skip explains why. We require this explanation in PR reviews — a bare skip without justification gets rejected. It takes 30 seconds to write the reason, and it saves the next engineer from wondering whether the skip was intentional or an oversight.
For organization-wide exceptions, maintain a .checkov.yml file in the repository root. This is cleaner than scattering skip comments across files when a policy genuinely doesn't apply to your use case:
# .checkov.yml
skip-check:
- CKV_AWS_144 # We don't use cross-region replication for non-critical buckets
- CKV_AWS_145 # Default encryption is handled at the org level via SCPTerratest Cost Management
Integration tests that deploy real resources cost real money. We track test costs with two mechanisms: a dedicated AWS cost allocation tag (terratest=true) on all resources created by tests, and a nightly cleanup Lambda that terminates any resource with that tag older than 4 hours.
The biggest cost driver isn't the resources themselves — it's the test duration. A test that takes 15 minutes to run might leave an RDS instance running for 14 of those minutes while it waits for the instance to become available. We've reduced costs by 40% with three changes: use smaller instance types in tests (db.t3.micro instead of db.r5.large), skip deletion protection (which slows down terraform destroy), and parallelize independent test steps.
Monthly test costs for our team of 8 engineers running tests across three modules: approximately $45. That's the cost of one prevented production incident per year in engineer time alone, without counting customer impact.
Combining Checkov and Terratest in Pull Requests
The PR workflow runs both tools but at different stages. Checkov runs on every push — it's fast enough (under 30 seconds) that there's no reason to skip it. The PR gets a check annotation showing which policies passed and which failed, with file-level comments pointing to the specific resource.
Terratest runs only when Checkov passes and the PR is approved by a reviewer. This ordering prevents wasting compute resources on tests for code that hasn't passed static analysis. The test results post to the PR as a status check with a link to the full test log. If tests fail, the log includes the specific assertion that failed and the Terraform plan diff that preceded it.
One thing we learned: run Checkov with the --compact flag in CI. The default output is verbose and creates PR check annotations for every passing policy, which clutters the PR review interface. Compact mode only annotates failures.