Secret Management Architecture: Vault, AWS Secrets Manager, and SOPS

Secrets Sprawl Is the Default State

Every engineering org I've audited has secrets in places they shouldn't be. Environment variables hardcoded in CI configs. API keys in Kubernetes ConfigMaps instead of Secrets. Database passwords in Terraform state files. The question isn't whether you have secrets sprawl — it's how bad it is and which tool will help you contain it.

The three most common options for centralized secret management are HashiCorp Vault, AWS Secrets Manager, and Mozilla SOPS. They serve different use cases and can complement each other, but most teams should pick one as the primary source of truth.

HashiCorp Vault: The Full-Featured Option

Vault does everything. Static secrets, dynamic secrets (generate short-lived database credentials on demand), encryption as a service, PKI certificate management, SSH key signing. It's the Swiss Army knife of secret management, and like most Swiss Army knives, most people use two of the twelve blades.

Running Vault in production is an operational commitment. It requires an HA backend (Consul, Raft, or a cloud storage backend), unsealing procedures (or auto-unseal with a cloud KMS), audit log management, and regular backup verification. The managed offering (HCP Vault) reduces this burden but adds cost and constrains your configuration options.

# Vault Kubernetes auth configuration
vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create a policy for the payments service
vault policy write payments-service - <<EOF
path "secret/data/payments/*" {
  capabilities = ["read"]
}
path "database/creds/payments-readonly" {
  capabilities = ["read"]
}
EOF

# Bind the policy to a Kubernetes service account
vault write auth/kubernetes/role/payments-service \
  bound_service_account_names=payments-service \
  bound_service_account_namespaces=production \
  policies=payments-service \
  ttl=1h

Dynamic Secrets

Dynamic secrets are Vault's killer feature. Instead of storing a database password that every developer and every service knows, Vault generates unique, short-lived credentials per service instance. If a credential leaks, it expires automatically. If you need to revoke a specific service's access, you revoke its lease without affecting other services.

# Configure the database secrets engine
vault secrets enable database

vault write database/config/payments-db \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/payments" \
  allowed_roles="payments-readonly,payments-readwrite" \
  username="vault_admin" \
  password="initial-password"

vault write database/roles/payments-readonly \
  db_name=payments-db \
  creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";" \
  default_ttl="1h" \
  max_ttl="24h"

The tradeoff: dynamic secrets add complexity to your application's startup and connection handling. The application needs a Vault client library, token renewal logic, and graceful handling of credential rotation. For services that maintain persistent database connection pools, rotating credentials means draining the old pool and establishing new connections — which can cause brief latency spikes.

AWS Secrets Manager: The Managed Path

If your infrastructure is primarily AWS and you don't need Vault's advanced features, Secrets Manager is simpler to operate. It's a managed service — no servers to run, no unsealing, no HA configuration. You store secrets, you retrieve them, AWS handles the rest.

Secrets Manager's automatic rotation feature works well for RDS database passwords and a few other integrated services. For everything else, you write a Lambda function that handles the rotation logic. This is where the simplicity starts to erode — a Secrets Manager deployment with 20 custom rotation Lambdas is its own operational burden.

# Terraform: Create a secret with automatic rotation
resource "aws_secretsmanager_secret" "db_password" {
  name        = "prod/payments-service/db-password"
  description = "Payments service database credentials"
  
  tags = {
    Service     = "payments"
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

resource "aws_secretsmanager_secret_rotation" "db_rotation" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.secret_rotation.arn
  
  rotation_rules {
    automatically_after_days = 30
  }
}

SOPS: Encrypted Files in Git

Mozilla SOPS takes a completely different approach. Instead of a centralized secret store, SOPS encrypts individual values within configuration files (YAML, JSON, ENV). The encrypted files live in your Git repository alongside your code, and decryption happens at deploy time using AWS KMS, GCP KMS, Azure Key Vault, or PGP keys.

# Encrypted SOPS file (note: only values are encrypted)
database:
  host: ENC[AES256_GCM,data:kN8sFQ==,iv:...,tag:...,type:str]
  port: 5432     # unencrypted - not a secret
  name: payments # unencrypted
  password: ENC[AES256_GCM,data:p8HL2bM=,iv:...,tag:...,type:str]
api_keys:
  stripe: ENC[AES256_GCM,data:a7Bx9Z2n...,iv:...,tag:...,type:str]
sops:
  kms:
    - arn: arn:aws:kms:us-east-1:123456789:key/abcd-1234
  encrypted_regex: ^(password|secret|key|token)$

SOPS works well for small teams and simple deployments. It doesn't require running any infrastructure, and secrets versioning comes free with Git. The downsides: no dynamic secrets, no centralized access control (access is controlled by KMS key policies, which are coarse-grained), and no audit trail beyond Git history.

Choosing Based on Your Actual Needs

Small team (under 20 engineers), primarily one cloud: AWS Secrets Manager or SOPS. You don't need the operational overhead of Vault.

Medium team (20-100 engineers), multiple services: AWS Secrets Manager or Vault. If you need dynamic database credentials or cross-cloud secret management, Vault. If you're AWS-only and static secrets are fine, Secrets Manager.

Large team (100+ engineers), complex compliance requirements: Vault. The audit logging, policy engine, and dynamic secrets capabilities justify the operational investment. Run HCP Vault if you can afford it — the self-hosted operational burden is real.

Many organizations use SOPS for application configuration (non-sensitive settings and a few secrets) and Vault or Secrets Manager for the actual sensitive credentials. This hybrid approach works well — SOPS handles the 80% case (config files with a few secrets), and the centralized store handles the 20% that needs rotation, auditing, and dynamic generation.

Kubernetes Secret Injection Patterns

Regardless of which backend you choose, the last mile — getting secrets into running containers — deserves attention. Kubernetes Secrets are the native mechanism, but they're base64-encoded (not encrypted) and stored in etcd. Encrypting etcd at rest helps, but anyone with RBAC access to read Secrets in a namespace can decode them.

The External Secrets Operator (ESO) syncs secrets from Vault, Secrets Manager, or other backends into Kubernetes Secrets. It reconciles periodically, so secret rotations propagate automatically. The sync interval introduces a delay — a rotated secret might take up to the reconciliation period to appear in Kubernetes. For most services this is fine, but for security-critical rotations (like a compromised credential), you might need to trigger an immediate sync.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payments-db-credentials
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: payments-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/payments/database
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/payments/database
        property: password

CSI Secret Store Driver is an alternative approach that mounts secrets as files in the pod's filesystem instead of creating Kubernetes Secret objects. This avoids storing secrets in etcd entirely — the secret exists only in the pod's tmpfs mount. The tradeoff is that environment variable injection doesn't work with CSI volumes, so your application needs to read secrets from files instead of environment variables. Some frameworks handle this natively (Spring Cloud Vault, for example), others need wrapper scripts.

Secret Rotation Strategy

Rotation policy depends on the secret type. Database passwords should rotate every 30-90 days. API keys for external services should rotate at least annually. Encryption keys follow their own lifecycle based on compliance requirements (PCI DSS mandates annual rotation for data-encrypting keys).

The hardest part of rotation isn't the rotation itself — it's coordinating the rotation across all consumers. If you rotate a database password, every service that uses that password needs to pick up the new credential without downtime. Dynamic secrets solve this elegantly (each service has its own credential, so rotation is independent). For static secrets, the pattern is: create the new credential, update the secret store, wait for all consumers to pick up the new value (verify through monitoring), then revoke the old credential. Skipping the verification step is how rotation causes outages.

Build rotation runbooks before you need them. When a secret is compromised, you don't want to be figuring out the rotation procedure under pressure. Every secret in your inventory should have a documented rotation procedure, and that procedure should be tested at least annually — preferably through automated rotation that runs continuously.