Multi-Account Deployments Without the Manual Overhead
Managing CloudFormation across 20+ AWS accounts by hand isn't just tedious — it's genuinely dangerous. One missed account during a security group update means one environment running with outdated firewall rules. StackSets were built to fix exactly this problem.
I've deployed StackSets across organizations ranging from 15 to 200+ accounts. The complexity scales non-linearly, and there are operational pitfalls that AWS documentation covers only in passing. Here's what you actually need to know.
How StackSets Work Under the Hood
A StackSet is a CloudFormation template plus a set of target accounts and regions. When you create or update a StackSet, CloudFormation creates individual stack instances in each target. Each stack instance is a regular CloudFormation stack in the target account — it has its own resources, events, and outputs.
The critical detail: StackSets use IAM roles for cross-account access. The administration account assumes a role in each target account to create and manage stacks. These roles must exist before you can deploy anything. For AWS Organizations-managed StackSets, AWS creates service-linked roles automatically. For self-managed StackSets, you create them manually — which is a chicken-and-egg problem I'll address shortly.
# Deploy a StackSet to all accounts in an OU
aws cloudformation create-stack-set --stack-set-name security-baseline --template-body file://security-baseline.yaml --permission-model SERVICE_MANAGED --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false --capabilities CAPABILITY_NAMED_IAM
# Create stack instances in specific OUs
aws cloudformation create-stack-instances --stack-set-name security-baseline --deployment-targets OrganizationalUnitIds='["ou-abc123","ou-def456"]' --regions '["us-east-1","eu-west-1"]'The --auto-deployment flag is important. With it enabled, any new account added to the targeted OUs automatically receives a stack instance. Without it, you need to manually add stack instances when accounts are created. I've seen organizations forget this and discover three months later that a dozen new accounts have no security baseline deployed.
Service-Managed vs Self-Managed Permissions
Service-managed StackSets use AWS Organizations to handle cross-account access. You don't create or manage IAM roles — AWS does it through a service-linked role. This is simpler and should be your default choice if you're using AWS Organizations (and you should be).
Self-managed StackSets require two roles: AWSCloudFormationStackSetAdministrationRole in the admin account, and AWSCloudFormationStackSetExecutionRole in every target account. Here's the chicken-and-egg problem: you need the execution role in target accounts before you can deploy StackSets to those accounts. But how do you deploy the role itself?
Options, in order of my preference:
Use a separate Organizations SCP that grants the admin account permission to assume roles in member accounts, then deploy the execution role via a service-managed StackSet. Once it exists, switch your other StackSets to self-managed if you need the additional flexibility self-managed provides.
Or use the Organizations API directly with a Lambda function that creates the execution role in each new account as part of your account vending process. This is what most large organizations do, and it integrates cleanly with account factory patterns.
Template Design for StackSets
StackSet templates need to be account-agnostic and region-agnostic. This sounds obvious, but it's where most StackSet deployments hit their first snag. Hardcoded AMI IDs, region-specific resource ARNs, account-specific bucket names — all of these break when the template deploys to a different account or region.
Parameters:
Environment:
Type: String
Default: production
AllowedValues: [production, staging, development]
Mappings:
RegionConfig:
us-east-1:
AmiId: ami-0abcdef1234567890
AzCount: 3
eu-west-1:
AmiId: ami-0fedcba0987654321
AzCount: 3
ap-southeast-1:
AmiId: ami-0112233445566778899
AzCount: 2
Resources:
SecurityAuditBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub "security-audit-${AWS::AccountId}-${AWS::Region}"
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256The ${AWS::AccountId} and ${AWS::Region} pseudo parameters are essential. They ensure resource names don't collide across accounts. Without them, you'll get "bucket already exists" errors that fail the deployment in every account after the first.
Handling Regional Differences
Not every AWS region supports every service. If your StackSet template creates a GuardDuty detector, it'll fail in regions where GuardDuty isn't available. Use Conditions to handle this gracefully:
Conditions:
IsGuardDutyRegion: !Or
- !Equals [!Ref "AWS::Region", "us-east-1"]
- !Equals [!Ref "AWS::Region", "us-west-2"]
- !Equals [!Ref "AWS::Region", "eu-west-1"]
Resources:
GuardDutyDetector:
Type: AWS::GuardDuty::Detector
Condition: IsGuardDutyRegion
Properties:
Enable: trueDeployment Ordering and Failure Tolerance
By default, StackSets deploy to all accounts in parallel. This is usually fine, but there are cases where ordering matters — like when one stack's output (a VPC peering connection ID) is needed by another stack in a different account.
The FailureToleranceCount parameter controls how many accounts can fail before the entire operation stops. Setting it to 0 means one failure stops everything. Setting it to the total account count means every account tries regardless of failures. I've found a tolerance of 10% strikes the right balance — it prevents a cascade from a template bug while allowing for transient API errors in individual accounts.
aws cloudformation create-stack-instances --stack-set-name security-baseline --deployment-targets OrganizationalUnitIds='["ou-abc123"]' --regions '["us-east-1"]' --operation-preferences FailureToleranceCount=2,MaxConcurrentCount=10,RegionConcurrencyType=PARALLELMaxConcurrentCount limits how many accounts deploy simultaneously. For StackSets that create IAM roles (which are global resources), I set this lower to avoid API throttling. AWS IAM's API rate limits don't care that you're deploying across accounts — they throttle based on the calling principal, which is the admin account's role.
Drift Detection and Updates
Stack drift is when the actual state of resources doesn't match the CloudFormation template. Someone manually modifies a security group, deletes a tag, or resizes an instance through the console. StackSets support drift detection, but it's not automatic — you have to initiate it.
We run drift detection weekly via a Lambda function that calls detect-stack-set-drift and posts the results to a Slack channel. When drift is detected, we evaluate whether the manual change was intentional (and needs to be incorporated into the template) or accidental (and needs to be reverted by updating the stack). The answer isn't always "revert the drift" — sometimes the manual change was a hotfix that hasn't been templated yet, and reverting it would break production.
StackSet updates follow the same deployment model as creation. You update the template, and CloudFormation rolls out changes across accounts. For critical StackSets (security baseline, networking), I stage the rollout: deploy to the sandbox OU first, validate, then promote to staging, then production. This catches template issues before they affect production accounts, and it's straightforward to implement with sequential create-stack-instances calls targeting different OUs.
Troubleshooting Common StackSet Failures
The most common failure I encounter is "Account [account-id] should have [role-name] with a trust relationship to [admin-account]." This means the execution role doesn't exist or doesn't trust the admin account. For service-managed StackSets, this usually means the organization trust policy isn't enabled in the admin account settings. For self-managed, it means you need to create the execution role in the target account.
The second most common failure is resource naming collisions. If you're deploying a StackSet to 50 accounts and the template creates a resource with a static name (not account-specific), the first account succeeds and the remaining 49 fail. Always use ${AWS::AccountId} and ${AWS::Region} in resource names for StackSet templates.
Timeout failures are trickier. StackSets have an operation timeout that defaults to one hour. If you're deploying to many accounts and the MaxConcurrentCount is low, the operation might not reach all accounts within the timeout. Increase the timeout or increase concurrency. I've seen organizations set MaxConcurrentCount to 5 for "safety" across 100 accounts, resulting in deployments that take two hours and regularly time out.
StackSets vs Terraform for Multi-Account
StackSets aren't the only way to manage multi-account infrastructure. Terraform with provider aliases can do the same thing. The question is which tool fits your team better.
StackSets excel when you need to deploy the same template to many accounts with minimal variation. Security baselines, GuardDuty detectors, CloudTrail configurations — resources that are identical or nearly identical across all accounts. The auto-deployment feature for new accounts is genuinely useful and has no easy Terraform equivalent.
Terraform excels when the infrastructure varies significantly between accounts or when you need to reference outputs across accounts. If your production account needs different network configurations than your development accounts, and your application deployment depends on networking outputs, Terraform's cross-state references and module composition give you more flexibility.
We use both. StackSets manage the security baseline that's identical everywhere. Terraform manages the per-account infrastructure that varies based on the account's purpose. The boundary is clear: if every account gets the same thing, StackSets. If accounts differ, Terraform.