Cost Optimization Is an Engineering Practice, Not a Quarterly Review
Most cloud cost optimization happens in crisis mode. Someone notices the AWS bill jumped 40% last month, a Slack thread erupts, engineers spend a week finding and fixing the biggest offenders, and then nobody thinks about cost again until the next spike. This reactive cycle wastes more money than the spikes themselves, because the slow drift between crises — slightly oversized instances, forgotten development environments, underutilized reserved capacity — adds up to more than the occasional runaway job.
A cost optimization framework makes cost visibility and action continuous rather than episodic. It doesn't require a dedicated FinOps team (though one helps at scale). It requires tooling, processes, and cultural norms that make cost awareness part of regular engineering work.
Visibility: You Can't Optimize What You Can't See
The first step is making costs visible to the people who control them. Cloud provider billing dashboards exist, but they're organized by service, not by team or application. An engineer looking at the AWS Cost Explorer sees $47,000 in EC2 charges but can't tell which service is responsible for which portion.
Cost allocation tags bridge this gap. Every cloud resource should have tags for team, service, environment, and cost center. Enforce tagging through IaC policies (Terraform/Checkov rules, AWS SCP tag conditions) rather than documentation — documented standards get ignored, enforced standards don't.
# Checkov custom policy: require cost allocation tags
# checkov/custom_checks/required_tags.py
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
from checkov.common.models.enums import CheckResult
class RequiredCostTags(BaseResourceCheck):
def __init__(self):
name = "Ensure required cost allocation tags"
id = "CUSTOM_001"
supported_resources = [
"aws_instance", "aws_rds_cluster", "aws_elasticache_cluster",
"aws_lambda_function", "aws_ecs_service"
]
super().__init__(name=name, id=id, categories=[], supported_resources=supported_resources)
def scan_resource_conf(self, conf):
tags = conf.get("tags", [{}])[0]
required = ["team", "service", "environment", "cost-center"]
missing = [t for t in required if t not in tags]
return CheckResult.PASSED if not missing else CheckResult.FAILEDCost Dashboards per Team
Once tags are in place, build per-team cost dashboards. The dashboard should show: total spend this month vs last month, top 5 resources by cost, spend trend over the past 6 months, and any resources flagged as potentially wasteful. Update it daily and make it accessible without requiring cloud console access — many developers don't have billing permissions.
We've had success with a weekly Slack digest that posts each team's top cost changes: "payments-team EC2 spend increased 23% ($1,240 → $1,525) — new staging environment added." Concise, actionable, and visible without seeking it out.
Right-Sizing Compute Resources
Right-sizing is the single highest-impact optimization for most organizations. AWS's Cost Explorer Rightsizing Recommendations, GCP's VM Rightsizing, and third-party tools like Spot.io all analyze CPU and memory utilization to recommend smaller instance types.
The common mistake is looking at average utilization instead of peak utilization. A service that averages 15% CPU but peaks at 90% during daily batch processing isn't overprovisioned — it's correctly sized for its workload pattern. The real waste is the service averaging 5% CPU with a peak of 12% running on an m5.2xlarge because someone copied a Terraform module from the production service that actually needs that capacity.
# Kubernetes resource right-sizing with VPA recommendations
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: payment-service-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: payment-service
updatePolicy:
updateMode: "Off" # recommendation only, don't auto-apply
resourcePolicy:
containerPolicies:
- containerName: '*'
minAllowed:
cpu: 100m
memory: 128Mi
maxAllowed:
cpu: 4
memory: 8GiRunning VPA in recommendation mode (updateMode: "Off") gives you right-sizing data without automatically changing resource allocations. Review the recommendations weekly and apply them through normal deployment processes rather than letting the autoscaler make unreviewed changes.
Reserved Instances and Savings Plans
Commitment-based discounts (Reserved Instances, Savings Plans, CUDs) offer 30-60% savings over on-demand pricing for stable workloads. The risk is overcommitting — buying a 1-year reserved instance for a service that gets decommissioned in 4 months.
Start with Compute Savings Plans (AWS) or CUDs (GCP) rather than instance-specific reservations. They're flexible across instance families and sizes, so you still get the discount if you right-size or change instance types. The discount is smaller (around 30% vs 40% for specific RIs), but the flexibility is worth it for most workloads.
Cover your stable baseline — the compute that you know you'll run for the next year regardless of what happens — with commitments. Leave your variable and experimental workloads on-demand or spot. A good target is 60-70% commitment coverage. Going higher risks paying for capacity you don't end up using.
Spot and Preemptible Instances
Spot instances (AWS), Preemptible VMs (GCP), and Spot VMs (Azure) offer 60-90% savings for workloads that can handle interruption. Batch jobs, CI runners, stateless web servers behind a load balancer, and data processing pipelines are natural fits.
The operational requirement is graceful handling of interruption. AWS gives you a 2-minute warning before terminating a spot instance. Your application needs to checkpoint its work, drain connections, or hand off to another instance within that window. Kubernetes node pools with spot instances and pod disruption budgets handle this well for containerized workloads.
Storage and Data Transfer Costs
Storage costs creep up invisibly. Old EBS snapshots, unused S3 buckets with versioning enabled (storing every version of every deleted file), Elasticsearch indexes retaining 18 months of logs — these accumulate silently because nobody watches storage line items the way they watch compute.
S3 lifecycle policies are the easiest win. Move infrequently accessed data to S3 Infrequent Access after 30 days, to Glacier after 90, and delete after whatever your retention policy requires. For versioned buckets, set a lifecycle rule that expires non-current versions after 30 days — this alone can cut storage costs by 40-60% for buckets with heavy write activity.
resource "aws_s3_bucket_lifecycle_configuration" "logs" {
bucket = aws_s3_bucket.application_logs.id
rule {
id = "log-lifecycle"
status = "Enabled"
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 90
storage_class = "GLACIER"
}
expiration {
days = 365
}
noncurrent_version_expiration {
noncurrent_days = 30
}
}
}Data transfer costs are the hidden tax of cloud computing. Cross-AZ transfer in AWS costs $0.01/GB in each direction. Cross-region costs more. Internet egress costs $0.09/GB. For services that transfer terabytes between AZs (like a Kafka cluster replicating across 3 AZs), data transfer can exceed the compute cost of the instances themselves.
Audit your top data transfer line items quarterly. Often the fix is architectural — colocating services in the same AZ, using VPC endpoints instead of internet gateways for AWS service access, or caching frequently accessed data closer to the consumer.
Building the Culture
Tools and processes aren't enough without cultural support. Engineers need to understand that cost efficiency is a feature, not an afterthought. Include cost impact in architecture reviews. Celebrate teams that reduce their cloud spend. Make cost a visible metric alongside latency, error rate, and deployment frequency.
The most effective pattern I've seen: give each team a monthly cloud budget, make actual spend visible against that budget weekly, and let teams keep any savings as "innovation budget" they can spend on tools, training, or experimentation. When saving money directly benefits the team, cost awareness becomes self-sustaining.
Automated Waste Detection
Scheduled scans that identify wasteful resources are the lowest-effort cost optimization. Run weekly checks for: EC2 instances with average CPU utilization below 5% over 14 days, EBS volumes in the "available" state (detached from any instance), Elastic IPs not associated with running instances, load balancers with zero healthy targets, and RDS instances with no connections in the past 7 days.
These checks catch resources that were provisioned for a project and forgotten when the project ended. In one audit, we found $14,000 per month in orphaned resources — development databases from a project that ended six months ago, a load balancer test setup that was never cleaned up, and EBS snapshots from a migration that completed a year prior.
# AWS CLI: find detached EBS volumes
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].{ID:VolumeId,Size:Size,Created:CreateTime}" \
--output table
# Find unattached Elastic IPs
aws ec2 describe-addresses \
--query "Addresses[?AssociationId==null].{IP:PublicIp,ID:AllocationId}" \
--output tableAutomate these checks and send results to team Slack channels weekly. Include estimated monthly cost for each identified resource so teams can prioritize cleanup. Don't auto-delete — let teams verify and clean up themselves. Auto-deletion of resources you think are unused is how you cause outages by removing something that looked idle but was actually critical infrastructure for a batch job that runs monthly.
For Kubernetes environments, look at pods requesting resources they don't use. A pod requesting 2 CPU cores but averaging 0.1 cores wastes 1.9 cores of cluster capacity that could serve other workloads or allow the cluster to scale down. VPA recommendations (in recommendation-only mode) provide the data; acting on it through regular right-sizing reviews turns that data into savings.