Prometheus Scaling Strategies: Thanos vs Cortex vs Victoria Metrics

When Prometheus Can't Keep Up Anymore

Prometheus works beautifully for a single cluster. Then you add a second. Then a third. Suddenly you're running a dozen Prometheus instances, each with its own data silo, and nobody can answer "what's the 95th percentile latency across all regions?" without stitching together twelve different dashboards.

I've operated monitoring stacks at that scale for three years. The problem isn't Prometheus itself -- it's that Prometheus was designed as a single-node system in a world that increasingly needs multi-cluster, multi-region observability.

The Core Problem: Federation Doesn't Scale

Prometheus federation -- where a global Prometheus scrapes aggregated metrics from leaf instances -- was the original answer. It breaks down around 10 million active time series. The global instance becomes a bottleneck, queries time out, and you're stuck choosing between data resolution and query performance.

Federation also loses granularity. You pre-aggregate at the leaf level, which means the global view can only answer questions you anticipated. "Show me p99 latency by endpoint" works if you federated that specific recording rule. "Show me p99 latency for endpoints matching /api/v2/*" doesn't, because the raw data never left the leaf instance.

Thanos: The Sidecar Approach

Thanos bolts onto existing Prometheus instances with a sidecar that uploads TSDB blocks to object storage (S3, GCS, or Azure Blob). A central Querier component provides a unified PromQL interface across all instances.

# Thanos sidecar alongside Prometheus
containers:
- name: prometheus
  image: prom/prometheus:v2.51.0
  args:
    - '--storage.tsdb.min-block-duration=2h'
    - '--storage.tsdb.max-block-duration=2h'
    - '--storage.tsdb.retention.time=6h'
- name: thanos-sidecar
  image: thanosio/thanos:v0.35.0
  args:
    - sidecar
    - '--tsdb.path=/prometheus/data'
    - '--objstore.config-file=/etc/thanos/objstore.yml'
    - '--prometheus.url=http://localhost:9090'

Compaction and Downsampling

Thanos Compactor merges uploaded blocks and creates downsampled versions -- 5-minute and 1-hour resolution alongside raw data. Long-range queries automatically hit downsampled data, making "show me CPU usage over 6 months" fast instead of impossible.

# Thanos compactor configuration
args:
  - compact
  - '--data-dir=/var/thanos/compact'
  - '--objstore.config-file=/etc/thanos/objstore.yml'
  - '--retention.resolution-raw=30d'
  - '--retention.resolution-5m=180d'
  - '--retention.resolution-1h=365d'
  - '--compact.concurrency=4'
  - '--downsample.concurrency=4'

Raw data kept for 30 days, 5-minute resolution for 6 months, 1-hour resolution for a year. Storage cost stays manageable because downsampled data is tiny compared to raw.

Query Architecture

The Querier fans out PromQL queries to multiple Store APIs -- sidecars for recent data, the Store Gateway for historical data in object storage. From a Grafana dashboard perspective, you point at the Querier and everything looks like one giant Prometheus.

I've seen this work cleanly up to about 500 million active time series across 40 Prometheus instances. Beyond that, the Querier's fan-out starts showing latency problems on wide queries.

Cortex: Multi-Tenant by Design

Cortex (now largely succeeded by Grafana Mimir, but the architecture is the same) takes a different approach. Instead of bolting onto Prometheus, it replaces the storage backend entirely. Prometheus remote-writes metrics into Cortex, which handles distribution, replication, and querying.

# Prometheus remote write to Cortex
remote_write:
  - url: http://cortex-distributor:9009/api/v1/push
    headers:
      X-Scope-OrgID: team-payments
    queue_config:
      max_samples_per_send: 5000
      batch_send_deadline: 5s
      max_shards: 30

The Ring Architecture

Cortex uses a hash ring (backed by Consul, etcd, or memberlist) to distribute time series across ingesters. Each series is replicated to three ingesters by default. When an ingester fails, its peers already have copies of the data.

This makes Cortex genuinely horizontally scalable. Need more write throughput? Add more ingesters. More query capacity? Add more queriers. Each component scales independently based on its specific bottleneck.

Multi-Tenancy

Cortex was built for multi-tenancy from day one. The X-Scope-OrgID header isolates tenants completely -- their data never intermixes in storage, and per-tenant rate limits prevent one noisy team from degrading the platform for everyone else.

limits:
  ingestion_rate: 200000          # samples/sec per tenant
  ingestion_burst_size: 400000
  max_series_per_metric: 50000
  max_series_per_user: 5000000
  max_global_series_per_user: 10000000

Running this as a shared platform for 15 teams, we found the tenant isolation was critical. One team accidentally deployed a cardinality bomb -- a label with request IDs, producing millions of unique series. Cortex's per-tenant limits caught it before it affected anyone else.

Victoria Metrics: The Efficiency Play

Victoria Metrics optimizes for storage efficiency and query speed on a single node. Its compression is substantially better than Prometheus -- we consistently see 0.4-0.8 bytes per data point versus Prometheus's 1.3-1.5 bytes. On a 200 million series dataset, that's the difference between 8TB and 20TB of storage.

# Victoria Metrics single-node
docker run -d \
  --name victoria-metrics \
  -v /data/vm:/victoria-metrics-data \
  victoriametrics/victoria-metrics:v1.101.0 \
  -retentionPeriod=12 \
  -storageDataPath=/victoria-metrics-data \
  -httpListenAddr=:8428 \
  -search.maxUniqueTimeseries=5000000 \
  -dedup.minScrapeInterval=30s

VictoriaMetrics Cluster Mode

The cluster version splits into three components: vminsert (ingestion), vmselect (queries), and vmstorage (storage). Each scales independently, similar to Cortex but with less operational overhead -- no external coordination service like Consul required.

# vminsert
args:
  - '--storageNode=vmstorage-0:8400'
  - '--storageNode=vmstorage-1:8400'
  - '--storageNode=vmstorage-2:8400'
  - '--replicationFactor=2'

# vmstorage
args:
  - '--retentionPeriod=6'
  - '--storageDataPath=/data'
  - '--dedup.minScrapeInterval=30s'

MetricsQL Extensions

Victoria Metrics extends PromQL with useful functions. range_median, histogram_quantile improvements, and label_graphite_group make complex queries simpler. These extensions are backward-compatible -- standard PromQL still works.

Decision Framework

Choose Thanos if you have existing Prometheus instances you can't replace, want to keep data close to the source with global querying, and your scale is under 500 million series. The sidecar model minimizes disruption.

Choose Cortex/Mimir if you need genuine multi-tenancy, want a centralized metrics platform as a service for multiple teams, or are starting fresh without legacy Prometheus infrastructure. The operational complexity is higher but the ceiling is highest.

Choose Victoria Metrics if storage cost and query performance matter most, your team is small and can't operate a distributed system with many components, or you want the simplest path from single-node to cluster mode. The single-node option handles surprisingly large workloads.

Cost Comparison at Scale

At 100 million active series with 1-year retention on AWS:

Thanos: ~$2,400/month (mostly S3 storage + Querier compute). Compaction reduces storage substantially.

Cortex/Mimir: ~$4,200/month (ingesters need memory, plus S3 storage). Higher compute cost for the distributed architecture.

Victoria Metrics: ~$1,800/month (better compression = less storage, single-node handles more before requiring cluster mode).

Don't pick based on cost alone. The cheapest option that requires an extra engineer to operate isn't actually cheapest. Match the system to your team's operational maturity and the scale you'll hit in the next two years, not just today's requirements.