Log Aggregation Architecture: Loki vs...

Log Storage Is the Budget Line Nobody Planned For

Every observability team I've worked with has had the same reckoning. Elasticsearch cluster costs hit $15,000/month, someone asks "do we really need all these logs?", and the conversation about alternatives begins. Grafana Loki enters that conversation as the cost-effective challenger, but the tradeoffs are real and poorly understood.

I've migrated two organizations from Elasticsearch to Loki and helped a third decide to stay on Elasticsearch. The right choice depends on how your team actually queries logs, not just how much you're spending.

Architecture: Inverted Index vs. Label Index

Elasticsearch indexes every word in every log line. This makes arbitrary full-text search fast -- "find all logs containing 'connection refused' from any service in the last hour" returns in seconds. The cost is storage: each log line requires 1.5-3x its raw size after indexing.

Loki only indexes metadata labels (service name, pod, namespace, log level). The log content itself is compressed and stored as chunks without indexing. Queries filter by labels first, then grep through the matching chunks for content patterns.

# Loki configuration for object storage
schema_config:
 configs:
 - from: 2024-01-01
 store: tsdb
 object_store: s3
 schema: v13
 index:
 prefix: loki_index_
 period: 24h

storage_config:
 tsdb_shipper:
 active_index_directory: /loki/index
 cache_location: /loki/index_cache
 aws:
 s3: s3://us-east-1/company-loki-chunks
 bucketnames: company-loki-chunks
 region: us-east-1

What This Means in Practice

Searching by known labels in Loki is fast. "Show me all ERROR logs from the payment service in the last hour" executes in under a second because Loki only reads chunks matching those labels.

Searching by unknown content is slow. "Find all logs containing 'timeout' across all services in the last 24 hours" means Loki reads and decompresses every chunk for that time range. On a busy cluster producing 50GB of logs per hour, that query takes minutes.

Elasticsearch handles both patterns equally well because everything is indexed. The question is whether your team primarily uses structured queries (Loki is fine) or exploratory grep-style searches (Elasticsearch is necessary).

Deployment Models

Loki Simple Scalable

The recommended deployment for most organizations. Three target types -- read, write, and backend -- running as separate deployment groups sharing the same binary.

# Helm values for simple scalable mode
loki:
 commonConfig:
 replication_factor: 3
 schemaConfig:
 configs:
 - from: "2024-01-01"
 store: tsdb
 object_store: s3
 schema: v13
 index:
 prefix: loki_index_
 period: 24h

write:
 replicas: 3
 resources:
 requests:
 cpu: "2"
 memory: 4Gi

read:
 replicas: 3
 resources:
 requests:
 cpu: "2"
 memory: 8Gi

backend:
 replicas: 2
 resources:
 requests:
 cpu: "1"
 memory: 2Gi

Elasticsearch Cluster Sizing

An equivalent Elasticsearch deployment for the same log volume needs more compute and storage:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
 name: logs-cluster
spec:
 version: 8.14.0
 nodeSets:
 - name: hot
 count: 3
 config:
 node.roles: ["data_hot", "ingest", "master"]
 podTemplate:
 spec:
 containers:
 - name: elasticsearch
 resources:
 requests:
 memory: 16Gi
 cpu: "4"
 volumeClaimTemplates:
 - metadata:
 name: elasticsearch-data
 spec:
 resources:
 requests:
 storage: 500Gi
 storageClassName: gp3
 - name: warm
 count: 2
 config:
 node.roles: ["data_warm"]
 podTemplate:
 spec:
 containers:
 - name: elasticsearch
 resources:
 requests:
 memory: 8Gi
 cpu: "2"
 volumeClaimTemplates:
 - metadata:
 name: elasticsearch-data
 spec:
 resources:
 requests:
 storage: 2Ti

Ingestion Pipeline Comparison

Both platforms accept logs from standard agents. Promtail and Grafana Alloy push to Loki; Filebeat and Fluentd push to Elasticsearch. The pipeline configuration determines how much structure your logs have at query time.

# Promtail pipeline for structured logs
scrape_configs:
 - job_name: kubernetes
 kubernetes_sd_configs:
 - role: pod
 pipeline_stages:
 - docker: {}
 - match:
 selector: '{app="api"}'
 stages:
 - regex:
 expression: '^(?P<timestamp>\S+) (?P<level>\w+) (?P<msg>.*)$'
 - labels:
 level:
 - timestamp:
 source: timestamp
 format: "2006-01-02T15:04:05.000Z"

Loki's reliance on labels means getting the pipeline right matters more. Too few labels and queries scan too much data. Too many labels (high cardinality -- like request IDs) and the index explodes, negating the cost advantage.

This connects to the ideas in CloudFormation StackSets for Multi-Account AWS Infrastructur.

Query Language Comparison

LogQL (Loki) borrows PromQL syntax. KQL (Elasticsearch/Kibana) is its own language. Both are powerful; LogQL feels natural to Prometheus users, KQL to anyone who's used Splunk or similar tools.

# LogQL: error rate by service over 5 minutes
sum(rate({namespace="production"} |= "error" [5m])) by (app)

# LogQL: parse and filter structured logs
{app="api"} | json | status >= 500 | line_format "{{.method}} {{.path}} {{.status}}"

# KQL equivalent
response.status >= 500 AND kubernetes.namespace: "production"

LogQL's metric queries are unique. You can derive metrics from log patterns without a separate metrics pipeline. "Count of 5xx errors per minute by endpoint" works as a LogQL query piped into a Grafana panel.

Cost Breakdown

For a mid-sized production environment ingesting 200GB of logs per day with 30-day retention:

Elasticsearch on AWS (OpenSearch): 3 hot nodes (r6g.xlarge) + 2 warm nodes (r6g.large) + storage. Roughly $4,200/month.

Loki on AWS: 3 write + 3 read + 2 backend pods (much smaller instances) + S3 storage. Roughly $1,100/month. The S3 storage for compressed chunks is dramatically cheaper than EBS volumes for Elasticsearch indexes.

This connects to the ideas in Capacity Planning Models for Cloud-Native Applications.

That's a 73% cost reduction. In our actual migration, we saw 68% savings because we needed beefier read nodes for the grep-heavy query patterns our team used.

When Elasticsearch Wins

Security and compliance use cases. SIEM tools expect full-text search across arbitrary fields. SOC analysts grep for IP addresses, domain names, and attack signatures they couldn't predict in advance. Loki's label-first model makes this painful.

Complex log analytics. Business intelligence from log data -- "how many unique users hit endpoint X with status 200 in the EU region this week" -- works better with Elasticsearch's aggregation framework.

Teams already using Kibana dashboards extensively. Migration cost isn't just infrastructure -- it's rebuilding dozens of dashboards and retraining analysts on a new query language.

When Loki Wins

Teams already running Prometheus and Grafana. Loki integrates natively with the Grafana stack, and the correlation between metrics and logs using the same label set is genuinely useful for incident response.

Cost-sensitive environments where log retention matters more than search flexibility. If most queries are "show me logs from service X at time Y" rather than "find all logs containing pattern Z anywhere," Loki handles that efficiently at a fraction of the cost.

Kubernetes-native environments where pod labels provide natural log organization. Loki's label model maps perfectly onto Kubernetes metadata.

Log Aggregation Architecture: Loki vs Elasticsearch for Cost-Effective Observability