Log Aggregation Architecture: Loki vs Elasticsearch for Cost-Effective Observability

Log Storage Is the Budget Line Nobody Planned For

Every observability team I've worked with has had the same reckoning. Elasticsearch cluster costs hit $15,000/month, someone asks "do we really need all these logs?", and the conversation about alternatives begins. Grafana Loki enters that conversation as the cost-effective challenger, but the tradeoffs are real and poorly understood.

I've migrated two organizations from Elasticsearch to Loki and helped a third decide to stay on Elasticsearch. The right choice depends on how your team actually queries logs, not just how much you're spending.

Architecture: Inverted Index vs. Label Index

Elasticsearch indexes every word in every log line. This makes arbitrary full-text search fast -- "find all logs containing 'connection refused' from any service in the last hour" returns in seconds. The cost is storage: each log line requires 1.5-3x its raw size after indexing.

Loki only indexes metadata labels (service name, pod, namespace, log level). The log content itself is compressed and stored as chunks without indexing. Queries filter by labels first, then grep through the matching chunks for content patterns.

# Loki configuration for object storage
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
  aws:
    s3: s3://us-east-1/company-loki-chunks
    bucketnames: company-loki-chunks
    region: us-east-1

What This Means in Practice

Searching by known labels in Loki is fast. "Show me all ERROR logs from the payment service in the last hour" executes in under a second because Loki only reads chunks matching those labels.

Searching by unknown content is slow. "Find all logs containing 'timeout' across all services in the last 24 hours" means Loki reads and decompresses every chunk for that time range. On a busy cluster producing 50GB of logs per hour, that query takes minutes.

Elasticsearch handles both patterns equally well because everything is indexed. The question is whether your team primarily uses structured queries (Loki is fine) or exploratory grep-style searches (Elasticsearch is necessary).

Deployment Models

Loki Simple Scalable

The recommended deployment for most organizations. Three target types -- read, write, and backend -- running as separate deployment groups sharing the same binary.

# Helm values for simple scalable mode
loki:
  commonConfig:
    replication_factor: 3
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

write:
  replicas: 3
  resources:
    requests:
      cpu: "2"
      memory: 4Gi

read:
  replicas: 3
  resources:
    requests:
      cpu: "2"
      memory: 8Gi

backend:
  replicas: 2
  resources:
    requests:
      cpu: "1"
      memory: 2Gi

Elasticsearch Cluster Sizing

An equivalent Elasticsearch deployment for the same log volume needs more compute and storage:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logs-cluster
spec:
  version: 8.14.0
  nodeSets:
  - name: hot
    count: 3
    config:
      node.roles: ["data_hot", "ingest", "master"]
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 16Gi
              cpu: "4"
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        resources:
          requests:
            storage: 500Gi
        storageClassName: gp3
  - name: warm
    count: 2
    config:
      node.roles: ["data_warm"]
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 8Gi
              cpu: "2"
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        resources:
          requests:
            storage: 2Ti

Ingestion Pipeline Comparison

Both platforms accept logs from standard agents. Promtail and Grafana Alloy push to Loki; Filebeat and Fluentd push to Elasticsearch. The pipeline configuration determines how much structure your logs have at query time.

# Promtail pipeline for structured logs
scrape_configs:
  - job_name: kubernetes
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - docker: {}
      - match:
          selector: '{app="api"}'
          stages:
            - regex:
                expression: '^(?P<timestamp>\S+) (?P<level>\w+) (?P<msg>.*)$'
            - labels:
                level:
            - timestamp:
                source: timestamp
                format: "2006-01-02T15:04:05.000Z"

Loki's reliance on labels means getting the pipeline right matters more. Too few labels and queries scan too much data. Too many labels (high cardinality -- like request IDs) and the index explodes, negating the cost advantage.

Query Language Comparison

LogQL (Loki) borrows PromQL syntax. KQL (Elasticsearch/Kibana) is its own language. Both are powerful; LogQL feels natural to Prometheus users, KQL to anyone who's used Splunk or similar tools.

# LogQL: error rate by service over 5 minutes
sum(rate({namespace="production"} |= "error" [5m])) by (app)

# LogQL: parse and filter structured logs
{app="api"} | json | status >= 500 | line_format "{{.method}} {{.path}} {{.status}}"

# KQL equivalent
response.status >= 500 AND kubernetes.namespace: "production"

LogQL's metric queries are unique. You can derive metrics from log patterns without a separate metrics pipeline. "Count of 5xx errors per minute by endpoint" works as a LogQL query piped into a Grafana panel.

Cost Breakdown

For a mid-sized production environment ingesting 200GB of logs per day with 30-day retention:

Elasticsearch on AWS (OpenSearch): 3 hot nodes (r6g.xlarge) + 2 warm nodes (r6g.large) + storage. Roughly $4,200/month.

Loki on AWS: 3 write + 3 read + 2 backend pods (much smaller instances) + S3 storage. Roughly $1,100/month. The S3 storage for compressed chunks is dramatically cheaper than EBS volumes for Elasticsearch indexes.

That's a 73% cost reduction. In our actual migration, we saw 68% savings because we needed beefier read nodes for the grep-heavy query patterns our team used.

When Elasticsearch Wins

Security and compliance use cases. SIEM tools expect full-text search across arbitrary fields. SOC analysts grep for IP addresses, domain names, and attack signatures they couldn't predict in advance. Loki's label-first model makes this painful.

Complex log analytics. Business intelligence from log data -- "how many unique users hit endpoint X with status 200 in the EU region this week" -- works better with Elasticsearch's aggregation framework.

Teams already using Kibana dashboards extensively. Migration cost isn't just infrastructure -- it's rebuilding dozens of dashboards and retraining analysts on a new query language.

When Loki Wins

Teams already running Prometheus and Grafana. Loki integrates natively with the Grafana stack, and the correlation between metrics and logs using the same label set is genuinely useful for incident response.

Cost-sensitive environments where log retention matters more than search flexibility. If most queries are "show me logs from service X at time Y" rather than "find all logs containing pattern Z anywhere," Loki handles that efficiently at a fraction of the cost.

Kubernetes-native environments where pod labels provide natural log organization. Loki's label model maps perfectly onto Kubernetes metadata.