The Network Policy Problem in Production Clusters
If you've ever run a Kubernetes cluster without network policies, you've essentially given every pod unrestricted access to every other pod. That's fine for a dev environment where three engineers share a namespace. It's not fine when you're running 200 microservices across multiple teams and one compromised container can reach your database pods directly.
I've operated clusters with both Calico and Cilium in production, and the decision between them isn't as straightforward as most comparison posts suggest. The right choice depends on your team's operational maturity, your performance requirements, and whether you need L7 visibility.
How Kubernetes Network Policies Actually Work
Before comparing implementations, it's worth understanding what the native NetworkPolicy API actually does — and what it doesn't. A NetworkPolicy is a namespace-scoped resource that selects pods via labels and defines ingress and egress rules. The API itself doesn't enforce anything. Your CNI plugin does the enforcement.
Here's a basic deny-all policy that blocks all ingress to pods in a namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-ingress
namespace: production
spec:
podSelector: {}
policyTypes:
- IngressThis tells whatever CNI is running to drop all inbound traffic to pods in the production namespace unless another policy explicitly allows it. Simple enough. The complications start when you need something the standard API can't express.
What the Standard API Can't Do
The native NetworkPolicy API has some painful gaps. You can't write a cluster-wide default policy — every namespace needs its own. You can't match on DNS names, only IP blocks or pod selectors. There's no way to express L7 rules like "allow HTTP GET but block HTTP POST." And logging? The API has no concept of it. You're flying blind unless your CNI adds observability on top.
Calico's Architecture and Trade-offs
Calico's been around since 2016, and it shows in both good and bad ways. The architecture is mature and battle-tested. Felix, the per-node agent, programs iptables rules (or eBPF dataplane if you opt in) based on policies stored in etcd or the Kubernetes API. Typha sits between Felix and the API server to reduce load when you've got hundreds of nodes.
In my experience, Calico's strongest selling point is GlobalNetworkPolicy. This CRD lets you write one policy that applies across all namespaces:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: deny-external-egress
spec:
selector: env == 'production'
types:
- Egress
egress:
- action: Allow
destination:
selector: all()
- action: DenyThat policy blocks all external egress from production-labeled workloads while still allowing cluster-internal communication. Try doing that with native NetworkPolicies — you'll need one per namespace, and you'll miss any new namespace someone creates next Tuesday.
Calico's iptables vs eBPF Dataplane
Calico historically used iptables for packet filtering. It works, but iptables performance degrades linearly with rule count. At around 5,000 services, we started seeing noticeable latency increases in connection setup time — not in steady-state throughput, but in the initial SYN processing. Each new connection had to traverse a longer chain.
Calico's eBPF dataplane fixes this. It bypasses iptables entirely, programs BPF maps for policy evaluation, and handles service routing in eBPF. We measured a 25% reduction in p99 connection setup latency after switching on a 300-node cluster. The catch? You lose compatibility with anything else that expects iptables rules to exist — kube-proxy iptables mode, some service mesh implementations, and custom iptables rules your team might have added.
Cilium's Architecture and Strengths
Cilium was built on eBPF from day one, and that architectural decision permeates everything. There's no iptables fallback. The agent compiles BPF programs that attach to the TC (traffic control) hook on each pod's veth pair. Policy evaluation, service load balancing, and observability all happen in kernel space.
Where Cilium really shines is Hubble, its built-in observability layer. Running hubble observe gives you real-time flow logs that show source pod, destination pod, L4 port, L7 protocol details, and whether the flow was allowed or denied. I've used Hubble to debug connectivity issues that would've taken hours with tcpdump alone:
$ hubble observe --namespace production --verdict DROPPED
TIMESTAMP SOURCE DESTINATION TYPE VERDICT SUMMARY
Sep 15 14:23:01.234 production/api-server production/cache-redis L4 DROPPED TCP Flags: SYN
Sep 15 14:23:01.891 production/worker-batch kube-system/coredns L4 DROPPED UDP 53That output tells me immediately that my api-server can't reach Redis and my batch workers can't resolve DNS. With Calico, I'd be checking iptables counters on the node and correlating timestamps.
L7 Policies With Cilium
Cilium's CiliumNetworkPolicy CRD supports L7-aware rules. You can restrict a pod to only making GET requests to specific URL paths:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: api-http-policy
namespace: production
spec:
endpointSelector:
matchLabels:
app: frontend
egress:
- toEndpoints:
- matchLabels:
app: api-server
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: "/api/v1/.*"This is genuinely useful when you've got a frontend that should only read from your API, never write. The enforcement happens in the kernel via an HTTP parser compiled into the BPF program. It adds some latency — we measured about 0.3ms per request on average — but it's a fraction of what a sidecar proxy would add.
Performance Comparison in Practice
I've benchmarked both on identical hardware — 50-node clusters running AMD EPYC 7543 processors, 25Gbps network interfaces, using iperf3 for throughput and custom Go tools for connection rate testing.
Throughput with no policies applied: Calico eBPF hit 23.1 Gbps, Cilium hit 23.4 Gbps. Statistically insignificant difference. With 500 policies applied, Calico eBPF dropped to 22.7 Gbps while Cilium stayed at 23.2 Gbps. The gap widened at 2,000 policies — Calico eBPF at 21.8 Gbps, Cilium at 22.9 Gbps. Cilium's BPF map lookups scale better than Calico's approach to policy evaluation.
Connection setup rate told a more interesting story. Calico eBPF handled 48,000 new connections per second with 500 policies. Cilium managed 52,000. At 2,000 policies, Calico dropped to 39,000 while Cilium held at 49,000. If your workloads create and tear down connections frequently — think short-lived HTTP requests with no connection pooling — Cilium's advantage compounds.
Operational Complexity
Here's where the comparison gets uncomfortable. Cilium is harder to operate. When something goes wrong with Calico, you can inspect iptables rules or BPF maps with familiar tools. When something goes wrong with Cilium, you're reading BPF bytecode or running cilium bpf policy get to inspect compiled programs. The learning curve is steeper.
Cilium also has tighter kernel version requirements. You'll need Linux 4.19 minimum, and realistically 5.10+ for full feature support. Calico's iptables dataplane runs on anything. Its eBPF dataplane needs 5.3+ but degrades gracefully to iptables on older kernels.
Upgrades are another consideration. Calico upgrades tend to be straightforward — update the DaemonSet, Felix restarts, policies re-sync. Cilium upgrades sometimes require BPF program recompilation on every node, and I've hit cases where a kernel/Cilium version mismatch caused temporary connectivity drops during rolling updates. The team's documented this and it's gotten better, but it's still something you need to test in staging before rolling to production.
Making the Decision
After running both in production, here's how I'd frame the decision. Choose Calico if your team is smaller, you need to support older kernel versions, or you're primarily implementing L3/L4 policies. Calico's GlobalNetworkPolicy and NetworkSet CRDs cover most enterprise security requirements without the operational overhead of Cilium.
Choose Cilium if you need L7 visibility, your workloads have high connection churn, or you're already investing in eBPF-based observability. Hubble alone is worth the migration cost for teams that spend significant time debugging service-to-service communication issues.
Don't choose based on benchmarks alone. Both are fast enough for the vast majority of workloads. Choose based on what your team can operate reliably at 2 AM when something breaks.
Monitoring and Alerting for Network Policies
Whichever CNI you pick, you'll need visibility into policy enforcement. Denied connections don't generate application-level errors - the TCP SYN simply never reaches the destination. Without flow monitoring, you're left guessing why a service can't connect.
Cilium's Hubble solves this natively. For Calico, you'll need to enable flow logs explicitly. On the eBPF dataplane, set flowLogsFlushInterval in the FelixConfiguration resource. On the iptables dataplane, enable LOG targets for denied packets - though this can generate significant log volume on busy clusters.
I run a Grafana dashboard that tracks denied connections per namespace per hour. A sudden spike in denials after a deployment usually means someone added a new service that the network policies don't account for. Catching that within minutes of deployment instead of after a customer reports an error is worth the monitoring investment.
One thing I've learned the hard way: test network policies in a staging environment that mirrors production's policy set. A policy that works in dev (where there are fewer services and simpler communication patterns) might break in production because it doesn't account for cross-namespace communication with the monitoring stack or the ingress controller. Policy testing frameworks like netpol-analyzer can simulate traffic patterns against your policies before you apply them.