Zero-Overhead Microsegmentation: Harnessing eBPF and Cilium for Multi-Cloud Kubernetes Security | Ecstaticloud Blog

As multi-cloud Kubernetes deployments scale into thousands of microservices across heterogeneous environments like AWS EKS, GCP GKE, and on-premises clusters, securing inter-service communication becomes paramount. For years, the industry relied on service meshes powered by sidecar proxy architectures (such as Envoy injected via Istio or Linkerd) to achieve microsegmentation, mutual TLS (mTLS), and Layer 7 observability.

However, the "sidecar tax" has caught up with platform engineering teams.

Injecting a dedicated proxy container into every application pod introduces crippling resource overhead, inflates cloud bills, and adds latency to every network hop.

In this deep dive, we will explore how eBPF (Extended Berkeley Packet Filter) and Cilium eliminate the sidecar proxy entirely. By moving network policy enforcement, encryption, and observability directly into the Linux kernel, you can achieve zero-overhead microsegmentation across multi-cloud Kubernetes infrastructure.

The Death of the Sidecar Tax

To understand why eBPF is revolutionary, we must first analyze why the sidecar proxy pattern fails at scale.

In a traditional sidecar model, outbound network traffic from an application container does not flow directly to the network interface. Instead, it undergoes a complex, multi-step redirection path:

[ Pod Space ]
+-------------------------------------------------------------+
| +-------------------+             +-----------------------+ |
| | Application Pod   |             | Envoy Sidecar Proxy   | |
| | (Userspace)       |             | (Userspace)           | |
| +---------+---------+             +-----------^-----------+ |
|           |                                   |             |
|           | Socket write                      | Socket read |
|           v                                   |             |
| +---------+-----------------------------------+-----------+ |
| | Linux Kernel TCP/IP Stack & iptables PREROUTING         | |
| +---------------------------------------------------------+ |
+-------------------------------------------------------------+

Context Switches & Double TCP Traversals: A single outbound request traverses the kernel’s TCP/IP stack four times (App $\rightarrow$ Kernel $\rightarrow$ Sidecar Proxy $\rightarrow$ Kernel $\rightarrow$ Wire). This causes heavy context switching between userspace and kernel space, adding 2ms to 10ms of latency per hop.
Resource Bloat: Each sidecar proxy consumes between 30MB and 150MB of RAM, alongside dedicated CPU cycles for keepalives, telemetry processing, and dynamic configuration syncing (xDS). In a cluster with 2,000 pods, you may waste upwards of 200GB of RAM and dozens of CPU cores just to run the security proxy layer.
Operational Complexity: Injected sidecars complicate pod lifecycle management, break job completions (Kubernetes CronJobs hanging because the sidecar never exits), and introduce security vulnerabilities through expanded attack surfaces.

The eBPF Paradigm: Running Code at Kernel Speed

eBPF shifts the control plane and data plane directly into the Linux kernel. Rather than intercepting packets in userspace via proxy processes, eBPF allows developers to execute sandboxed, bytecode programs directly inside the kernel execution hooks (tracepoints, kprobes, cgroups, and TC / Traffic Control filters).

[ Application Pod ]
       |
       | Socket Write (e.g., sys_sendmsg)
       v
+---------------------------------------------------------------+
| Linux Kernel                                                  |
|  +---------------------------------------------------------+  |
|  | eBPF Socket Program (sockmap / sk_msg)                  |  |
|  | -> Bypasses entire TCP/IP stack for local pods          |  |
|  +----------------------------+----------------------------+  |
|                               |                               |
|                               v                               |
|  +---------------------------------------------------------+  |
|  | TC (Traffic Control) Ingress/Egress eBPF Program        |  |
|  | -> L3/L4 Security Policies & Identity Map Lookup        |  |
|  +----------------------------+----------------------------+  |
+------------------------------|--------------------------------+
                               v
                       [ Network Wire ]

Stack Bypass with `sockmap` and `sk_msg`

When two pods on the same worker node communicate under Cilium, traffic does not traverse iptables or the full host routing stack. Cilium attaches an eBPF program to the socket layer (sockmap).

When sys_sendmsg is invoked by the application, eBPF intercepts the socket buffer (sk_buff), reads its metadata, and writes it directly to the destination socket's receive queue (sk_msg). The standard TCP/IP stack traversal is bypassed completely, enabling sub-millisecond latencies that approach bare-metal IPC (Inter-Process Communication).

Identity-Based Microsegmentation (No More IP Firewalls)

Traditional network policies rely on IP addresses. In multi-cloud Kubernetes environments, IP addresses are dynamic, ephemeral, and fragmented across cloud boundaries (AWS VPC CNI, GCP Alias IPs, Azure CNI).

Cilium decouples network security policies from IP addresses completely. Instead, it assigns a Numeric Security Identity to pods based on their cryptographic state and Kubernetes metadata labels.

Label Extraction: Cilium Agent extracts pod metadata (e.g., app=payment-gateway, env=production).
Identity Allocation: The agent communicates with the cluster control plane (or CRD-backed KV store) to map this label set to a global Security Identity ID (e.g., ID 25413).
Kernel Map Sync: The identity map is pushed down to the host kernel's eBPF maps (BPF_MAP_TYPE_HASH).
Packet Encapsulation: When a packet leaves the veth pair, eBPF attaches the Security Identity metadata directly to the packet buffer (via Geneve/VXLAN overlay headers or native routing metadata).

When the receiving node's NIC gets the packet, an eBPF TC program extracts the source security ID, performs an eBPF hash lookup against its policy table, and accepts or drops the packet in a fraction of a microsecond—long before userspace memory allocations occur.

Implementing Zero-Trust Policies with Cilium

Cilium provides CiliumNetworkPolicy (and cluster-wide CiliumClusterwideNetworkPolicy) CRDs that extend native Kubernetes network policies with L3, L4, and L7 capabilities, including FQDN egress filtering and identity matching across multi-cloud boundaries.

Production-Grade `CiliumNetworkPolicy`

Below is an enterprise policy demonstrating strict zero-trust rules for a payment-service running in a multi-cloud environment:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-zero-trust
  namespace: finance
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
      env: production

  # Ingress Rules (L3/L4)
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api-gateway
            env: production
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

  # Egress Rules (L3/L4/L7 + Multi-Cloud Egress)
  egress:
    # Allow communication to internal database cluster across clusters
    - toEndpoints:
        - matchLabels:
            app: postgres-db
            io.cilium.k8s.policy.cluster: aws-us-east-1
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP

    # Allow L7 Restricted HTTP access to Stripe API via FQDN
    - toFQDNs:
        - matchName: "api.stripe.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
          rules:
            http:
              - method: "POST"
                path: "/v1/charges"
              - method: "GET"
                path: "/v1/charges/.*"

What Happens Under the Hood?

L3/L4 Enforcement: Executed entirely in the eBPF kernel space. No proxy overhead.
L7 Enforcement (rules.http): For L7 rules, Cilium dynamically steers only matching HTTP traffic to an optimized host-level Envoy instance (shared per node, rather than per pod), while keeping pure L3/L4 traffic strictly in kernel space.
FQDN Enforcement: Cilium monitors DNS responses via an eBPF DNS proxy hook, dynamic-binding target IPs of api.stripe.com to security identities on the fly.

Multi-Cloud Connectivity: Cilium ClusterMesh & Transparent WireGuard

Building high-throughput microsegmentation across multi-cloud Kubernetes clusters (e.g., EKS in AWS and GKE in GCP) usually requires complex VPN gateways, cloud-native interconnects, and heavy sidecar mesh configurations.

Cilium simplifies this through ClusterMesh combined with Transparent Kernel Encryption.

  +-----------------------------------+       +-----------------------------------+
  | AWS EKS (Cluster 1)               |       | GCP GKE (Cluster 2)               |
  | Pod A (Identity: 104)             |       | Pod B (Identity: 208)             |
  +-----------------+-----------------+       +-----------------+-----------------+
                    |                                           |
                    v                                           v
  +-----------------+-----------------+       +-----------------+-----------------+
  | eBPF Ingress/Egress TC Program    |       | eBPF Ingress/Egress TC Program    |
  | Encrypts with WireGuard (Kernel)  |       | Decrypts WireGuard in Kernel      |
  +-----------------+-----------------+       +-----------------+-----------------+
                    |                                           |
                    +========== WireGuard Tunnel (UDP 51820) ===+

Enabling Transparent WireGuard Encryption

Rather than terminating TLS inside a userspace sidecar (mTLS via Envoy), Cilium implements mTLS/encryption at the network layer using Linux kernel module WireGuard or IPsec.

To enable zero-overhead cross-cluster encryption, update your Cilium Helm configuration:

# Helm values override for Cilium multi-cloud deployment
clustermesh:
  useAPIServer: true
  config:
    enabled: true
    clusters:
      - name: aws-us-east-1
        address: 10.100.0.10
        port: 2379
      - name: gcp-europe-west1
        address: 10.200.0.10
        port: 2379

encryption:
  enabled: true
  type: wireguard
  wireguard:
    userspaceFallback: false # Enforce pure Linux Kernel WireGuard module

When Pod A in AWS calls Pod B in GCP:

The eBPF egress program determines that the destination endpoint belongs to remote cluster gcp-europe-west1.
It attaches the Identity tag and hands the packet directly to the kernel WireGuard interface (cilium_wg0).
Encryption happens via native assembly-optimized kernel crypto instructions (AVX-512).
The packet travels across public/private transit over UDP port 51820 without proxy-induced packet drops or context switching.

Sub-Millisecond Observability with Hubble

Observability in sidecar meshes requires tracing headers (x-request-id, x-b3-traceid) to be parsed and forwarded by every proxy, eating CPU cycles and bloating payload sizes.

Cilium leverages Hubble, an observability platform built natively on eBPF. Because eBPF sits at the kernel layer, Hubble captures network flows, HTTP requests, gRPC calls, and drop reasons with zero modifications to application code.

Inspecting Microsegmentation in Real-Time

Using the Hubble CLI, security architects can observe kernel-level policy drops live:

# Stream live network flows across all clusters filtered by policy drops
hubble observe --type drop --follow \
  --namespace finance \
  --label app=payment-service

Output:

TIMESTAMP             SOURCE                           DESTINATION                      TYPE          VERDICT
Oct 24 14:02:11.102   finance/payment-service-x8f2     prod-db/postgres-0:5432          to-endpoint   FORWARDED (Cipher: WireGuard)
Oct 24 14:02:15.891   untrusted/malicious-pod-9bc1     finance/payment-service:8080     to-endpoint   DROPPED (Policy denied: identity 10921 -> 25413)

You can inspect BPF maps directly on worker nodes using cilium CLI tools to verify data-plane policies loaded in the Linux kernel:

# View active security identity mappings inside BPF maps
cilium bpf policy get --endpoint 1422

Sidecar vs. eBPF Performance Benchmark

Key Takeaways for Cloud Platform Teams

Reclaim Infrastructure Budget: Eliminating sidecar containers frees up to 20-30% of compute capacity across enterprise Kubernetes clusters.
True Zero-Trust Security: Microsegmentation moves from the application level down to the Linux kernel. Even if a pod container is fully compromised at root level, kernel eBPF hooks prevent the execution of raw sockets or unauthorized lateral movements.
Multi-Cloud Fabric: Cilium ClusterMesh seamlessly unites heterogeneous multi-cloud deployments under a single identity engine without requiring complex edge API gateways.

Transitioning from legacy sidecar meshes to eBPF-native security with Cilium represents a fundamental evolution in cloud-native platform engineering. By executing security at the kernel layer, platform architects no longer need to compromise on performance to achieve strict microsegmentation and compliance across multi-cloud environments.