Beyond Sidecars: Building Zero-Overhead eBPF Security Policies for Multi-Tenant Kubernetes | Ecstaticloud Blog

For years, the sidecar pattern has been the uncontested standard for managing service mesh security, observability, and traffic routing in Kubernetes. By injecting a proxy container—most notably Envoy—alongside every application container, platforms achieved fine-grained Mutual TLS (mTLS) and Layer 7 policy enforcement.

However, operating sidecars at scale introduces a steep toll: the "sidecar tax." In high-throughput, multi-tenant clusters, duplicating proxies across thousands of pods yields massive CPU/memory overhead and inflates tail latencies (p99) due to repetitive network stack traversals and context switches.

Enter eBPF (Extended Berkeley Packet Filter). By executing sandboxed, bytecode programs directly within the Linux kernel, eBPF allows us to enforce zero-trust security policies, perform Layer 7 routing, and gather deep runtime telemetry without modifying pods or injecting sidecar containers.

In this post, we will explore how replacing sidecars with eBPF-native architectures eliminates latency overhead, walk through multi-tenant security design using Cilium, and construct a custom eBPF kernel program to intercept unauthorized system calls.

The Sidecar Tax vs. Kernel-Space Datapaths

To understand why eBPF is revolutionary, we must dissect the packet path of a traditional sidecar architecture.

The Sidecar Packet Path

When Pod A communicates with Pod B via an Envoy sidecar:

The application writes data to a socket.
Traffic hits iptables rules (PREROUTING/OUTPUT) inside the pod's network namespace.
The kernel redirects packets via loopback to the Envoy proxy user-space process.
Envoy processes L7 rules, terminates TLS, and writes to a new socket.
The packet travels through the node's TCP/IP stack, down to the virtual Ethernet (veth) pair.
The host routes the packet to Pod B's veth interface.
Pod B’s iptables again redirects traffic to Pod B's Envoy proxy.
Envoy processes the packet and finally forwards it to Application B via loopback.

[ Application A ] ──(socket)──> [ iptables ] ──(loopback)──> [ Envoy Proxy A ]
                                                                    │
                                                            (veth pair / host bridge)
                                                                    │
                                                                    ▼
[ Application B ] <──(loopback)── [ Envoy Proxy B ] <──(iptables) ──┘

This model incurs two user-to-kernel-space context switches per proxy and forces packets through the Linux networking stack eight times.

The eBPF Datapath

eBPF attaches programs directly to network hooks such as XDP (eXpress Data Path), TC (Traffic Control), and cgroup/socket layer maps (sockmap).

When Pod A sends a packet using an eBPF-based service mesh (like Cilium):

The application writes to the socket.
An eBPF program attached at the socket layer (sockmap) bypasses the host TCP/IP stack entirely if destination sockets reside on the same node, copying data directly from socket buffer to socket buffer (sk_msg).
For cross-node traffic, an eBPF program at the TC layer handles encapsulation, security policy validation, and routing inline within the kernel.

[ Application A ] ──(socket)──> [ eBPF BPF_MAP_TYPE_SOCKMAP ] ──> [ Application B Socket ]
                                              │
                                   (Cross-node encapsulation)
                                              │
                                              ▼
                                     [ WireGuard / IPsec ]

By eliminating user-space proxies, context switches, and network stack traversals, eBPF reduces network latency to near-native hardware speed.

Multi-Tenant Security with Cilium Network Policies

In a multi-tenant Kubernetes environment, strong isolation at Layer 3, Layer 4, and Layer 7 is non-negotiable. Cilium abstracts eBPF into custom resources like CiliumNetworkPolicy (CNP) and CiliumClusterwideNetworkPolicy (CCNP).

Unlike standard Kubernetes NetworkPolicies—which rely on iptables or IPVS and scale poorly ($O(N)$ linear inspection complexity)—Cilium compiles policies into eBPF BPF maps ($O(1)$ hash map lookups).

Production Example: Hardening Tenant Isolation and L7 Traffic

Consider a multi-tenant application where Tenant alpha's payment service must only communicate with the shared billing API over HTTPS, restricted strictly to POST /v1/charge endpoints.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: tenant-alpha-payment-isolation
  namespace: tenant-alpha
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
      tenant: alpha
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api-gateway
            tenant: alpha
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
  egress:
    # Restrict egress to the billing service in the management namespace
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: platform-services
            app: billing-api
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
          rules:
            http:
              - method: "POST"
                path: "/v1/charge.*"
    # Allow DNS resolution via CoreDNS explicitly
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*.platform.internal"

What Happens Under the Hood?

L3/L4 Filtering: Processed via eBPF programs loaded at the tc (Traffic Control) ingress/egress hooks. If an incoming packet's security identity (derived from pod labels and managed centrally) isn't in the BPF map, the packet is immediately dropped in the kernel before memory allocation.
L7 Filtering: If L7 rules are specified, Cilium dynamically diverts only the matching flows to an optimized, host-level Envoy instance managed by Cilium. Non-L7 traffic bypasses Envoy completely.

Deep-Dive: Building a Custom eBPF Kernel Tracepoint for Runtime Security

While network policy enforcement secures the wire, zero-trust requires verifying runtime behavior inside the container context. Traditional sidecars cannot monitor malicious process executions, unauthorized file access, or container escapes.

Using libbpf and C, we can write an eBPF program that hooks into the kernel’s sys_enter_execve tracepoint to detect unauthorized process execution inside our high-throughput Kubernetes pods.

1. The Kernel-Side eBPF Program (`exec_monitor.bpf.c`)

// +build ignore

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

#define TASK_COMM_LEN 16

struct event {
    u32 pid;
    u32 uid;
    u32 container_id;
    char comm[TASK_COMM_LEN];
    char filename[256];
};

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(max_entries, 1024);
} events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_execve")
int tracepoint_syscalls_sys_enter_execve(struct trace_event_raw_sys_enter *ctx) {
    u64 id = bpf_get_current_pid_tgid();
    u32 pid = id >> 32;
    u32 uid = bpf_get_current_uid_gid();

    struct event event_data = {};
    event_data.pid = pid;
    event_data.uid = uid;

    // Get current process name
    bpf_get_current_comm(&event_data.comm, sizeof(event_data.comm));

    // Extract file path executed from tracepoint arguments
    const char *filename_ptr = (const char *)ctx->args[0];
    bpf_probe_read_user_str(&event_data.filename, sizeof(event_data.filename), filename_ptr);

    // Send security event payload to user-space ring buffer / perf map
    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU, &event_data, sizeof(event_data));

    return 0;
}

char _license[] SEC("license") = "GPL";

2. Compiling and Loading the Program

Compile the eBPF code to BPF bytecode target using clang:

clang -O2 -g -target bpf -D__TARGET_ARCH_x86 -c exec_monitor.bpf.c -o exec_monitor.bpf.o

When loaded into the Linux kernel via bpftool or a Go-based manager (like cilium/ebpf), this code executes every time a process calls execve(). It captures the PID, UID, calling command, and executable path at zero copy speed, streaming alerts to security agents without touching container user-space.

Architectural Benchmarks: Sidecars vs. eBPF

To illustrate the real-world impact of migrating from an Envoy sidecar implementation to eBPF-native enforcement, consider these metrics gathered from a multi-tenant test cluster running 1,000 microservice replicas handling 50,000 HTTP requests per second:

Performance Tuning Strategies for High-Throughput eBPF Clusters

To realize the true zero-overhead potential of eBPF in production, apply the following kernel-level optimizations:

1. Enable eBPF Host-Reachable Services

By default, Kubernetes cluster traffic hits kube-proxy iptables rules. Cilium can completely replace kube-proxy by attaching eBPF programs directly to socket layer hooks (connect, sendmsg).

Configure Cilium's Helm chart:

kubeProxyReplacement: true
k8sServiceHost: API_SERVER_IP
k8sServicePort: API_SERVER_PORT
ebpf:
  masquerade: true

2. Leverage Native XDP Mode

If your NIC drivers support XDP (eXpress Data Path) (e.g., ixgbe, i40e, mlx5), instruct Cilium to execute packet filtering directly on the Network Interface Card (NIC) driver ring buffer before allocating a kernel sk_buff.

bpf:
  xdpMode: "native"

This guarantees that DDoS attacks or unauthorized cross-tenant packet floods are dropped before consuming system CPU cycles.

Key Takeaways

Eliminate Sidecar Bloat: Transferring network policies, observability, and routing into eBPF kernel hooks drastically lowers p99 latency and reclaims compute resources previously eaten by Envoy instances.
Deterministic Scale: BPF map lookups execute in $O(1)$ constant time, providing stable network enforcement performance regardless of whether your cluster has 10 or 10,000 security rules.
Unified Security Surface: Combining Layer 3/4/7 network policies via Cilium with custom eBPF system call tracepoints yields a robust, zero-trust posture across both network traffic and host execution.

By shifting your cloud-native security model from user-space sidecars down to kernel-space eBPF programs, you build a foundation optimized for high throughput, absolute multi-tenant isolation, and minimal operational complexity.