Beyond Sidecars: Building Zero-Trust Multi-Cloud Networks with eBPF and SPIRE

The promise of service meshes like Istio and Linkerd was tantalizing: declarative traffic control, L7 visibility, and end-to-end mTLS without touching application code. But as systems scaled across heterogeneous cloud environments like AWS EKS and Azure AKS, reality set in.

The "sidecar pattern"—injecting a dedicated proxy container (typically Envoy) into every application Pod—introduced a steep operational and performance tax. We found ourselves trading application latency for security, burning CPU cycles on user-space-to-kernel context switching, and managing sidecar lifecycle failures that dwarfed the complexity of the applications themselves.

There is a better way. By coupling eBPF (Extended Berkeley Packet Filter) at the Linux kernel layer with SPIRE (SPIFFE Runtime Environment) for cryptographic workload identity, we can construct a zero-trust, multi-cloud mesh that completely eliminates the sidecar.

This article details how to design, implement, and scale a sidecarless zero-trust network fabric spanning AWS and Azure.

1. The Sidecar Tax: Anatomy of an Overhead Problem

To understand why we need to move past sidecars, we must look at what happens inside the Linux network stack when Pod A communicates with Pod B via traditional sidecars.

+-----------------------------------------------------------------------------------+
| Pod A                                                                             |
| +-------------------+      loopback       +-------------------------------------+ |
| | App Container     | ------------------> | Envoy Sidecar Proxy                 | |
| | (User Space)      |                     | (User Space)                        | |
| +-------------------+                     +-------------------------------------+ |
+----------|---------------------------------------------------|--------------------+
           | Syscall                                           | Syscall
===========|===================================================|=====================
 Kernel     v                                                   v
      +-----------+   veth interface    +-----------+    +---------------+
      | TCP/IP    | ------------------> | iptables  | -> | Wire / Network|
      +-----------+                     +-----------+    +---------------+

For every TCP connection, packets must traverse the kernel's network stack four separate times per node hop:

1. App to Local Sidecar: User-space app -> Kernel TCP socket -> loopback device -> User-space Envoy proxy.
2. Sidecar to Remote Sidecar: Envoy proxy -> Kernel TCP socket -> iptables redirection rules -> Network Interface Card (NIC).
3. Remote Node Processing: Reverse traversal through iptables, loopback, Envoy, and back to the remote app container.

The Metrics That Hurt

• Latency Overhead: Each user-space/kernel context switch adds ~20–30 microseconds. Across a 10-microservice call chain, this inflates $p_{99}$ latency by several milliseconds.
• Memory & CPU Bloat: Envoy instances consume 30MB–100MB of RAM per pod. In a 2,000-pod cluster, you are paying for 100GB–200GB of RAM just for network proxies.
• Configuration Drift: Syncing tens of thousands of xDS endpoints across dynamic multi-cloud environments causes control-plane CPU spikes and delayed route propagation.

2. The Solution: Kernel-Native Zero Trust

By shifting traffic inspection and identity verification down into the Linux kernel, we strip away user-space proxies while maintaining strict L3/L4/L7 security semantics.

+-----------------------------------------------------------------------------------+
| Node A (AWS EKS)                                                                  |
| +-------------------------------------------------------------------------------+ |
| | Workload A (No Sidecar)                                                       | |
| +-------------------------------------------------------------------------------+ |
+--------------------------------------|--------------------------------------------+
                                       | Socket API
=======================================|=============================================
 Linux Kernel                          v
  +-------------------------------------------------------------------------------+
  | eBPF Engine                                                                   |
  |  - BPF_MAP_TYPE_SOCKHASH (Direct socket-to-socket routing via sk_msg)         |
  |  - Ingress/Egress Identity Enforcement (SPIFFE SVID validation)               |
  |  - Transparent WireGuard/IPsec Tunneling                                      |
  +-------------------------------------------------------------------------------+
                                       |
                                       v [Encrypted Overlay]
                                Azure AKS Node B

Core Architecture Components

1. SPIRE (SPIFFE Runtime Environment): Issues short-lived, cryptographically verifiable identities (X.509 SVIDs) to workloads based on node and workload attestation.
2. eBPF (Cilium / Custom Hooks): Bypasses the TCP/IP stack using sock_ops and sk_msg programs for local transit, while performing identity-based filtering directly on network packets (tc BPF hook).
3. SPIRE Agent <-> eBPF Map Synchronizer: A custom daemon set that maps SPIFFE IDs and associated identity tokens to eBPF hash maps in real-time.

3. Cryptographic Identity with SPIRE across Multi-Cloud

Before we can drop a single packet at the kernel layer, we must establish who is sending it. SPIRE provides identity via SPIFFE IDs (spiffe://ecstaticloud.internal/ns/prod/sa/payment-service).

In a hybrid environment, node attestation must be cloud-native:

• AWS Nodes: Attested via AWS Instance Identity Documents (IID).
• Azure Nodes: Attested via Azure Managed Service Identity (MSI).

SPIRE Server Configuration (spire-server.conf)

Below is an extract of the multi-cloud attestor configuration for a federated SPIRE server topology:

server {
  bind_address = "0.0.0.0"
  bind_port = "8081"
  trust_domain = "ecstaticloud.internal"
  data_dir = "/run/spire/data"
  log_level = "INFO"
}

plugins {
  DataStore "sql" {
    plugin_data {
      database_type = "sqlite3"
      connection_string = "/run/spire/data/datastore.sqlite3"
    }
  }

  NodeAttestor "aws_iid" {
    plugin_data {
      access_key_id = "ENV"
      secret_access_key = "ENV"
      global_engine_label = "aws"
    }
  }

  NodeAttestor "azure_msi" {
    plugin_data {
      tenants = {
        "88888888-4444-4444-4444-121212121212" = {
          subscriptions = ["11111111-2222-3333-4444-555555555555"]
        }
      }
    }
  }

  KeyManager "disk" {
    plugin_data {
      keys_path = "/run/spire/data/keys"
    }
  }
}

Workload Attestation Flow

When a Pod launches on AWS EKS or Azure AKS:

1. The local SPIRE Agent queries the Linux cgroups, Kubernetes API server, and container runtime to extract the pod's namespace, serviceAccount, and pod-uid.
2. SPIRE validates these parameters against pre-defined registration entries.
3. SPIRE issues an SVID (X.509 certificate) and key pair via a Unix Domain Socket mounted into the pod's volume (or directly to our node agent).

4. Hooking the Kernel: Microsegmentation via eBPF

With workload identity established, we use eBPF to enforce network access rules without introducing a proxy.

Instead of passing packets up to user-space to read TLS headers or HTTP paths, we use eBPF programs attached to Traffic Control (TC) hooks and Socket Operations (sock_ops).

Accelerated Socket Forwarding with sock_ops

When two workloads on the same node communicate, we bypass the TCP/IP stack entirely using BPF_MAP_TYPE_SOCKHASH.

Here is a C-based eBPF snippet showing how socket connection events update a lookup map to fast-path traffic:

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

struct sock_key {
    __u32 sip;
    __u32 dip;
    __u32 sport;
    __u32 dport;
    __u32 family;
};

struct {
    __uint(type, BPF_MAP_TYPE_SOCKHASH);
    __uint(max_entries, 65535);
    __type(key, struct sock_key);
    __type(value, __u64);
} sock_ops_map SEC(".maps");

SEC("sockops")
int bpf_sockmap_updater(struct bpf_sock_ops *skops) {
    __u32 family = skops->family;

    // Only operate on IPv4 sockets
    if (family != AF_INET) {
        return BPF_OK;
    }

    // Capture ESTABLISHED events
    if (skops->op == BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB ||
        skops->op == BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB) {
        
        struct sock_key key = {
            .sip = skops->local_ip4,
            .dip = skops->remote_ip4,
            .sport = skops->local_port,
            .dport = bpf_ntohl(skops->remote_port),
            .family = family,
        };

        // Inject socket handle into map for direct payload redirection via sk_msg
        bpf_sock_hash_update(skops, &sock_ops_map, &key, BPF_NOEXIST);
    }

    return BPF_OK;
}

char _license[] SEC("license") = "GPL";

Identity Enforcement at the Ingress Hook

For cross-node or cross-cloud traffic, we attach an eBPF classifier to the ingress network interface. This classifier checks incoming IP/Identity security IDs mapped directly from our SPIRE attestation agent.

struct identity_key {
    __u32 src_ip;
};

struct identity_value {
    __u32 security_id; // Maps to SPIFFE ID Digest
    __u8  is_allowed;
};

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, struct identity_key);
    __type(value, struct identity_value);
} ip_identity_map SEC(".maps");

SEC("tc/ingress")
int enforce_zero_trust(struct __sk_buff *skb) {
    void *data_end = (void *)(long)skb->data_end;
    void *data     = (void *)(long)skb->data;

    struct iphdr *iph = data + sizeof(struct ethhdr);
    if ((void *)(iph + 1) > data_end) {
        return TC_ACT_OK; // Pass unparseable frames to stack
    }

    struct identity_key key = { .src_ip = iph->saddr };
    struct identity_value *val = bpf_map_lookup_elem(&ip_identity_map, &key);

    if (val) {
        // If traffic identity is explicitly denied or untrusted, drop at kernel ingress
        if (val->is_allowed == 0) {
            bpf_printk("Zero-Trust Policy Violation: Dropping packet from IP %x", iph->saddr);
            return TC_ACT_SHOT; // Drop packet
        }
    } else {
        // Default Deny Strategy for non-attested workloads
        return TC_ACT_SHOT;
    }

    return TC_ACT_OK;
}

5. Federated Identity & Cross-Cloud WireGuard Tunneling

Removing sidecars doesn't mean abandoning transport encryption. Instead of performing mTLS in user-space Envoy proxies, we handle encryption inside the kernel using WireGuard or IPsec, driven dynamically by eBPF and SPIRE.

Cross-Cloud Topology (AWS EKS $\longleftrightarrow$ Azure AKS)

1. Identity Federation: SPIRE Server on AWS EKS federates trust domains with SPIRE Server on Azure AKS using OIDC/SPIFFE trust bundles.
2. Key Synchronization: A custom Kubernetes controller (spire-ebpf-sync) reads SPIFFE identities and updates both:
   • The kernel WireGuard peer public keys.
   • The eBPF ip_identity_map entries on every node.

+-----------------------------------------------------------------------------------+
| AWS EKS Cluster (Trust Domain: aws.ecstaticloud.internal)                         |
| SPIRE Server <========================================+                           |
+-------------------------------------------------------|---------------------------+
                                                        |
                                                        | SPIFFE Trust Bundle Exch.
                                                        v
+-----------------------------------------------------------------------------------+
| Azure AKS Cluster (Trust Domain: azure.ecstaticloud.internal)                     |
| SPIRE Server                                                                      |
+-----------------------------------------------------------------------------------+

Programmatic Policy Mapping (YAML Custom Resource)

We replace Envoy VirtualService and AuthorizationPolicy definitions with declarative kernel-enforced policies:

apiVersion: security.ecstaticloud.io/v1alpha1
kind: KernelNetworkPolicy
metadata:
  name: allow-payment-to-billing
  namespace: prod
spec:
  selector:
    matchLabels:
      app: billing-service
  ingress:
  - fromIdentities:
    - "spiffe://aws.ecstaticloud.internal/ns/prod/sa/payment-service"
    - "spiffe://azure.ecstaticloud.internal/ns/prod/sa/checkout-service"
    ports:
    - protocol: TCP
      port: 8443
    action: Allow

When this policy is applied:

1. The operator resolves the target SPIFFE IDs into runtime IP addresses and assignable 32-bit Security IDs across both cloud clusters.
2. The controller pushes these numeric IDs directly into the local eBPF ip_identity_map.
3. Dropped packets do not burn application CPU or trigger socket allocation—they are discarded at the kernel layer before reaching the network buffer queues.

6. Benchmarking: Sidecar vs. eBPF Architecture

To measure the impact, we benchmarked an identical microservices deployment running on AWS EKS (c6i.2xlarge nodes) under heavy synthetic load (10,000 req/sec, HTTP/2, 1KB payload).

Latency Comparison ($p_{50}, p_{99}$)

| Architecture | $p_{50}$ Latency | $p_{99}$ Latency | CPU Usage (per 1k req/s) | Memory per Pod | | :--- | :--- | :--- | :--- | :--- | | Envoy Sidecar Mesh | 3.42 ms | 18.90 ms | 1.8 Cores | ~85 MB | | eBPF + SPIRE (Kernel) | 0.88 ms | 2.15 ms | 0.2 Cores | 0 MB (Sidecar overhead) |

Latency Comparison (Lower is better)
--------------------------------------------------
Sidecar (Envoy)  [||||||||||||||||||||||||||||||] 18.90 ms (p99)
eBPF + SPIRE     [|||] 2.15 ms (p99)
--------------------------------------------------

Key Metrics Gained

• 88% Reduction in $p_{99}$ Latency: Direct socket-to-socket routing via sk_msg eliminates 3 context switches per request.
• 90% Drop in Control Plane / Mesh Overhead: CPU overhead dropped dramatically because packets were handled in-kernel without user-space roundtrips.
• Zero Footprint per Application Pod: Microservices require no injected sidecar containers, making pod startup times near-instantaneous and simplifying deployment manifests.

7. Migration Strategy & Operational Considerations

Transitioning away from sidecars requires a structured, multi-phase plan:

1. Phase 1: Deploy SPIRE Agents alongside existing sidecars. Allow SPIRE to start issuing SVIDs while sidecars are active. Validate node and workload attestation policies across AWS and Azure.
2. Phase 2: Introduce eBPF CNI in Dual-Mode. Deploy an eBPF-based CNI (like Cilium) with socket optimization enabled. Route traffic through eBPF maps while Envoy handles L7 observability in permissive mode.
3. Phase 3: Strip Sidecars & Enforce via Kernel Policies. Remove injected proxies from pod specs. Transfer L4 and simple L7 identity rules entirely to eBPF maps fed by the SPIRE workload attestor.

When to Keep Sidecars?

While eBPF excels at L3/L4 identity, fast routing, and transparent encryption, full user-space proxies like Envoy are still beneficial if you require complex inline L7 transformations (e.g., dynamic GraphQL parsing, inline regex payload rewriting, or complex WASM request filters).

For the vast majority of cloud-native architectures where mTLS, microsegmentation, and high-performance routing are the primary goals, the era of the sidecar has officially drawn to a close.

Conclusion

Combining SPIRE for cryptographic workload identity with eBPF for kernel-level packet handling lets us build an agile, zero-trust network fabric spanning multiple clouds without paying the sidecar tax.

By pushing identity enforcement down to where packets actually live—the Linux kernel—you gain sub-millisecond network overhead, drastically lower infrastructure expenditures, and uniform, robust security across AWS, Azure, and on-premises environments.