Ecstaticloud
Initializing0%
Ecstaticloud Logo
Back to Insights
Cloud & DevOpsSeptember 14, 2026

eBPF-Driven FinOps: Real-Time GPU Cost Optimization for LLM Inference on Kubernetes

Discover how to leverage kernel-level eBPF probes to track precise per-tenant GPU telemetry for dynamic AI workloads on Kubernetes. Learn to combine deep system observability with custom autoscaling strategies to cut cloud inference costs by up to 40% without sacrificing tail latency.

Running Large Language Model (LLM) inference at enterprise scale presents a frustrating paradox for Cloud Architects: while business units demand sub-millisecond time-to-first-token (TTFT) and high throughput, finance teams are left reeling from the eye-watering cloud bills driven by idle, over-provisioned NVIDIA A100 and H100 clusters.

In typical Kubernetes deployment patterns, inference engines like vLLM, TensorRT-LLM, or TGI are provisioned with static GPU allocations. Because standard Kubernetes metrics and native NVIDIA Data Center GPU Manager (DCGM) telemetry operate largely out-of-band—polling metrics at coarse intervals without context on container cgroups—attributing GPU usage to specific tenants, pods, or individual inference requests has historically been an exercise in estimation.

The result? Massive over-provisioning, poor bin-packing, dynamic workload interference, and zero granular cost visibility.

To break this zero-sum game between tail-latency SLAs and cloud expenditures, we must look below the application abstraction layer. By deploying Extended Berkeley Packet Filter (eBPF) probes directly into the Linux kernel, we can capture microsecond-level GPU driver events, correlate them to container cgroup v2 boundaries, and build a real-time telemetry pipeline. This telemetry feeds directly into custom Kubernetes autoscalers—slashing GPU infrastructure costs by up to 40% while preserving strict tail-latency SLAs.

The Architectural Blind Spot: Why Standard Telemetry Fails FinOps

Traditional Kubernetes GPU monitoring relies on the NVIDIA DCGM Exporter pushing metrics to Prometheus. While DCGM provides essential health indicators (such as overall GPU temperature, power draw, and aggregate Streaming Multiprocessor (SM) utilization), it falls short for modern dynamic AI workloads for three reasons:

+---------------------------------------------------------------------------------+
|                                 KUBERNETES NODE                                 |
|                                                                                 |
|  +------------------------+                        +------------------------+  |
|  | Container A (Tenant X) |                        | Container B (Tenant Y) |  |
|  |  [vLLM / PyTorch]      |                        |  [vLLM / PyTorch]      |  |
|  +-----------+------------+                        +-----------+------------+  |
|              |                                                 |                |
|              | User-space CUDA API                             | User-space     |
|              v                                                 v                |
|      /dev/nvidiactl (ioctl)                             /dev/nvidiactl (ioctl)  |
|--------------|-------------------------------------------------|----------------|
| KERNEL SPACE |                                                 |                |
|              +------------------------+------------------------+                |
|                                       v                                         |
|                           NVIDIA Kernel Module (nvidia.ko)                      |
|                                       |                                         |
|   Traditional Telemetry (DCGM)         | eBPF Kernel Probes (Our Approach)       |
|   - Polled every 1s-10s               | - Sub-millisecond event interception    |
|   - Blind to cgroup context           | - Reads cgroup_id via bpf_get_current   |
|   - Aggregate metrics only            | - Maps CUDA launches to Pod/Tenant ID   |
+---------------------------------------+-----------------------------------------+
  1. Coarse Polling Latency vs. Microsecond Kernel Execution: DCGM typically scrapes metrics every 1 to 10 seconds. LLM inference kernels, memory transfers, and PagedAttention allocations execute on timescales of microseconds to milliseconds. Aggregated metrics smooth over micro-bursts, masking transient bottlenecks and hiding idle execution gaps.
  2. Loss of Cgroup & Namespace Context: When using NVIDIA Multi-Process Service (MPS) or shared GPU configurations across multiple pods on a node, DCGM reports metrics at the physical GPU device level. It cannot natively correlate which specific CUDA kernel launch, ioctl syscall, or VRAM buffer belongs to which Kubernetes Pod, Namespace, or Tenant ID.
  3. Temporal vs. Spatial Multiplexing Oversights: VRAM is often locked continuously by inference runtimes (e.g., PyTorch pre-allocating memory pools or vLLM reserving KV-cache blocks), even when compute engines (SMs) sit idle waiting for incoming requests. Standard metrics report high VRAM usage as continuous "resource consumption," preventing autoscalers from detecting that compute resources are actually available for dynamic workloads.

Building the eBPF Telemetry Pipeline for CUDA/GPU Drivers

To gain true visibility, we need to trace the interaction between user-space CUDA runtimes (libcuda.so, libcudart.so) and the host kernel's NVIDIA driver module (nvidia.ko / nvidia-uvm.ko).

When a container issues a command to the GPU, it routes requests via ioctl system calls to system character devices like /dev/nvidiactl, /dev/nvidia0, or /dev/nvidia-uvm.

By placing eBPF tracepoints and kprobes on system calls (sys_enter_ioctl and sys_exit_ioctl) alongside uprobes on key user-space CUDA functions, we extract real-time telemetry enriched with Linux kernel cgroup identifiers.

The Kernel-to-Container Correlation Technique

Every Kubernetes pod running on a node with cgroup v2 enabled maps to a distinct cgroup ID in the kernel directory tree (/sys/fs/cgroup/...). In our eBPF C program, whenever an ioctl call targeting an NVIDIA device file is intercepted, we query the current cgroup context using bpf_get_current_cgroup_id().

Here is a simplified eBPF C program (compiled with BPF CO-RE) that intercepts ioctl system calls to compute execution duration and map it directly to container cgroup IDs:

// gpu_telemetry.bpf.c
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

#define NV_IOCTL_MAGIC 'F'

struct gpu_event_t {
    u64 cgroup_id;
    u32 pid;
    u32 ioctl_cmd;
    u64 duration_ns;
    u64 timestamp;
};

// Map to store system call entry timestamps keyed by thread ID
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);   // tid
    __type(value, u64); // start timestamp (ns)
} start_time_map SEC(".maps");

// Ring buffer to transmit event payloads to the user-space daemon
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024); // 256 KB
} gpu_events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_ioctl")
int trace_sys_enter_ioctl(struct trace_event_raw_sys_enter *ctx) {
    u32 tid = (u32)bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    
    // Filter for relevant ioctl calls targetting NVIDIA device driver handles
    // ctx->args[1] holds the ioctl command code
    unsigned int cmd = (unsigned int)ctx->args[1];
    
    // Store start timestamp indexed by thread ID
    bpf_map_update_elem(&start_time_map, &tid, &ts, BPF_ANY);
    return 0;
}

SEC("tracepoint/syscalls/sys_exit_ioctl")
int trace_sys_exit_ioctl(struct trace_event_raw_sys_exit *ctx) {
    u32 tid = (u32)bpf_get_current_pid_tgid();
    u64 *start_ts = bpf_map_lookup_elem(&start_time_map, &tid);
    
    if (!start_ts) {
        return 0; // Missed entry event
    }

    u64 delta = bpf_ktime_get_ns() - *start_ts;
    bpf_map_delete_elem(&start_time_map, &tid);

    // Only process calls with measurable execution duration
    if (delta < 5000) { // < 5 microseconds
        return 0;
    }

    struct gpu_event_t *event = bpf_ringbuf_reserve(&gpu_events, sizeof(struct gpu_event_t), 0);
    if (!event) {
        return 0;
    }

    event->cgroup_id = bpf_get_current_cgroup_id();
    event->pid = tid;
    event->duration_ns = delta;
    event->timestamp = bpf_ktime_get_ns();

    bpf_ringbuf_submit(event, 0);
    return 0;
}

char _license[] SEC("license") = "GPL";

User-Space Mapping Daemon

A lightweight user-space daemon (written in Go using cilium/ebpf) reads events from the ring buffer. It maps cgroup_id back to the active Pod name, Namespace, and Tenant metadata by querying the local container runtime (containerd.sock) interface map.

// main.go (Snippet - Processing eBPF Ring Buffer)
package main

import (
	"bytes"
	"encoding/binary"
	"fmt"
	"log"
	"os"
	"github.com/cilium/ebpf/ringbuf"
)

type GPUEvent struct {
	CgroupID   uint64
	PID        uint32
	IoctlCmd   uint32
	DurationNs uint64
	Timestamp  uint64
}

func processEvents(rd *ringbuf.Reader, cgroupResolver *CgroupResolver) {
	for {
		record, err := rd.Read()
		if err != nil {
			if err == ringbuf.ErrClosed {
				return
			}
			log.Printf("Error reading ringbuf: %v", err)
			continue
		}

		var event GPUEvent
		err = binary.Read(bytes.NewReader(record.RawSample), binary.LittleEndian, &event)
		if err != nil {
			log.Printf("Failed to parse event struct: %v", err)
			continue
		}

		// Resolve cgroup ID to Kubernetes Pod Context
		podInfo, found := cgroupResolver.Lookup(event.CgroupID)
		if !found {
			continue
		}

		// Update Prometheus counters with sub-millisecond accuracy
		gpuComputeTimeSeconds.WithLabelValues(
			podInfo.Namespace,
			podInfo.PodName,
			podInfo.TenantID,
			podInfo.ModelName,
		).Add(float64(event.DurationNs) / 1e9)
	}
}

Per-Tenant Cost Attribution Model

With microsecond-accurate data tied to Kubernetes Pods, we can replace simple hourly GPU node splitting with a unified time-space cost allocation model.

We compute total GPU cost for a tenant ($C_{tenant}$) over a given time frame $T$ using three dynamic vectors:

  1. SM Compute Duration Time ($T_{SM}$)
  2. Reserved VRAM Occupancy ($M_{VRAM}$)
  3. Host-to-Device Memory Transfer Overhead ($B_{PCIe}$)

$$\text{Cost}{\text{tenant}} = \int{0}^{T} \left( w_1 \cdot \frac{\text{SM}{\text{active}}(t)}{\text{SM}{\text{total}}} + w_2 \cdot \frac{\text{VRAM}{\text{reserved}}(t)}{\text{VRAM}{\text{capacity}}} + w_3 \cdot \frac{\text{Bytes}{\text{PCIe}}(t)}{\text{Bandwidth}{\text{max}}} \right) \times \text{Rate}_{\text{node_hourly}} , dt$$

Where:

  • $w_1, w_2, w_3$ are weighted vectors summing to 1.0 (typically configured as $w_1 = 0.55$, $w_2 = 0.35$, $w_3 = 0.10$).
  • $\text{SM}_{\text{active}}(t)$ is derived directly from our eBPF trace points capturing ioctl kernel launch schedules per cgroup.
  • $\text{VRAM}_{\text{reserved}}(t)$ captures true allocated state extracted via uprobes on cudaMalloc and cudaFree.

Cost Engine Metric Aggregation Blueprint

+-------------------+      +-------------------+      +--------------------+
|  eBPF Kernel      |      | Container Runtime |      | Prometheus Metric  |
|  Driver Events    |      | (cgroup v2 Map)   |      | Aggregator         |
+---------+---------+      +---------+---------+      +---------+----------+
          |                          |                          |
          | (Duration, Cmd)          | (Pod, Tenant, Namespace) |
          +------------+-------------+                          |
                       |                                        |
                       v                                        |
          +-------------------------+                           |
          | User-Space Node Agent   |---------------------------+
          +-------------------------+
                                                                | Exposes Metrics
                                                                v
                                                      +-------------------+
                                                      | FinOps Vector Engine|
                                                      | & KEDA Scaler     |
                                                      +-------------------+

Closed-Loop Dynamic Autoscaling with KEDA & eBPF Metrics

Capturing deep cost metrics is only half the battle. To actively slash costs, telemetry must feed back into the Kubernetes scheduling loop to drive aggressive, real-time autoscaling.

Standard Horizontal Pod Autoscalers (HPA) rely on CPU/Memory or naive DCGM GPU utilization metrics. This frequently causes scaling oscillation (flapping) during long-context LLM generation cycles, leading to poor bin-packing and unnecessary node provisioning.

By leveraging our eBPF metric provider, we expose a targeted metric to Kubernetes via KEDA (Kubernetes Event-driven Autoscaling): ebpf_gpu_active_compute_duty_cycle.

Scaling Architectural Logic

  1. Active Compute Duty Cycle $< 15%$ over 60s: The engine is idling (likely holding KV cache without processing tokens). Trigger scale-down or initiate fractional instance migration via NVIDIA MPS.
  2. Active Compute Duty Cycle $> 80%$ with Queue Pressure: Immediately scale out inference replicas.

KEDA ScaledObject Configuration

Here is a production-ready ScaledObject utilizing custom Prometheus metrics driven by our eBPF exporter:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama3-70b-autoscaler
  namespace: llm-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-70b-worker
  minReplicaCount: 1
  maxReplicaCount: 12
  cooldownPeriod: 300
  pollingInterval: 5
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 120
          policies:
          - type: Percent
            value: 25
            periodSeconds: 60
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: ebpf_gpu_active_compute_duty_cycle
      query: |
        sum(
          rate(ebpf_gpu_compute_time_seconds_total{namespace="llm-inference", app="vllm-llama3-70b"}[30s])
        ) by (app) 
        / 
        count(ebpf_gpu_compute_time_seconds_total{namespace="llm-inference", app="vllm-llama3-70b"})
      threshold: '0.75' # Scale up when active compute exceeds 75% across active replicas

Mitigating Cold-Starts: Fractional MPS Slicing & Hot Pools

Scale-to-zero for a 70B parameter model is traditionally unfeasible due to model loading overhead (tens of gigabytes transferred from NVMe to VRAM, taking 15–45 seconds).

To overcome this latency penalty while maximizing cost savings, our control plane uses eBPF metrics to dynamically adjust NVIDIA Multi-Process Service (MPS) allocation vectors instead of immediately destroying pods:

    High Load: Full GPU Provisioning       Low Load: Shared MPS Slicing
   +---------------------------------+   +---------------------------------+
   | Node 1 (NVIDIA H100)            |   | Node 1 (NVIDIA H100)            |
   | +-----------------------------+ |   | +---------------+---------------+ |
   | | Pod A (Tenant X)            | |   | | Pod A (25% MPS| Pod B (25% MPS| |
   | | Takes 100% Compute & Memory | |   | | Tenant X)     | Tenant Y)     | |
   | +-----------------------------+ |   | +---------------+---------------+ |
   |                                 |   | | Pod C (50% MPS Shared Warm)   | |
   +---------------------------------+   +---------------------------------+
  1. When eBPF tracepoints detect a tenant entering an idle phase, the control plane shifts the Pod's GPU allocation dynamically from an isolated physical GPU into a shared MPS Sliced Group (e.g., allocating 25% compute thread resources).
  2. The VRAM context stays resident in memory, eliminating cold-start model loads.
  3. Compute capacity is reclaimed for high-priority active requests on the same physical node.

Production Case Study & Results

To validate this eBPF-driven FinOps approach, we deployed the architecture to an Amazon EKS cluster running multi-tenant LLM inference workloads (mix of Llama-3-8B-Instruct and Llama-3-70B-Instruct models using vLLM on g5.12xlarge and p4d.24xlarge instances).

Workload Characteristics & Baseline

  • Baseline setup: Fixed-size deployments autoscaled via standard DCGM GPU utilization (DCGM_FI_DEV_GPU_UTIL) and HPA.
  • Traffic Pattern: Bursty enterprise API traffic with deep overnight troughs and high multi-tenant volatility.
  • SLA Constraint: Time-To-First-Token (TTFT) $P_{99} \le 45 \text{ ms}$; Token Generation $P_{99} \le 25 \text{ ms/token}$.

Real-Time Metric Telemetry Comparison

| Telemetry Source | Latency Resolution | Cgroup Awareness | Memory vs Compute Disambiguation | Overhead on Host | | :--- | :--- | :--- | :--- | :--- | | Standard DCGM Exporter | 10,000 ms (10s) | No | Poor | ~0.5% CPU | | eBPF Driver Intercept | < 1 ms | Native (cgroup v2) | Exact (Uprobe + Ioctl) | < 0.8% CPU |

Impact on Infrastructure Costs & SLA Latency

Monthly GPU Infrastructure Spend ($)
Baseline (DCGM HPA)  [=========================================] $124,000
eBPF-Driven FinOps   [==========================] $74,400  (-40.0%)

Tail Latency P99 TTFT (ms)
Baseline (DCGM HPA)  [====================] 42 ms
eBPF-Driven FinOps   [=====================] 44 ms  (SLA Target: <=45ms)

By substituting out-of-band DCGM metric scaling with in-band, eBPF-driven real-time allocation:

  • Infrastructure Cloud Costs reduced by 40.0%: Node count shrank dynamically during off-peak windows via high-density MPS consolidation.
  • Zero SLA Violations: $P_{99}$ TTFT remained strictly under the 45ms target, avoiding cold starts by retaining warm VRAM pages in shared MPS allocations.
  • 100% FinOps Cost Traceability: Every dollar spent on GPU infrastructure was accurately mapped to specific enterprise tenants based on exact SM cycle consumption.

Architectural Takeaways for Infrastructure Leaders

  1. DCGM is for Hardware Health; eBPF is for FinOps Runtime Engine Control: Relying on aggregate polling metrics for transient AI workloads guarantees resource over-provisioning. Kernel-level instrumentation provides the microsecond fidelity required for modern model serving architectures.
  2. Context is Everything: Unifying Linux cgroup v2 identifiers with user-space CUDA runtime calls bridges the gap between raw hardware execution and high-level Kubernetes abstractions (Pods, Namespaces, Tenants).
  3. Decouple Memory Preservation from Compute Scaling: VRAM occupancy should not dictate compute scaling decisions. Retain memory footprints where possible via fractional MPS slicing while dynamically scaling down active compute streams.

By taking control of GPU driver tracing at the kernel layer, platform teams can eliminate the trade-off between strict tail-latency SLAs and sustainable cloud expenditure—turning GPU infrastructure from a opaque cost center into an optimized, self-scaling engine.