In the era of multi-tenant Kubernetes clusters, cost management has evolved from a monthly finance review into a real-time engineering challenge. As organizations scale to thousands of pods across hundreds of nodes, standard cloud billing constructs—like AWS Cost Explorer or GCP Billing—break down. They tell you what your EC2 instances or compute nodes cost, but they cannot tell you which specific microservice, team, or API endpoint generated that $50,000 spike in inter-AZ network egress last Tuesday.
At Ecstaticloud, we ran into this exact boundary. Traditional pod-cost allocation models relying on cAdvisor metrics or resource requests vs. limits left us with massive blind spots. Short-lived pods, bursty I/O, and opaque network data transfers led to over-provisioned infrastructure and unallocated "dark matter" cloud spend.
To solve this, we engineered an automated FinOps pipeline powered by eBPF (Extended Berkeley Packet Filter). By hooking directly into kernel tracepoints, we captured network egress, CPU runtime, and disk I/O at the cgroup level with minimal overhead. The result? A 40% reduction in idle cloud spend across our fleet without degrading application latency or availability.
Here is an architectural deep dive into how we built it.
The Limitations of Traditional K8s Cost Accounting
Most Kubernetes cost-allocation tools (including OpenCost and Kubecost) fundamentally rely on two data sources:
- Kubernetes API Server Specs: Resource
requestsandlimits. - Metrics Server / cAdvisor: Sampled CPU/memory utilization via
/procfilesystem polling.
While useful for baseline estimates, these sources introduce severe inaccuracies at scale:
+-----------------------------------------------------------------------+
| TRADITIONAL COST BLINDSPOTS |
+-----------------------------------------------------------------------+
| 1. Static Allocation Fallacy: Billing based on K8s "Requests" |
| ignores actual CPU core execution time (Idle waste). |
| |
| 2. Network Egress Black Hole: cAdvisor tracks total interface bytes |
| but cannot differentiate intra-node, inter-AZ, or public egress. |
| |
| 3. High Metric Sampling Latency: Short-lived pods (CronJobs, CI/CD) |
| spin up and die between 15-second Prometheus scrapings. |
+-----------------------------------------------------------------------+
When inter-AZ cross-talk costs $0.01 per GB and internet egress hits $0.09 per GB, allocating network costs evenly based on CPU usage is flawed. We needed micro-second CPU accounting and destination-aware packet tracking per Pod container.
The Architecture: eBPF-Driven Cost Attribution
eBPF allows us to execute sandboxed programs inside the Linux kernel without changing kernel source code or loading kernel modules. Because Kubernetes containers are fundamentally Linux processes managed by cgroups (v1 or v2), eBPF probes can observe kernel events, tag them with the underlying cgroup_id, and expose this telemetry to user-space collectors.
Here is the high-level architecture of the Ecstaticloud FinOps pipeline:
+------------------------------------------------------------------------+
| KERNEL SPACE |
| |
| +------------------------+ +----------------------------+ |
| | tracepoint:sched_switch| | kprobe:tcp_sendmsg | |
| +-----------+------------+ +-------------+--------------+ |
| | | |
| v v |
| [ Extract Task / cgroup ] [ Extract Socket / Dest IP ] |
| | | |
| +------------------+------------------+ |
| | |
| v |
| BPF_MAP_TYPE_RINGBUF |
+----------------------------------+-------------------------------------+
|
v
+----------------------------------+-------------------------------------+
| USER SPACE |
| |
| +----------------------------------+ |
| | ecstaticloud-finops-daemon | |
| +----------------+-----------------+ |
| | |
| Correlate cgroup_id -> Pod Name / Namespace / Node |
| | |
| v |
| +-----------------------+ |
| | ClickHouse / Prom | |
| +-----------+-----------+ |
| | |
| v |
| +-----------------------+ |
| | FinOps Engine & VPA | |
| +-----------------------+ |
+------------------------------------------------------------------------+
Key Kernel Tracepoints & Probes Utilized:
sched/sched_switch: Triggers whenever the kernel switches execution from one process thread to another. We compute exact CPU execution time down to the nanosecond for each container.kprobe/tcp_sendmsg/kretprobe/tcp_sendmsg: Hooks into the TCP stack to inspect outbound payloads, matching destination IP addresses against IP ranges (Intra-VPC, Cross-AZ, Cross-Region, and Public Internet).
Step-by-Step Implementation: Building the eBPF Micro-Cost Probe
Let’s look at the underlying code that powers kernel-level resource attribution.
1. The Kernel-Space Program (C / eBPF)
The eBPF program hooks into sched_switch to calculate CPU runtime per cgroup and writes to a ring buffer.
// pod_cost_tracer.bpf.c
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
struct cpu_event_t {
u64 cgroup_id;
u64 pid;
u64 runtime_ns;
};
// Map to store thread start timestamps
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, u32); // PID
__type(value, u64); // Timestamp (ns)
__uint(max_entries, 10240);
} start_time_map SEC(".maps");
// Ring buffer for pushing metrics to userspace
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024);
} cpu_events SEC(".maps");
SEC("tracepoint/sched/sched_switch")
int handle_sched_switch(struct trace_event_raw_sched_switch *ctx) {
u64 ts = bpf_ktime_get_ns();
u32 prev_pid = ctx->prev_pid;
u32 next_pid = ctx->next_pid;
// Account time for the process getting switched out
u64 *start_ts = bpf_map_lookup_elem(&start_time_map, &prev_pid);
if (start_ts) {
u64 delta = ts - *start_ts;
u64 cgroup_id = bpf_get_current_cgroup_id();
struct cpu_event_t *event = bpf_ringbuf_reserve(&cpu_events, sizeof(struct cpu_event_t), 0);
if (event) {
event->cgroup_id = cgroup_id;
event->pid = prev_pid;
event->runtime_ns = delta;
bpf_ringbuf_submit(event, 0);
}
bpf_map_delete_elem(&start_time_map, &prev_pid);
}
// Set start time for the process getting switched in
bpf_map_update_elem(&start_time_map, &next_pid, &ts, BPF_ANY);
return 0;
}
char LICENSE[] SEC("license") = "GPL";
2. Network Egress Classification Probe
Network egress pricing depends on the target IP. We intercept tcp_sendmsg to determine packet destinations before transmission.
SEC("kprobe/tcp_sendmsg")
int BPF_KPROBE(tcp_sendmsg_entry, struct sock *sk, struct msghdr *msg, size_t size) {
u64 cgroup_id = bpf_get_current_cgroup_id();
u32 daddr = SK_BPF_CB(sk)->sk->sk_daddr; // Extract destination IPv4 address
// Custom helper struct to send to collector
struct net_event_t event = {
.cgroup_id = cgroup_id,
.bytes = size,
.dest_ip = daddr
};
// Emit event to Network Ring Buffer...
return 0;
}
3. Userspace Collector & Kubernetes Correlator (Go)
The user-space daemon (built using cilium/ebpf and client-go) consumes events from the BPF ring buffer, maps cgroup_ids to Kubernetes Pod Metadata, and calculates actual micro-costs.
package main
import (
"context"
"fmt"
"net"
"os"
"path/filepath"
"github.com/cilium/ebpf/ringbuf"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
)
type CpuEvent struct {
CgroupId uint64
Pid uint32
RuntimeNs uint64
}
func main() {
// Initialize K8s Client
config, _ := rest.InClusterConfig()
clientset, _ := kubernetes.NewForConfig(config)
// Open eBPF Ring Buffer Reader
// (Assuming bpfModule loaded via cilium/ebpf)
rd, err := ringbuf.NewReader(bpfObjects.CpuEvents)
if err != nil {
panic(err)
}
defer rd.Close()
for {
record, err := rd.Read()
if err != nil {
continue
}
event := parseCpuEvent(record.RawSample)
podMetadata := resolveCgroupToPod(clientset, event.CgroupId)
// Calculate Cost: e.g., $0.000000008 per CPU nanosecond
costUSD := float64(event.RuntimeNs) * 0.000000000008
fmt.Printf("[FINOPS COST] Namespace: %s | Pod: %s | Runtime: %dns | Micro-Cost: $%f\n",
podMetadata.Namespace, podMetadata.Name, event.RuntimeNs, costUSD)
}
}
// Map cgroup v2 inode ID to Kubernetes Pod object
func resolveCgroupToPod(k8s *kubernetes.Clientset, cgroupId uint64) metav1.ObjectMeta {
// In production, this matches local cgroupfs paths (/sys/fs/cgroup/kubepods.slice/...)
// cached in an in-memory thread-safe map populated by pod informers.
return metav1.ObjectMeta{Name: "payment-service-6799d-x9z2", Namespace: "prod"}
}
Translating Telemetry to Financial Signals: The Engine
Collecting nanoseconds and raw byte counts is step one. Transforming this telemetry into financial signals requires joining eBPF metrics with cloud provider pricing APIs (AWS Price List API / GCP Cloud Billing API).
Micro-Cost Formulating Rules:
-
Exact CPU Execution Cost: $$\text{Cost}{\text{cpu}} = \left( \frac{\text{Runtime}{\text{ns}}}{3.6 \times 10^{12}} \right) \times \text{vCPU Hour Rate}$$
-
Network Egress Dynamic Attribution: $$\text{Cost}{\text{net}} = \sum (\text{Bytes}{\text{dest}} \times \text{TierRate}(\text{dest}))$$
- Where $\text{TierRate}(\text{dest})$ evaluates:
0.00ifdestis inside the same Availability Zone.$0.01/GBifdestis cross-AZ in the same region.$0.02/GBifdestis cross-Region.$0.09/GBifdestis public internet / CDN.
- Where $\text{TierRate}(\text{dest})$ evaluates:
Real-World Analytics: ClickHouse FinOps Schema
We push processed eBPF events into ClickHouse due to its compression efficiency and speed on analytical queries.
CREATE TABLE finops.pod_micro_costs
(
timestamp DateTime64(3, 'UTC'),
namespace String,
pod_name String,
node_id String,
cpu_runtime_ns UInt64,
network_bytes_intra_az UInt64,
network_bytes_inter_az UInt64,
network_bytes_public UInt64,
calculated_cost_usd Float64
)
ENGINE = MergeTree()
ORDER BY (timestamp, namespace, pod_name);
Uncovering Over-Provisioned Pods via SQL:
With eBPF precision, we can execute real-time efficiency queries comparing K8s requested costs vs. actual eBPF runtime costs:
SELECT
namespace,
pod_name,
sum(calculated_cost_usd) AS actual_ebpf_cost,
-- Querying baseline requested cost derived from node instance static rate
(sum(cpu_runtime_ns) / 3600000000000) * 0.0416 AS requested_static_cost,
(1 - (actual_ebpf_cost / requested_static_cost)) * 100 AS percentage_idle_waste
FROM finops.pod_micro_costs
WHERE timestamp >= NOW() - INTERVAL 7 DAY
GROUP BY namespace, pod_name
HAVING requested_static_cost > 10
ORDER BY percentage_idle_waste DESC;
How Ecstaticloud Slashed Idle Cloud Spend by 40%
By installing our eBPF FinOps agent across our production clusters, we unlocked three critical optimization levers:
1. Dynamic Rightsizing Based on True Kernel Usage
Traditional Vertical Pod Autoscalers (VPA) rely on cAdvisor metrics that can smooth over spikes or misinterpret allocated memory pages. Using nanosecond CPU telemetry, we built an automated controller that right-sized CPU resource requests down to match real execution profiles.
# Output of our automated FinOps Rightsizing Controller
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: dynamic-rightsize-payment-api
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: payment-api
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: '*'
# Constrained strictly using eBPF peak execution delta metrics
minAllowed:
cpu: "25m"
memory: "64Mi"
maxAllowed:
cpu: "500m"
memory: "512Mi"
2. Eliminating Inter-AZ "Dark Egress"
The network probe revealed that microservices were randomly querying database read-replicas situated in adjacent Availability Zones, silently racking up thousands of dollars in inter-AZ charges.
We leveraged this data to institute Topology-Aware Hints and update core Service meshes (Cilium / Istio) to route traffic strictly within the same zone:
apiVersion: service.k8s.io/v1
kind: Service
metadata:
name: catalog-db-replica
annotations:
# Restricts routing to local AZ based on eBPF traffic identification
service.kubernetes.io/topology-mode: Auto
3. Idle Buffer Elimination via Node-Pool Autotuning
Because eBPF provided hyper-accurate capacity modeling, we safely reduced our cluster-autoscaler buffer capacity from 30% headroom down to 8% without risking out-of-memory (OOM) kills or pod scheduling starvation during load bursts.
Operational Considerations & Lessons Learned
Deploying eBPF at scale across heterogeneous fleets presents engineering trade-offs. Here are key items to plan for:
- Kernel Version Consistency: eBPF requires modern Linux kernels. Features like
BPF_MAP_TYPE_RINGBUFrequire Linux 5.8+. Ensure your node AMIs (e.g., Bottlerocket, Ubuntu 22.04, Flatcar) run modern kernels with CO-RE (Compile Once – Run Everywhere) enabled via BTF (/sys/kernel/btf/vmlinux). - Probe Overhead Minimization: High-throughput tracepoints like
sched_switchexecute millions of times per second per core. Never perform string manipulation or complex lookups inside the BPF C code. Push raw integer IDs (cgroup_id,pid) to user-space and perform enrichment async. - Ephemeral short-lived workloads: For short-lived pods (e.g., Serverless Knative or CI worker pods), flush ring buffers immediately on
sys_exitto prevent dropped events when namespaces detach.
Conclusion
Standard Kubernetes metric aggregation is no longer sufficient for modern FinOps engineering. Relying on coarse approximations leads to bloated cluster allocations and surprising network egress invoices.
By moving cost accounting directly into the Linux kernel via eBPF, Ecstaticloud established an indisputable source of truth for resource execution. We achieved pinpoint accuracy down to individual Kubernetes pods, transformed idle capacity into dynamic auto-scaling policies, and ultimately shaved 40% off our monthly cloud bill.
What's Next?
- Try introducing eBPF-based metrics to your staging clusters using open-source tools like Cilium Hubble or Inspektor Gadget.
- Hook eBPF telemetry directly into custom Kubernetes controllers to transform cost visibility into automated, real-time infrastructure tuning.