When operating large-scale Kubernetes clusters running hundreds of microservices, traditional observability stacks frequently fall short during high-severity incidents. Metrics, logs, and distributed traces give us visibility, but during complex, multi-system cascading failures, they generate an overwhelming amount of signal noise. SREs end up sifting through thousands of log lines and metric spikes while the Mean Time to Resolution (MTTR) ticks away.
What if your platform could detect sub-second kernel-level anomalies and autonomously repair the cluster before an alert ever reaches a human engineer?
By combining eBPF (Extended Berkeley Packet Filter) for low-overhead, deep-kernel telemetry with a Local LLM (Large Language Model) fine-tuned for deterministic operational diagnosis, we can construct a closed-loop, self-healing Kubernetes control plane.
In this post, we will walk through the architecture, system design, and implementation details of an autonomous remediation system running fully inside your infrastructure.
High-Level System Architecture
The self-healing pipeline relies on a zero-trust, edge-first architecture. It listens to kernel events via eBPF, enriches raw traces with Kubernetes API metadata, runs inference through a locally hosted LLM running on dedicated in-cluster accelerators, and executes constrained remediation via a custom Kubernetes Operator.
┌─────────────────────────────────────────────────────────────────────────┐
│ KUBERNETES NODE │
│ │
│ ┌────────────────────┐ ┌──────────────────────────────────────┐ │
│ │ User / App Space │ │ Kernel Space │ │
│ │ │ │ │ │
│ │ ┌──────────────┐ │ │ ┌────────────────────────────────┐ │ │
│ │ │ Microservice │ │ │ │ eBPF Probes / Ring Buffers │ │ │
│ │ └──────┬───────┘ │ │ │ (Tetragon / Custom C Hooks) │ │ │
│ └─────────┼──────────┘ └──┴───────────────┬───────────────────┘ │
└────────────┼─────────────────────────────────────┼──────────────────────┘
│ Socket Drops / Memory Pressure │ Raw Kernel Events
▼ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ EBPF TELEMETRY AGGREGATOR │
│ (Filters raw perf buffer -> correlates K8s metadata) │
└──────────────────────────────────┬──────────────────────────────────────┘
│ Enriched Event JSON
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ LOCAL LLM REASONING ENGINE │
│ (vLLM / Llama 3 8B Instruct / JSON Mode / Structural Output) │
└──────────────────────────────────┬──────────────────────────────────────┘
│ Structured Remediation Plan
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ REMEDIATION OPERATOR (CRD) │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ Safety Engine: Rate Limiters, Circuit Breakers, RBAC Bounds │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ Approved Action │
│ ▼ │
│ ┌───────────────────────────────┐ │
│ │ Kubernetes API (Patch/Restart)│ │
│ └───────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Why Local LLMs?
Running LLMs in the public cloud for real-time cluster remediation introduces three major operational bottlenecks:
- Network Latency: Outbound HTTP API calls introduce variable latency (500ms–3000ms), breaking sub-second remediation targets.
- Data Privacy & Security: Kernel event telemetry contains sensitive runtime information (IPs, syscall arguments, socket metadata). Sending raw kernel dumps to third-party endpoints creates security compliance risks.
- Cost & Reliability: Public LLM API rate limits and unexpected downtime can render your self-healing control plane useless during major cluster-wide disruptions.
By deploying lightweight, quantised models (such as Llama-3-8B-Instruct or Mistral-7B-Instruct) locally using vLLM or Ollama backed by GPU/NPU nodes, inference runs with sub-200ms latency at zero variable API cost.
Phase 1: Deep Kernel Telemetry via eBPF
Traditional sidecars or APM agents rely on user-space metrics or log scraping, which miss critical, low-level system failures such as TCP socket leaks, silent kernel OOMs, and deadlocked threads.
We use eBPF via a Cilium Tetragon TracingPolicy to hook into kernel functions like tcp_drop, do_exit, and sys_enter_write without modifying application code or introducing user-space performance penalties.
Tetragon TracingPolicy for Anomaly Detection
The following configuration hooks directly into kernel TCP drops and process state transitions, bubbling up anomalies to user space in real time:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: kernel-anomaly-detector
namespace: kube-system
spec:
kprobes:
# Hook kernel TCP socket drops
- call: "tcp_drop"
syscall: false
args:
- index: 0
type: "sock"
- index: 1
type: "sk_buff"
selectors:
- matchArgs:
- index: 0
operator: "NotDList"
values:
- "127.0.0.1"
matchActions:
- action: Post
# Hook process terminations caused by signals (e.g., SIGSEGV, SIGKILL)
- call: "get_signal"
syscall: false
args:
- index: 0
type: "ksignal"
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "9" # SIGKILL
- "11" # SIGSEGV
matchActions:
- action: Post
Phase 2: Event Aggregation & Context Enrichment
Raw eBPF telemetry is high-volume and lacks high-level Kubernetes application awareness. Before passing the anomaly payload to our Local LLM, we must aggregate kernel events over a sliding time window and enrich them with live cluster metadata via the Kubernetes API.
Here is the Python enrichment daemon running inside the cluster that transforms kernel telemetry into diagnostic context:
import json
import asyncio
from kubernetes_asyncio import client, config
class EventContextEnricher:
def __init__(self):
self.k8s_api = None
async def initialize(self):
config.load_incluster_config()
self.k8s_api = client.CoreV1Api()
async def enrich_ebpf_event(self, raw_event_json: str) -> dict:
event = json.loads(raw_event_json)
pod_name = event.get("pod_name")
namespace = event.get("namespace", "default")
# Fetch live pod status, resource limits, and recent events
pod_info = {"status": "Unknown", "restart_count": 0, "logs": ""}
try:
pod = await self.k8s_api.read_namespaced_pod(name=pod_name, namespace=namespace)
pod_info["status"] = pod.status.phase
if pod.status.container_statuses:
pod_info["restart_count"] = pod.status.container_statuses[0].restart_count
# Get recent pod logs (last 20 lines)
logs = await self.k8s_api.read_namespaced_pod_log(
name=pod_name, namespace=namespace, tail_lines=20
)
pod_info["logs"] = logs
except Exception as e:
pod_info["error"] = f"Failed to fetch K8s context: {str(e)}"
# Construct enriched diagnostic payload
enriched_payload = {
"timestamp": event.get("timestamp"),
"kernel_event": {
"function": event.get("function_name"),
"signal": event.get("signal_number"),
"arg_details": event.get("args")
},
"kubernetes_context": {
"namespace": namespace,
"pod_name": pod_name,
"status": pod_info["status"],
"restart_count": pod_info["restart_count"],
"tail_logs": pod_info["logs"]
}
}
return enriched_payload
Phase 3: Deterministic LLM Reasoning Engine
Standard LLM text output is unsuited for autonomous cluster management due to potential hallucinations and unpredictable formatting. To solve this, we force the local LLM to output strictly formatted Pydantic/JSON schemas using function calling or structured decoding engines (e.g., Guidance, Outlines, or vLLM’s JSON schema mode).
Defining the Remediation Schema
from pydantic import BaseModel, Field
from typing import Literal, Optional
class RemediationPlan(BaseModel):
anomaly_summary: str = Field(
description="Brief diagnosis of the root cause based on telemetry."
)
confidence_score: float = Field(
description="Confidence level between 0.0 and 1.0."
)
action_type: Literal[
"RESTART_POD",
"SCALE_DEPLOYMENT",
"ISOLATE_NETWORK",
"CLEAR_EVICTION_CACHE",
"NO_ACTION"
] = Field(description="The exact remediation primitive to execute.")
target_resource: str = Field(
description="Name of the pod or deployment to act upon."
)
namespace: str = Field(description="Target Kubernetes namespace.")
parameters: Optional[dict] = Field(
default={},
description="Action parameters, e.g., replica count or network isolation rules."
)
Prompt Engineering for Operational Safety
Below is the inference engine interfacing with local vLLM endpoints using structured output enforcement:
import requests
from pydantic import TypeAdapter
VLLM_ENDPOINT = "http://vllm-service.kube-system.svc.cluster.local:8000/v1/chat/completions"
SYSTEM_PROMPT = """
You are an expert Autonomous Kubernetes SRE Engine.
You receive low-level eBPF kernel telemetry enriched with live Kubernetes context.
Analyze the problem and produce a structured remediation plan matching the JSON schema.
CRITICAL RULES:
1. ONLY recommend actions if confidence score is >= 0.85.
2. If TCP drops are caused by socket exhaustion, DO NOT restart immediately; isolate network or drain first.
3. If process exit code is 137 (OOM), recommend scaling memory or restarting the target controller safely.
4. Output strict JSON matching the schema. Do not add markdown or extra text.
"""
def generate_remediation_plan(enriched_event: dict) -> RemediationPlan:
prompt = f"Enriched Event Payload:\n{json.dumps(enriched_event, indent=2)}"
payload = {
"model": "meta-llama/Meta-Llama-3-8B-Instruct",
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt}
],
"temperature": 0.0, # Zero temperature for deterministic evaluation
"response_format": {
"type": "json_object",
"schema": RemediationPlan.model_json_schema()
}
}
response = requests.post(VLLM_ENDPOINT, json=payload, timeout=2.0)
response_json = response.json()
raw_content = response_json["choices"][0]["message"]["content"]
# Parse and validate schema deterministic guarantee
plan = RemediationPlan.model_validate_json(raw_content)
return plan
Phase 4: Closed-Loop Remediation Operator
To execute the LLM's remediation decision safely, we use a custom controller pattern. We do not grant the LLM direct access to the Kubernetes API. Instead, the LLM emits a plan object, which is ingested by a Remediation Operator. The operator acts as a safety gate.
Structural Safety Features
- Circuit Breakers: Limit auto-remediations to maximum $N$ actions per namespace per hour to prevent cascading reboots.
- Confidence Thresholding: Any LLM plan with a
confidence_score$< 0.85$ is downgraded to a human approval queue (e.g., Slack/PagerDuty notification). - Blast Radius Control: High-impact operations (e.g., deleting PVs, modifying core namespaces) are explicitly blocked at the controller RBAC level.
class SafetyEngine:
def __init__(self, max_actions_per_hour: int = 5):
self.action_history = []
self.max_actions = max_actions_per_hour
def is_action_safe(self, plan: RemediationPlan) -> tuple[bool, str]:
# Rule 1: Confidence score gate
if plan.confidence_score < 0.85:
return False, f"Confidence too low: {plan.confidence_score}"
# Rule 2: Protected namespaces
if plan.namespace in ["kube-system", "cert-manager", "monitoring"]:
return False, f"Execution blocked on protected namespace: {plan.namespace}"
# Rule 3: Rate limiting / Circuit breaking
now = asyncio.get_event_loop().time()
# Clean history older than 1 hour (3600 seconds)
self.action_history = [t for t in self.action_history if now - t < 3600]
if len(self.action_history) >= self.max_actions:
return False, "Rate limit exceeded: Circuit breaker tripped!"
return True, "Action approved"
async def execute_remediation(plan: RemediationPlan, safety: SafetyEngine):
safe, reason = safety.is_action_safe(plan)
if not safe:
print(f"[REMEDIATION REJECTED]: {reason}")
# Route to Slack / Opsgenie alert pipeline
return
print(f"[EXECUTING REMEDIATION]: {plan.action_type} on {plan.target_resource}")
# Record action timestamp for rate limiter
safety.action_history.append(asyncio.get_event_loop().time())
# Execute against Kubernetes API
async with client.ApiClient() as api_client:
core_v1 = client.CoreV1Api(api_client)
if plan.action_type == "RESTART_POD":
await core_v1.delete_namespaced_pod(
name=plan.target_resource,
namespace=plan.namespace,
body=client.V1DeleteOptions(grace_period_seconds=0)
)
elif plan.action_type == "ISOLATE_NETWORK":
# Apply dynamic NetworkPolicy to block bad traffic
pass
# Additional action handling hooks...
Real-World Scenario: Socket Leak Diagnostics
To see the system in action, let's look at how it handles a complex runtime incident.
1. The Incident
A backend microservice (order-processor) suffers from an unresolved file descriptor/socket leak due to a buggy third-party HTTP client library. Over time, all available sockets are exhausted. The application stops accepting traffic, but HTTP health probes pass because the control framework thread remains alive.
2. Detection (eBPF)
Tetragon catches kernel-level tcp_drop events continuously firing for the backend socket, along with failed system calls (sys_enter_connect returning -EMFILE / Too many open files).
{
"function_name": "tcp_drop",
"pid": 41209,
"args": {"sk_state": "TCP_SYN_SENT", "error": "EMFILE"},
"pod_name": "order-processor-78d49bbfd-x92zk",
"namespace": "production"
}
3. Diagnosis (Local LLM)
The payload is enriched with pod log fragments showing java.net.SocketException: Too many open files and sent to the local Llama-3-8B-Instruct endpoint. The model synthesizes the telemetry into a structured diagnosis:
{
"anomaly_summary": "Process has run out of file descriptors (EMFILE) resulting in TCP drops while HTTP thread passes health checks.",
"confidence_score": 0.96,
"action_type": "RESTART_POD",
"target_resource": "order-processor-78d49bbfd-x92zk",
"namespace": "production",
"parameters": {
"force": true
}
}
4. Remediation & Verification
The Remediation Operator checks the rate limiters, verifies confidence thresholds ($0.96 > 0.85$), and deletes the bad pod instance. Kubernetes spins up a clean replacement pod immediately.
Total elapsed time from kernel socket drop to cluster recovery: 680ms.
Trade-Offs and Production Realities
While autonomous self-healing reduces SRE workload, running this setup in production requires clear boundary parameters:
| Challenge | Impact | Mitigation Strategy | | :--- | :--- | :--- | | Model Hallucinations | Destructive actions executed on cluster resources. | Enforce strict JSON Schema decoding (vLLM/Outlines) and strict validation layers inside the operator logic. | | Flapping / CrashLoops | LLM repeatedly restarts a fundamentally broken deployment. | Implement exponential backoff rate limiting per pod/deployment ID. | | Resource Overhead | Local LLM inference requires cluster GPU resources. | Host models on dedicated GPU nodes (e.g., NVIDIA L4 or T4 instances) shared across namespaces using vLLM dynamic batching. | | Security Auditing | Difficulty auditing why a decision was made. | Store full LLM prompt/response traces inside an append-only, immutable audit log index (e.g., Elasticsearch or Loki). |
Conclusion
The combination of eBPF and Local LLMs fundamentally changes platform engineering. By replacing static human-defined alert runbooks with active runtime intelligence, we can resolve complex microservice failures at the kernel level before end users are impacted.
Building a self-healing Kubernetes cluster doesn't mean giving an AI free rein over your infrastructure. It's about building a multi-layered safety pipeline: using eBPF for fast telemetry, a Local LLM for contextual reasoning, and Kubernetes Operators to enforce safety boundaries.