Moving Generative AI from an impressive internal demo to an enterprise-grade production system is where the hard realities of distributed systems hit home. While prototyping a Retrieval-Augmented Generation (RAG) pipeline requires little more than LangChain and a few API keys, running one at scale—processing thousands of concurrent queries with sub-second retrieval SLAs and 99.99% availability—demands rigorous infrastructure engineering.
Unpredictable LLM latencies, rate limits on embedding endpoints, memory-bound vector indexing, and cascading failures across microservices are standard failure modes for non-resilient LLMOps implementations.
In this architectural deep dive, we will design and deploy an autonomous, self-healing RAG pipeline using Amazon EKS for cloud-native orchestration and AWS Bedrock for serverless foundation models. We will cover event-driven document ingestion, semantic caching, automated node provisioning with Karpenter, and real-time L7 observability using eBPF (Extended Berkeley Packet Filter).
Architectural Blueprint: The Dual-Plane RAG Pattern
To achieve high concurrency and fault isolation, enterprise RAG architectures must decouple the Data Ingestion Plane (asynchronous, throughput-optimized) from the Inference & Retrieval Plane (synchronous, latency-optimized).
[ Client Applications ]
│
▼
[ AWS ALB / Envoy Gateway ]
│
┌─────────────┴─────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ Inference Plane (EKS) │ │ Ingestion Plane (EKS) │
│ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │
│ │ FastAPI Query Svc │ │ │ │ Doc Parsing Workers │ │
│ └──────────┬──────────┘ │ │ └──────────▲──────────┘ │
└────────────┼────────────┘ └────────────┼────────────┘
│ │
┌─────────┴─────────┐ │
▼ ▼ │
┌─────────────┐ ┌─────────────┐ ┌──────┴──────┐
│ ElastiCache │ │ AWS Bedrock │ │ AWS SQS │
│ (Semantic) │ │ Models │ └──────▲──────┘
└─────────────┘ └─────────────┘ │
│ ▲ ┌──────┴──────┐
▼ │ │ S3 Document │
┌──────────────────────────┴┐ │ Lake │
│ Vector DB (OpenSearch / │ └─────────────┘
│ pgvector / Milvus) │
└───────────────────────────┘
Key Architectural Pillars
- Inference Plane: Hosted on EKS, utilizing Envoy Gateway for traffic routing, FastAPI for query processing, ElastiCache for Redis for semantic caching, and AWS Bedrock (Claude 3.5 Sonnet / Titan Embeddings) for generation and vector representations.
- Ingestion Plane: An event-driven, serverless-like worker pool on EKS triggered by S3 file events via AWS SQS and auto-scaled by KEDA (Kubernetes Event-driven Autoscaling).
- Storage Tier: OpenSearch Serverless or Amazon Aurora PostgreSQL (
pgvector) configured with HNSW (Hierarchical Navigable Small World) indexing for sub-10ms nearest-neighbor retrieval.
1. Asynchronous Event-Driven Ingestion Pipeline
Ingesting enterprise documents (PDFs, DOCX, Markdown) requires CPU-heavy parsing, tokenization, and API-bound embedding generation. Processing this synchronously in an API path degrades system availability.
Self-Healing Queue Processing with KEDA
We deploy ingestion workers as stateless Kubernetes Deployments. Rather than scaling based on raw CPU or Memory utilization (which poorly correlates with work queues), we scale directly on the AWS SQS queue depth using KEDA.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: rag-document-ingestor-scaler
namespace: llmops
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: rag-document-ingestor
minReplicaCount: 0
maxReplicaCount: 100
cooldownPeriod: 300
pollingInterval: 15
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/rag-ingestion-queue
queueLength: "10"
awsRegion: us-east-1
identityOwner: operator
Managing Rate Limits & Throttling with Exponential Backoff
AWS Bedrock enforces strict API quotas on InvokeModel and BedrockEmbeddings. When worker pods hit 429 Too Many Requests, naive retries cause cascading failures across the EKS cluster.
Below is a production-grade Python worker pattern using aioboto3 with jittered exponential backoff and dynamic token bucket throttling:
import asyncio
import random
import logging
import botocore.exceptions
from aioboto3 import Session
logger = logging.getLogger("rag_worker")
class ResilientBedrockEmbedder:
def __init__(self, region_name: str = "us-east-1"):
self.session = Session()
self.region_name = region_name
async def generate_embedding_with_retry(
self, text: str, max_retries: int = 6, base_delay: float = 0.5
) -> list[float]:
async with self.session.client("bedrock-runtime", region_name=self.region_name) as client:
for attempt in range(max_retries):
try:
response = await client.invoke_model(
modelId="amazon.titan-embed-text-v2:0",
contentType="application/json",
accept="application/json",
body=f'{{"inputText": "{text}"}}'
)
response_body = await response['body'].read()
import json
return json.loads(response_body)["embedding"]
except botocore.exceptions.ClientError as e:
error_code = e.response["Error"]["Code"]
if error_code in ["ThrottlingException", "TooManyRequestsException"]:
# Full Jitter Backoff Formula
sleep_time = random.uniform(0, min(30.0, base_delay * (2 ** attempt)))
logger.warning(
f"Throttled by Bedrock. Retry {attempt + 1}/{max_retries} "
f"in {sleep_time:.2f}s. Error: {e}"
)
await asyncio.sleep(sleep_time)
else:
logger.error(f"Non-retryable Bedrock error: {e}")
raise e
raise RuntimeError(f"Exhausted retries for text chunk: {text[:30]}...")
2. High-Concurrency Query Pipeline & Semantic Caching
To reduce cost and lower dynamic generation latency from ~1500ms to <20ms, an enterprise RAG pipeline must implement Semantic Caching.
Unlike exact-match caching (e.g., Redis string key lookup), semantic caching computes the vector embedding of an incoming prompt and searches a spatial index in Redis for previously answered queries with a cosine similarity above a specified threshold (e.g., $0.92$).
import json
import numpy as np
from redis.asyncio import Redis
from redis.commands.search.query import Query
class SemanticCacheManager:
def __init__(self, redis_client: Redis, similarity_threshold: float = 0.92):
self.redis = redis_client
self.threshold = similarity_threshold
self.vector_dim = 1024 # Dimension for Titan v2
async def get_cached_response(self, query_embedding: list[float]) -> str | None:
# Convert embedding to raw bytes for Redis Vector Search (FT.SEARCH)
embedding_bytes = np.array(query_embedding, dtype=np.float32).tobytes()
# Redis Vector Search query using KNN
redis_query = (
Query("*=>[KNN 1 @vector $vec AS score]")
.sort_by("score")
.paging(0, 1)
.return_fields("response", "score")
.dialect(2)
)
query_params = {"vec": embedding_bytes}
results = await self.redis.ft("idx:semantic_cache").search(redis_query, query_params)
if results.docs:
top_match = results.docs[0]
# Convert Redis score (cosine distance) to similarity
cosine_similarity = 1 - float(top_match.score)
if cosine_similarity >= self.threshold:
return top_match.response
return None
async def cache_response(self, query_embedding: list[float], response_text: str, ttl: int = 86400):
embedding_bytes = np.array(query_embedding, dtype=np.float32).tobytes()
key = f"cache:prompt:{hash(np.array(query_embedding).tobytes())}"
await self.redis.hset(
key,
mapping={
"vector": embedding_bytes,
"response": response_text
}
)
await self.redis.expire(key, ttl)
3. Dynamic Node Provisioning with Karpenter on EKS
RAG workloads require diverse EC2 compute profiles:
- Ingestion Workers: Memory-heavy and CPU-intensive for document chunking, optical character recognition (OCR), and PDF extraction.
- Vector DB / Search Node Proxies: High-network throughput and balanced RAM for vector indexes.
- Inference Routers: Light CPU, high availability, benefiting from ARM64 Graviton instances for cost efficiency.
Karpenter allows us to bypass standard EKS Node Groups and provision node hardware JIT (Just-In-Time) based on pod specs.
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: rag-compute-pool
spec:
template:
spec:
requirements:
- key: kubernetes.io/arch
operator: In
values: ["arm64", "amd64"]
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["5"]
nodeClassRef:
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
name: rag-node-class
limits:
cpu: 1000
memory: 2000Gi
disruption:
consolidationPolicy: WhenUnderutilized
consolidateAfter: 30s
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: rag-node-class
spec:
amiFamily: AL2023
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: eks-llmops-cluster
securityGroupSelectorTerms:
- tags:
kubernetes.io/cluster/eks-llmops-cluster: owned
tags:
Environment: Production
Workload: LLMOps-RAG
By leveraging AWS Graviton (c7g, m7g instances) alongside Spot capacity for processing queues, compute infrastructure costs can be reduced by up to 60-70% compared to static node pools.
4. Advanced eBPF Observability with Cilium
Traditional APM tooling falls short when dealing with LLMs. Application-level agents add unacceptable overhead to runtime loops, and they lack visibility into socket-level TCP retransmissions, TLS handshakes, or drop packets occurring during large vector payload transfers over the wire.
Using eBPF via Cilium and Hubble, we observe layer-7 HTTP/2 interactions with AWS Bedrock and internal Vector Databases directly at the Linux kernel level, without modifying pod code.
+-------------------------------------------------------+
| EKS Pod |
| +------------------+ +----------------------+ |
| | User Application | | OpenTelemetry Agent | |
| +--------┬---------+ +----------▲-----------+ |
+-----------│----------------------------│--------------+
USER SPACE │ (HTTP/2 Request) │
───────────────────┼────────────────────────────┼────────────────
KERNEL SPACE ▼ │ (Metrics/Traces)
+----------------------------------------┴--------------+
| eBPF Engine (Cilium Probe) |
| ├── Capture Socket Events |
| ├── Parse HTTP/2 Frames & Bedrock API Payload Headers|
| └── Compute Latency Buckets & 429 Status Codes |
+-------------------------------------------------------+
Deploying L7 Traffic Parsing for Bedrock Endpoints
Using a CiliumNetworkPolicy, we enforce secure egress controls while enabling deep packet inspection (DPI) on outgoing HTTPS calls to AWS Bedrock runtime endpoints:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: monitor-bedrock-egress
namespace: llmops
spec:
endpointSelector:
matchLabels:
app: rag-inference-api
egress:
- toFQDNs:
- matchName: "bedrock-runtime.us-east-1.amazonaws.com"
toPorts:
- ports:
- port: "443"
protocol: TCP
rules:
http:
- method: "POST"
path: "/model/.*"
Prometheus Metrics Extracted via eBPF Kernel Probes
With Cilium's Hubble exporter active, we automatically collect low-overhead kernel metrics that provide visibility into Bedrock API interactions:
hubble_http_requests_total{status="429"}: Detects rate-limiting across all EKS pods prior to pod-level logger intervention.hubble_http_request_duration_seconds_bucket: Tracks exact transport-layer processing latency vs. server processing delay.hubble_tcp_conntrack_slow_path: Identifies network bottleneck issues occurring during high-concurrency vector retrieval operations.
5. Resilience Engineering: Fault Injection and Circuit Breaking
In an enterprise LLMOps architecture, external dependencies will fail. AWS Bedrock may experience transient latency spikes, OpenSearch indexes may rebalance, or vector storage pods may crash under heavy load.
To prevent system-wide outage cascades, we implement circuit breaking and graceful degradation using an application gateway pattern.
Incoming Query
│
▼
┌──────────────┐ CLOSED
│ Circuit ├────────────────────────┐
│ Breaker │ │
└──────┬───────┘ ▼
│ OPEN ┌──────────────────┐
│ (High Failure Rate) │ AWS Bedrock API │
▼ └────────┬─────────┘
┌──────────────┐ │
│ Degraded │ │ Success
│ Mode │ ▼
│ (Local Vector│ ┌──────────────────┐
│ Search + BM25│ │ Synthesized │
│ Extracted │ │ Response │
│ Highlights) │ └──────────────────┘
└──────────────┘
Implementing Resilience Rules
- Circuit State: CLOSED: System operates normally. Requests route to Bedrock Embeddings -> Vector DB -> Bedrock Claude 3.5 Sonnet.
- Circuit State: OPEN (Triggered when Bedrock returns $>15%$ rate-limit errors or $P_{99}$ latency exceeds $4000\text{ms}$ over a rolling 60s window):
- Dynamic fallback shifts execution immediately away from multi-stage LLM generation.
- System switches to returning raw text snippets sourced directly from vector DB hybrid search (BM25 + Dense) with an explicit header stating the system is operating in high-availability fallback mode.
Summary Architecture Scorecard
| Operational Vector | Standard RAG Prototype | Autonomous Enterprise LLMOps (This Architecture) | | :--- | :--- | :--- | | Scaling Trigger | Pod CPU/Memory (HPA) | Real-time SQS Queue Depth & Token Metrics via KEDA | | Node Provisioning | Static Auto Scaling Groups | Dynamic, multi-arch Spot/On-Demand via Karpenter | | Response Latency | $1500\text{ms} - 4000\text{ms}$ (Every query hits LLM) | $<20\text{ms}$ for cached semantic queries; sub-second for novel queries | | Observability | App-level print/log statements | Non-intrusive eBPF (Cilium/Hubble) socket-level tracking | | Fault Isolation | Cascading timeouts across stack | Circuit breaking, dynamic rate-limiting, graceful fallback |
Conclusion
Building enterprise-grade LLMOps pipelines requires moving beyond simple orchestration frameworks and applying proven cloud-native architectural patterns. By pairing AWS Bedrock with Amazon EKS, leveraging KEDA and Karpenter for cloud elasticity, optimizing latency with semantic caching, and maintaining deep system visibility through eBPF, you can run production RAG systems that scale efficiently and recover autonomously from service disruptions.