Building a proof-of-concept Retrieval-Augmented Generation (RAG) system with a few lines of LangChain code and a local vector store is easy. Architecting a production-grade, enterprise-scale RAG pipeline capable of handling tens of thousands of concurrent requests while maintaining sub-100ms P99 query latencies is an entirely different engineering challenge.
At scale, RAG pipelines face critical failure modes:
- Memory Explosion: Unbounded vector index memory growth leading to Kubernetes
OOMKilledpods. - GPU Underutilization vs. Overspending: Idle GPU instances burning cloud budgets during low-traffic hours, juxtaposed with degraded query throughput during sudden spikes.
- Cascading Latency: Sequential execution of embedding generation, vector retrieval, sparse keyword matching, cross-encoder re-ranking, and LLM context synthesis.
This deep dive walks through an enterprise-grade cloud architecture built on AWS EKS, leveraging Karpenter and KEDA for dynamic node and pod autoscaling, distributed Qdrant/Milvus for vector storage, and an optimized, asynchronous inference engine.
1. Enterprise RAG Blueprint on AWS EKS
To achieve sub-100ms execution times for the retrieval and re-ranking phases, the infrastructure must decouple high-throughput ingestion from real-time query paths while enforcing strict compute isolation.
[ AWS ALB / Ingress ]
│
▼
[ Istio Service Mesh / Envoy ]
│
┌───────────────────────┴───────────────────────┐
▼ ▼
[ Semantic Cache ] [ RAG Gateway Service ]
(Redis Cluster/NVMe) │
│ │
(Hit: Return Context) (Miss) │
│ ▼
│ ┌──────────────────────────────────┐
│ │ Async Orchestrator (FastAPI) │
│ └────────────────┬─────────────────┘
│ │
│ ┌──────────────────────────────┼──────────────────────────────┐
│ ▼ ▼ ▼
│ [ Dense Retrieval ] [ Sparse Retrieval ] [ Query Transformation ]
│ Vector DB (Qdrant) (OpenSearch Cluster) (Hypothetical Doc Gen)
│ Sharded / HNSW Index BM25 Inverted Index │
│ │ │ │
│ └──────────────┬───────────────┘ │
│ ▼ │
│ [ Reciprocal Rank ] │
│ [ Fusion (RRF) ] │
│ │ │
│ ▼ │
│ [ GPU Re-ranker Pool ] <────────────────────────────────┘
│ (Triton/vLLM on EKS)
│ │
└─────────────────────────────┼──────────────────────────────────────────────┐
▼ ▼
[ Streaming LLM Engine ] [ Prometheus / Grafana ]
(AWS SageMaker / vLLM) (OpenTelemetry Tracing)
Infrastructure Core Components
- Control & Routing Layer: AWS ALB managed via the AWS Load Balancer Controller, with Istio for traffic splitting, retries, and mutual TLS (mTLS).
- Compute Plane: AWS EKS running Kubernetes 1.29+ with AWS VPC CNI for native IP allocation, utilizing AWS Graviton3 (c7g) for general microservices and NVIDIA GPUs (g5/g6 series) for ML workloads.
- Data Storage Layer: Distributed Vector DB running directly on EKS with local NVMe instance storage backed by EBS
gp3/io2for durable index snapshots.
2. Distributed Vector Database Topology: Sharding & NVMe Acceleration
When scaling vector databases to hundreds of millions of vectors (e.g., 1536-dimensional embeddings), maintaining the index purely in RAM becomes economically unviable. We utilize a hybrid memory/disk-backed Qdrant or Milvus architecture deployed via StatefulSets on EKS.
Memory & Index Calculation
For $N$ vectors of dimension $D$ using 32-bit floating-point precision, raw vector storage is calculated as:
$$\text{Memory}_{\text{raw}} = N \times D \times 4 \text{ bytes}$$
For 100 million 1536-dimensional vectors: $$\text{Memory}_{\text{raw}} = 100,000,000 \times 1536 \times 4 \approx 614.4 \text{ GB}$$
Adding HNSW (Hierarchical Navigable Small World) graph overhead ($\sim 20\text{--}25%$), total memory footprint exceeds 760 GB.
Scaled Topology Config with In-Memory Graphs & Mmap Payloads
To optimize costs while preserving sub-20ms vector retrieval, we configure the HNSW graph to reside in RAM while offloading vector payloads and quantized vectors to local NVMe SSDs via memory-mapped (mmap) storage.
Here is a production-optimized StatefulSet configuration snippet for Qdrant on EKS:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: qdrant-node
namespace: vector-db
spec:
serviceName: qdrant
replicas: 6
selector:
matchLabels:
app: qdrant
template:
metadata:
labels:
app: qdrant
spec:
containers:
- name: qdrant
image: qdrant/qdrant:v1.9.2
resources:
limits:
cpu: "16"
memory: "64Gi"
hugepages-2Mi: "2Gi"
requests:
cpu: "14"
memory: "56Gi"
env:
- name: QDRANT__STORAGE__STORAGE_PATH
value: "/qdrant/storage"
- name: QDRANT__SERVICE__GRPC_PORT
value: "6334"
volumeMounts:
- name: nvme-storage
mountPath: /qdrant/storage
ports:
- containerPort: 6334
name: grpc
volumes:
- name: nvme-storage
emptyDir:
medium: Memory
---
# High-performance StorageClass using AWS Local Storage Operator for Instance NVMe
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-nvme
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
Tuning Index Parameters for High Concurrency
To avoid latency degradations under heavy load, tune the vector database indexing parameters:
{
"hnsw_config": {
"m": 16,
"ef_construct": 128,
"full_scan_threshold": 10000,
"max_indexing_threads": 8,
"on_disk": true
},
"quantization_config": {
"scalar": {
"type": "int8",
"quantile": 0.99,
"always_ram": true
}
}
}
- Scalar Quantization (
int8): Reduces memory footprint by up to $4\times$ with $<1%$ loss in recall (Precision@10), allowing larger segments of vectors to remain cached in RAM.
3. Dynamic Autoscaling: Karpenter + KEDA for GPU Infrastructure
Standard Kubernetes Horizontal Pod Autoscaler (HPA) relying on CPU/Memory utilization is insufficient for RAG workloads. Embedding generation and cross-encoder re-ranking workloads are bound by GPU utilization, CUDA queue depths, and incoming request rates.
We use KEDA (Kubernetes Event-driven Autoscaling) for metric-based pod scaling, combined with Karpenter for fast instance provisioning.
[ Incoming Requests ] ──> [ Service Bus / Prometheus Metric ]
│
▼
[ KEDA ScaledObject ]
│
(Triggers Pod Autoscaling)
│
▼
[ Kubernetes Scheduler ]
│
(Pods Pending: No CPU/GPU)
│
▼
[ Karpenter Controller ]
│
(Directly calls AWS EC2 API)
│
▼
[ Provisions g5.2xlarge Node in ~45s ]
KEDA ScaledObject Manifest for GPU Re-rankers
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: reranker-gpu-scaler
namespace: rag-inference
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: cross-encoder-reranker
minReplicaCount: 2
maxReplicaCount: 20
cooldownPeriod: 300
pollingInterval: 15
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
metricName: reranker_queue_depth
query: sum(rate(reranker_request_duration_seconds_count[1m]))
threshold: '50'
Karpenter NodePool for NVIDIA GPU Provisioning
Karpenter bypasses the standard AWS Node Group abstractions, binding directly to EC2 APIs to provision nodes within 45 seconds.
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: gpu-inference-pool
spec:
template:
spec:
requirements:
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand", "spot"]
- key: node.kubernetes.io/instance-type
operator: In
values: ["g5.xlarge", "g5.2xlarge", "g5.4xlarge"]
nodeClassRef:
name: gpu-ec2-node-class
taints:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: gpu-ec2-node-class
spec:
amiFamily: Bottlerocket
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: eks-cluster-production
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: eks-cluster-production
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 100Gi
volumeType: gp3
iops: 3000
throughput: 250
4. Sub-100ms Query Path Optimization
To guarantee sub-100ms processing times (excluding final LLM text generation), queries follow an asynchronous pipeline incorporating Semantic Caching, Hybrid Search, and Batched Cross-Encoder Re-ranking.
Query ──> [ Semantic Cache Check ] ──(Hit)──> Return Context
│
(Miss)
│
├──> Asynchronous Embedding (Triton / TensorRT)
│
├──> Dense Vector Search (Qdrant) ┐
│ ├─> Reciprocal Rank Fusion (RRF)
└──> Sparse BM25 Search (OpenSearch) ┘ │
▼
[ Cross-Encoder Re-ranker ]
Python Asynchronous Query Orchestrator
The following asynchronous Python implementation executes hybrid retrieval, applies Reciprocal Rank Fusion (RRF), and pushes candidates through a GPU-accelerated re-ranker microservice:
import asyncio
import aiohttp
import numpy as np
from typing import List, Dict, Any
class AsyncRAGEngine:
def __init__(self, qdrant_url: str, opensearch_url: str, reranker_url: str):
self.qdrant_url = qdrant_url
self.opensearch_url = opensearch_url
self.reranker_url = reranker_url
self.session = aiohttp.ClientSession(
connector=aiohttp.TCPConnector(limit=100, keepalive_timeout=60)
)
async def _vector_search(self, embedding: List[float], top_k: int = 50) -> List[Dict]:
payload = {"vector": embedding, "limit": top_k, "with_payload": True}
async with self.session.post(f"{self.qdrant_url}/collections/docs/points/search", json=payload) as resp:
res = await resp.json()
return res.get("result", [])
async def _bm25_search(self, query_text: str, top_k: int = 50) -> List[Dict]:
payload = {"query": {"match": {"text": query_text}}, "size": top_k}
async with self.session.post(f"{self.opensearch_url}/docs/_search", json=payload) as resp:
res = await resp.json()
hits = res.get("hits", {}).get("hits", [])
return [{"id": h["_id"], "score": h["_score"], "payload": h["_source"]} for h in hits]
def _reciprocal_rank_fusion(self, dense_results: List[Dict], sparse_results: List[Dict], k: int = 60) -> List[Dict]:
rrf_scores = {}
doc_map = {}
for rank, doc in enumerate(dense_results):
doc_id = doc["id"]
doc_map[doc_id] = doc["payload"]
rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
for rank, doc in enumerate(sparse_results):
doc_id = doc["id"]
doc_map[doc_id] = doc["payload"]
rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
return [{"id": doc_id, "payload": doc_map[doc_id], "rrf_score": score} for doc_id, score in sorted_docs]
async def _rerank(self, query_text: str, candidates: List[Dict], top_n: int = 5) -> List[Dict]:
payload = {
"query": query_text,
"documents": [c["payload"]["text"] for c in candidates]
}
async with self.session.post(f"{self.reranker_url}/rerank", json=payload) as resp:
scores = await resp.json()
for idx, score in enumerate(scores["scores"]):
candidates[idx]["rerank_score"] = score
sorted_candidates = sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)
return sorted_candidates[:top_n]
async def retrieve(self, query_text: str, query_embedding: List[float]) -> List[Dict]:
# Step 1: Execute Dense and Sparse Search concurrently
dense_task = asyncio.create_task(self._vector_search(query_embedding, top_k=40))
sparse_task = asyncio.create_task(self._bm25_search(query_text, top_k=40))
dense_results, sparse_results = await asyncio.gather(dense_task, sparse_task)
# Step 2: Merge using Reciprocal Rank Fusion
fused_candidates = self._reciprocal_rank_fusion(dense_results, sparse_results, k=60)[:30]
# Step 3: Fast GPU Re-ranking
final_context = await self._rerank(query_text, fused_candidates, top_n=5)
return final_context
5. Resilience, Observability & Cost Optimization
Resilience: Circuit Breaking with Envoy & Istio
When vector databases or re-ranking workers experience thermal throttling or high latencies, circuit breakers must trip quickly to avoid cascading pipeline failure.
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reranker-circuit-breaker
namespace: rag-inference
spec:
host: cross-encoder-reranker.rag-inference.svc.cluster.local
trafficPolicy:
connectionPool:
tcp:
maxConnections: 1024
http:
http1MaxPendingRequests: 100
maxRequestsPerConnection: 10
outlierDetection:
consecutive5xxErrors: 3
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50
- Fallback Strategy: If the re-ranker fails or trips the circuit breaker, the orchestration layer catches the timeout exception and gracefully falls back to returning raw RRF results directly to the LLM.
Fine-Grained Cost Engineering
To scale sustainably, infrastructure component provisioning must be segmented by cost vs. availability requirements:
| Component | Target EC2 Instance Type | Provisioning Model | Reason |
| :--- | :--- | :--- | :--- |
| RAG Gateway / Orchestrator | c7g.2xlarge (AWS Graviton3) | Spot + On-Demand Mix | High concurrency ARM architecture; cost-optimized. |
| Vector DB Nodes (Primary) | i3en.3xlarge or r6i.2xlarge | On-Demand Only | Avoids split-brain states or re-indexing costs triggered by Spot terminations. |
| Vector DB Nodes (Read Replicas) | g5.xlarge / c6i.2xlarge | Spot Instances (70%) | Read-only replicas can tolerate terminations; handles read spikes cheaply. |
| Embedding / Re-rankers | g5.2xlarge (NVIDIA A10G) | Spot Instances (100%) | Stateless GPU inference tasks scaled fast via KEDA/Karpenter. |
Architectural Checklist for Production Readiness
Before launching a production RAG pipeline on AWS EKS, verify that your stack satisfies these infrastructure criteria:
- [ ] Network Latency: Are EKS worker nodes and storage nodes co-located within the same AWS Availability Zone using Cluster Placement Groups?
- [ ] Disk I/O Bottlenecks: Is
gp3storage provisioned with at least 3,000 IOPS and 250 MB/s throughput, or are local NVMe SSDs configured withmmapfor point lookup operations? - [ ] GPU Memory Warmup: Are cold-start delays mitigated using KEDA
minReplicaCountpadding and pre-warmed CUDA contexts on pod creation? - [ ] Distributed Tracing: Are OpenTelemetry span contexts propagated across query expansion, vector lookup, sparse search, and re-ranking phases to monitor latency bottlenecks?
- [ ] Vector Index Optimization: Is Scalar Quantization (
SQ8) or Product Quantization (PQ) enabled on vector indexes exceeding 50M dimensions to maximize RAM hit ratios?