Deploying Large Language Models (LLMs) into production presents a stark economic and architectural reality: GPU instances like NVIDIA A100s, H100s, or L40Ss are exceptionally expensive, and traffic patterns for generative AI applications are notoriously bursty.
If you static-provision your GPU compute for peak load, your cloud bill will explode. Conversely, if you rely on traditional Kubernetes autoscaling mechanisms—like the Horizontal Pod Autoscaler (HPA) driven by CPU/Memory utilization and the classic Cluster Autoscaler—your system will suffer from excruciatingly slow scale-outs, severe thrashing, and unacceptable request latencies.
To achieve cost efficiency without sacrificing SLA, we must build a system capable of zero-downtime, sub-minute predictive auto-scaling.
In this article, we will architect a production-grade LLM inference infrastructure on Amazon EKS using vLLM for optimized serving, Karpenter for just-in-time GPU node provisioning, and KEDA (Kubernetes Event-driven Autoscaling) driven by custom vLLM metrics.
Architectural Overview
The target topology decouples the control plane (request queue management and metric evaluation) from the data plane (vLLM inference pods dynamically scheduled on EC2 GPU instances via Karpenter).
[ Client Requests ]
│
▼
[ AWS ALB / NGINX Ingress ]
│
▼
┌────────────────────────────────────────────────────────┐
│ vLLM Deployment (EKS) │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ vLLM Pod #1 │ │ vLLM Pod #2 │ │ vLLM Pod #N │ │
│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ │
└──────────┼─────────────────┼─────────────────┼──────────┘
│ │ │
└─────────────────┼─────────────────┘
│ Exposes Metrics (:8000/metrics)
▼
[ Prometheus / Mimir ]
│
▼ Evaluates Queue Depth
[ KEDA Operator ]
│
▼ Scales Deployment Replicas
[ Kubernetes Control Plane ]
│
▼ Unschedulable Pods Trigger
[ Karpenter Controller ]
│
▼ Provisions Target GPU Nodes (JIT)
[ EC2 Spot / On-Demand Instances ]
(e.g., g5.2xlarge, g5.12xlarge, p4d.24xlarge)
Components Breakdown
- vLLM Engine: Handles high-throughput serving via PagedAttention and continuous batching, exposing internal queue and KV cache metrics via a
/metricsPrometheus endpoint. - KEDA (Kubernetes Event-driven Autoscaling): Polls Prometheus for custom vLLM queue metrics to drive rapid Horizontal Pod Autoscaling, bypassing generic CPU/RAM metrics.
- Karpenter v1.x: Evaluates pending unschedulable vLLM pods and provisions the exact EC2 GPU instances required in seconds, leveraging node consolidation and Spot/On-Demand mix strategies.
- Fast Storage Layer: Amazon S3 + Mountpoint for Amazon S3 / NVMe Instance Store caching to minimize model weight loading times during pod initialization.
Why Standard Autoscaling Fails for LLMs
To understand why this architecture is mandatory, we must examine how vLLM allocates hardware resources compared to traditional HTTP microservices.
The PagedAttention Allocation Paradox
Standard Kubernetes HPAs scale workloads based on metrics like container_cpu_allocation or container_gpu_memory_used. However, vLLM utilizes PagedAttention, which manages Key-Value (KV) cache memory dynamically in fixed-size blocks.
When a vLLM container initializes, it pre-allocates almost the entirety of the specified GPU VRAM (--gpu-memory-utilization, defaulting to 0.90 or 90%) to hold the KV cache and model weights.
+-------------------------------------------------------------------------+
| Physical GPU VRAM (24GB) |
+------------------------------------+------------------------------------+
| Model Weights (e.g., Llama-3-8B) | Pre-allocated KV Cache (vLLM) |
| ~16 GB | ~5.6 GB |
+------------------------------------+------------------------------------+
|<----------------------- ALWAYS SHOWS ~90% USED ------------------------>|
As a result:
- The GPU memory utilization metric will read ~90% constant utilization regardless of whether the pod is idle or handling 100 concurrent requests.
- Scaling on standard GPU memory usage causes the HPA to immediately max out replicas upon startup.
The Metric That Matters: Queue Depth & KV Cache Pressure
Instead of hardware utilization, LLM autoscaling must be driven by execution queue saturation. vLLM exposes specific metrics for this:
vllm:num_requests_waiting: The number of requests currently sitting in the processing queue waiting for GPU KV cache allocations.vllm:gpu_cache_usage_perc: The percentage of pre-allocated KV cache blocks currently occupied by active context windows.
If vllm:num_requests_waiting > 0, your current GPU capacity cannot process incoming context windows in real-time, directly degrading Time-to-First-Token (TTFT) and Inter-Token Latency (ITL).
Step 1: Deploying vLLM with Optimal Parameters on EKS
Let's start by configuring the vLLM deployment manifest. We will deploy Meta-Llama-3-8B-Instruct using dynamic local NVMe mounts for cache storage.
vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama3-8b
namespace: llm-serving
labels:
app.kubernetes.io/name: vllm-llama3-8b
spec:
replicas: 1
selector:
matchLabels:
app: vllm-llama3-8b
template:
metadata:
labels:
app: vllm-llama3-8b
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
spec:
# Ensure pods drop cleanly during node drain/consolidation
terminationGracePeriodSeconds: 120
containers:
- name: vllm-engine
image: vllm/vllm-openai:v0.6.3
args:
- "--model"
- "meta-llama/Meta-Llama-3-8B-Instruct"
- "--port"
- "8000"
- "--gpu-memory-utilization"
- "0.90"
- "--max-model-len"
- "8192"
- "--max-num-batched-tokens"
- "8192"
- "--download-dir"
- "/tmp/hf-cache"
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
key: token
ports:
- containerPort: 8000
name: http
resources:
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
requests:
cpu: "6"
memory: "24Gi"
nvidia.com/gpu: "1"
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 90
periodSeconds: 10
volumeMounts:
- mountPath: /tmp/hf-cache
name: model-cache
volumes:
- name: model-cache
emptyDir: {}
Step 2: Configuring KEDA for Metric-Driven Pod Scaling
Now that vLLM is exposing /metrics, we install KEDA and create a ScaledObject. This monitors Prometheus and triggers pod scaling as soon as vllm:num_requests_waiting increases.
keda-vllm-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-llama3-8b-autoscaler
namespace: llm-serving
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: vllm-llama3-8b
minReplicaCount: 1
maxReplicaCount: 10
cooldownPeriod: 300 # Prevent rapid scale-down thrashing (5 mins)
pollingInterval: 5 # Aggressive polling for rapid response
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
metricName: vllm_num_requests_waiting
query: sum(vllm:num_requests_waiting{namespace="llm-serving", pod=~"vllm-llama3-8b-.*"})
threshold: '2.0' # Scale up if total queued requests across instances > 2
- type: prometheus
metadata:
serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
metricName: vllm_gpu_cache_usage_perc
query: avg(vllm:gpu_cache_usage_perc{namespace="llm-serving", pod=~"vllm-llama3-8b-.*"})
threshold: '0.85' # Scale up if average KV cache utilization exceeds 85%
Step 3: Fast GPU Provisioning with Karpenter v1.x
When KEDA detects high queue depth, it immediately increments the Deployment replica count. Because GPU resources in the cluster are typically fully committed, these new pods are marked Pending by the K8s scheduler.
Traditional Cluster Autoscaler would take 3-5 minutes to evaluate node groups and trigger an EC2 Auto Scaling Group update. Karpenter evaluates unschedulable pod manifests in milliseconds and calls EC2 Fleet APIs directly.
Here is the production Karpenter manifest configured for GPU inference nodes using the karpenter.sh/v1 API.
karpenter-gpu-nodepool.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: gpu-inference-nodepool
spec:
template:
metadata:
labels:
workload: llm-inference
accelerator: nvidia-gpu
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: gpu-ec2nodeclass
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand", "spot"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["g", "p"]
- key: karpenter.k8s.aws/instance-family
operator: In
values: ["g5", "g6"] # EC2 G5 (A10G) and G6 (L4)
- key: karpenter.k8s.aws/instance-size
operator: In
values: ["2xlarge", "4xlarge", "12xlarge"]
taints:
- key: nvidia.com/gpu
value: "true"
effect: NoSchedule
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m
expireAfter: 7d
limits:
nvidia.com/gpu: "32" # Limit total cluster GPUs for cost safety
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: gpu-ec2nodeclass
spec:
amiSelectorTerms:
- alias: al2023@latest # Or Bottlerocket NVIDIA variants
role: "KarpenterNodeRole-EKS-Cluster"
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: "production-eks-cluster"
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: "production-eks-cluster"
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 150Gi
volumeType: gp3
iops: 3000
throughput: 250
deleteOnTermination: true
userData: |
#!/bin/bash
# Optimize NVMe local instance storage for model caching if available
if lsblk | grep -q nvme1n1; then
mkfs.ext4 /dev/nvme1n1
mkdir -p /tmp/hf-cache
mount /dev/nvme1n1 /tmp/hf-cache
chmod 777 /tmp/hf-cache
fi
To allow the vLLM deployment to land on these Karpenter-provisioned nodes, update your vllm-deployment.yaml spec with the corresponding tolerations and node affinity:
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
nodeSelector:
workload: llm-inference
Step 4: Eliminating Cold Starts & Achieving Zero Downtime
Scale-out latency for LLM pods consists of three phases:
$$\text{Total Scale Out Latency} = T_{\text{node_provision}} + T_{\text{container_pull}} + T_{\text{model_load}}$$
[ Karpenter Node Provisioning: 30-45s ] ──► [ Container Image Pull: 10-15s ] ──► [ Model Weights Load: 15-45s ] ──► [ Ready ]
To achieve sub-60-second operational readiness, we must optimize each stage of this lifecycle.
Strategy 1: Shared Model Caching via Amazon S3 Mountpoint or FSx for Lustre
Pulling 16GB–140GB of weights from Hugging Face Hub directly inside the container during cold-starts causes high latency and risk of API rate-limiting.
Instead, pre-stage raw model safetensors in Amazon S3 or FSx for Lustre, and mount them read-only into the node using Mountpoint for Amazon S3 CSI Driver.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: s3-llama3-weights-pvc
namespace: llm-serving
spec:
accessModes:
- ReadOnlyMany
storageClassName: s3-csi
resources:
requests:
storage: 500Gi
This bypasses network downloads entirely at pod boot; weights are streamed into GPU VRAM directly via high-speed AWS network interfaces.
Strategy 2: Graceful Termination and Connection Draining
When KEDA triggers a scale-down, or Karpenter consolidates an underutilized GPU node, active streaming LLM requests must not be severed mid-generation.
To ensure zero-downtime during scale-down operations:
- Enable
terminationGracePeriodSeconds: 120on the Pod spec. - Configure a PreStop lifecycle hook to drain incoming requests cleanly.
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"] # Gives ingress time to remove endpoint from LB
- Ensure Karpenter observes the Pod Disruption Budgets (PDB):
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: vllm-pdb
namespace: llm-serving
spec:
minAvailable: 1
selector:
matchLabels:
app: vllm-llama3-8b
Production Verification: Stress-Testing the Pipeline
To test this architecture under realistic load conditions, we can execute a continuous execution test using locust or k6 targeting the OpenAI-compatible /v1/completions endpoint exposed by vLLM.
1. Initial State (Baseline Load)
- Active Pods: 1 vLLM instance running on a
g5.2xlarge(1x NVIDIA A10G 24GB). - Metrics:
vllm:num_requests_waiting= 0,vllm:gpu_cache_usage_perc= 12%. - Karpenter Nodes: 1 active node.
2. Injecting Traffic Burst (Simulating 50 Concurrent Clients)
- 00:05s: Request concurrency surges.
vllm:num_requests_waitingjumps to 18. - 00:10s: KEDA detects queue depth exceeding threshold (
2.0) and calculates target replicas: $$\text{Target Replicas} = \left\lceil \text{Current Replicas} \times \left( \frac{\text{Current Metric Value}}{\text{Target Value}} \right) \right\rceil = \left\lceil 1 \times \left( \frac{18}{2} \right) \right\rceil = 9 \text{ replicas}$$ Deployment scaled from 1 to 9 replicas. - 00:12s: 8 new vLLM Pods enter
Pendingstate due to missing GPU capacity. - 00:14s: Karpenter intercepts pending pods, computes optimal node bin-packing, and issues raw
EC2:CreateFleetAPI calls for 8xg5.2xlargeinstances (or equivalent multi-GPU instances likeg5.12xlarge). - 00:48s: EC2 instances initialize, pass K8s node registration, and join cluster.
- 01:05s: Pods complete startup, mount S3 storage, pre-load model into GPU memory, and pass health checks.
- 01:10s: Ingress begins distributing traffic. Queue depth drops back to 0.
3. Traffic Recedes (Scale-Down and Consolidation)
- 10:00m: Traffic drops off. Queue depth remains 0 for > 5 minutes.
- 15:00m: KEDA smoothly scales down Deployment replicas from 9 to 1.
- 20:00m: Karpenter marks empty GPU nodes as underutilized and terminates them via node expiration policies, returning the infrastructure back to its minimal cost footprint.
Key Takeaways for Production Deployments
- Never scale LLMs on standard CPU/GPU utilization metrics. Use custom internal metrics (
vllm:num_requests_waitingandvllm:gpu_cache_usage_perc) via KEDA to react to real execution bottlenecks. - Decouple Node Provisioning from static Auto Scaling Groups. Leverage Karpenter to dynamically select from diverse GPU instance families (
g5,g6,p4) and capacity types (Spotvs.On-Demand) based on real-time availability. - Eliminate image and weight fetch bottlenecks. Use specialized local storage strategies, such as S3 Mountpoint combined with NVMe instance caches, to reduce cold-start duration.
- Enforce Graceful Termination. Set high
terminationGracePeriodSecondsalongside Pod Disruption Budgets so active streaming generation loops finish cleanly when Karpenter consolidates underutilized GPU instances.