Architecting Low-Latency, Cost-Efficient LLM Inference Pipelines on AWS EKS

Running Large Language Models (LLMs) in production is an exercise in managing aggressive trade-offs. On one hand, your product team demands sub-100 millisecond Time-to-First-Token (TTFT) and high token generation throughput. On the other hand, your finance department is watching your AWS bill escalate as high-memory GPU instances sit idle at 2 AM, waiting for sporadic request bursts.

Standard Kubernetes Horizontal Pod Autoscaler (HPA) strategies—built around CPU and Memory metrics—fall apart when applied to accelerated compute workloads. GPU memory (VRAM) allocation in LLM serving engines is largely static, as KV caches and model weights are pre-allocated upon pod initialization. Furthermore, naive node provisioning using Legacy Cluster Autoscaler is far too slow to handle the volatile traffic spikes characteristic of AI applications.

In this deep dive, we will design and deploy a production-grade, battle-tested LLM inference architecture on Amazon EKS. We will pair vLLM (for engine-level efficiency via PagedAttention) with Karpenter (for sub-minute GPU node provisioning) and KEDA (for metric-driven request-queue autoscaling).

Architectural Topology Overview

To achieve both high throughput and aggressive cost reduction, our infrastructure separates the orchestration layer from the heterogeneous accelerated hardware layer.

                     [ Ingress / ALB ]
                             │
                             ▼
               [ Istio Service Mesh / Router ]
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   [ KEDA Metrics Server ]          [ Prometheus / Grafana ]
     (Scrapes vLLM Queue)                     │
            │                                 │
            ▼                                 ▼
   [ HPA / ScaledObject ]            [ Observability ]
            │
            ▼
   [ vLLM Pod Deployments (AWS EKS) ]
            │
            ├── Node Provisioning Requests ──┐
            ▼                                ▼
┌───────────────────────┐        ┌────────────────────────┐
│  Karpenter Controller │        │  AWS EC2 Spot / On-Dem │
└───────────┬───────────┘        └───────────┬────────────┘
            │                                │
            └───────────────┬────────────────┘
                            ▼
        ┌───────────────────────────────────────┐
        │       Target GPU / Accelerators       │
        ├───────────────────┬───────────────────┤
        │ NVIDIA Tensor Core│ AWS Inferentia2   │
        │ (g5.xlarge/g6)    │ (inf2.8xlarge)    │
        └───────────────────┴───────────────────┘

Components Breakdown:

1. Inference Engine (vLLM): Replaces Hugging Face Transformers. Utilizes PagedAttention and continuous batching to maximize VRAM utilization and token throughput.
2. Node Provisioner (Karpenter): Bypasses Kubernetes NodeGroups to directly launch optimal EC2 instances (G5, G6, Inf2) based on un-schedulable pod constraints within 30–45 seconds.
3. Autoscaler (KEDA): Scales pods based on actual vLLM engine metrics (vllm:num_requests_waiting) rather than GPU duty cycles or CPU utilization.
4. Hardware Tier: Blends On-Demand instances for baseline capacity and Spot instances for burst capacity, utilizing multiple GPU families.

Engine Optimization: Why vLLM Matters

Traditional inference pipelines suffer from VRAM fragmentation. When serving models like Llama-3-70B or Mistral-7B, Key-Value (KV) cache memory dominates execution footprints. Standard approaches allocate contiguous memory blocks for the maximum sequence length, wasting up to 60–80% of unutilized VRAM per request.

Traditional KV Cache Allocation (Contiguous, Fragmented):
[ Token 1-50 (Used) ][               Reserved / Wasted VRAM (Unused)               ]

vLLM PagedAttention Allocation (Non-Contiguous, Dynamic Blocks):
[ Block 0 (Used) ] -> [ Block 4 (Used) ] -> [ Block 12 (Allocated on-demand) ]

Key vLLM Mechanics:

• PagedAttention: Manages KV cache in virtual memory pages, bringing allocation efficiency close to 100%.
• Continuous Batching: Dynamically inserts incoming requests into running iteration batches instead of waiting for entire batch completion (static batching).
• Tensor Parallelism (TP): Splits model layers across multiple GPUs on a single instance to satisfy memory footprints and accelerate matrix multiplication.

Provisioning Infrastructure with Karpenter

Standard Cluster Autoscaler uses EC2 Auto Scaling Groups (ASGs). When a GPU pod scales up, ASG spinning speeds, node joining delays, and daemonset initialization often take 3–5 minutes. Karpenter talks directly to the AWS EC2 API, evaluating pod resource demands, node selectors, and tolerations to build targeted instance configurations in seconds.

Here is the production-ready Karpenter configuration targeting high-efficiency NVIDIA GPU instances (g5 family with Ampere A10G GPUs).

1. Karpenter EC2NodeClass Manifest

This defines the AWS-specific parameters such as AMI choices, subnet selection, block device mappings (high IOPS NVMe disks for model caching), and security groups.

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: gpu-node-class
spec:
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: eks-cluster-production
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: eks-cluster-production
  instanceProfile: KarpenterNodeInstanceProfile-eks-cluster-production
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 200Gi
        volumeType: gp3
        iops: 3000
        throughput: 250
        encrypted: true
  tags:
    Workload: LLM-Inference
    Environment: Production

2. Karpenter NodePool Manifest

This defines constraints on architecture, capacity types (Spot vs. On-Demand), and specific GPU instance families.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: gpu-nodepool
spec:
  template:
    metadata:
      labels:
        workload: llm-inference
        accelerator: nvidia
    spec:
      nodeClassRef:
        name: gpu-node-class
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: instance.k8s.aws/instance-family
          operator: In
          values: ["g5", "g6"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s
    expireAfter: 720h
  limits:
    nvidia.com/gpu: 64

Deploying vLLM on EKS

Now we construct the Kubernetes deployment spec for a model such as mistralai/Mistral-7B-Instruct-v0.2.

Notice the critical optimizations inside this manifest:

1. Shared memory /dev/shm mount to prevent PyTorch Inter-Process Communication (IPC) memory crashes during multi-GPU processing.
2. gpu-memory-utilization set to 0.90, reserving 10% VRAM headroom for dynamic execution.
3. Node selectors and tolerations matching our Karpenter NodePool.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-mistral-7b
  namespace: llm-serving
  labels:
    app: vllm-mistral-7b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-mistral-7b
  template:
    metadata:
      labels:
        app: vllm-mistral-7b
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      nodeSelector:
        workload: llm-inference
        accelerator: nvidia
      containers:
        - name: vllm-engine
          image: vllm/vllm-openai:v0.4.2
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - "--model=mistralai/Mistral-7B-Instruct-v0.2"
            - "--tensor-parallel-size=1"
            - "--gpu-memory-utilization=0.90"
            - "--max-model-len=8192"
            - "--port=8000"
            - "--enable-chunked-prefill"
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token-secret
                  key: token
            - name: NCCL_DEBUG
              value: "WARN"
          ports:
            - containerPort: 8000
              name: http
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 15
          volumeMounts:
            - mountPath: /dev/shm
              name: dshm
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi

Intelligent Auto-scaling with KEDA

Using CPU or Memory metrics to scale LLMs leads to false triggers or severe capacity lag. Under heavy load, vLLM's CPU usage remains flat while its internal request queue spikes.

vLLM exposes native Prometheus metrics at /metrics. The defining metric for queue health is vllm:num_requests_waiting. If this value stays above 0, requests are actively queuing up, directly degrading TTFT and P99 latency.

We install KEDA (Kubernetes Event-driven Autoscaling) and define a ScaledObject that scales inference pods based on real-time metric thresholds scraped from Prometheus.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-mistral-7b
  minReplicaCount: 2
  maxReplicaCount: 10
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Percent
              value: 10
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting{app="vllm-mistral-7b"})
        threshold: '3'

Cost Optimization Playbook

Running GPU instances full-time in production without structural optimizations will burn through your cloud budget rapidly. Follow these four operational tenets to cut costs by 50–70%.

1. Mixed Spot & On-Demand Provisioning

Deploy baseline traffic (e.g., min capacity = 2 replicas) on On-Demand instances. Configure Karpenter to fulfill horizontal bursts entirely via EC2 Spot instances.

To mitigate interruption risk:

• Ensure model weights are pre-baked into custom AMI instances or backed by local NVMe caches (avoiding slow Hugging Face model downloads during cold starts).
• Intercept AWS Spot Interruption Notifications via karpenter-core to gracefully bleed traffic off marked nodes 120 seconds before termination.

2. AWS Inferentia2 (Inf2) Alternative

For high-volume production models, NVIDIA A10G (g5) GPUs are not always the most cost-effective option. AWS Inferentia2 (inf2 instances powered by AWS Neuron SDK) offers significantly higher performance-per-dollar ratios.

# Example Karpenter requirement targeting AWS Inferentia2
- key: instance.k8s.aws/instance-family
  operator: In
  values: ["inf2"]

Note: Models running on Neuron cores require upfront compilation via optimum-neuron or vLLM-Neuron backends, trading structural runtime flexibility for reduced inference cost.

3. Model Quantization Strategies (AWQ vs. FP8)

Deploying raw FP16 models requires massive VRAM overhead. Modern activation-aware quantization schemes preserve accuracy while lowering memory footprints drastically:

| Precision Format | VRAM Required (7B Model) | Relative Inference Speed | P99 Latency Impact | Recommended Use Case | | :--- | :--- | :--- | :--- | :--- | | FP16 / BF16 | ~16 GB | 1.0x (Baseline) | Low | Mission-critical / High-precision | | FP8 (TransformerEngine)| ~9 GB | 1.4x - 1.8x | Negligible | Default choice for Hopper/Ada GPUs | | AWQ (4-bit) | ~5.5 GB | 1.8x - 2.5x | Minimal | High-concurrency / Cost-sensitive |

To leverage AWQ in vLLM, append --quantization=awq to execution arguments and deploy pre-quantized weights (e.g., TheBloke/Mistral-7B-Instruct-v0.2-AWQ).

Performance Validation & Load Benchmarking

To prove the operational performance of our setup, we execute a concurrency load-test using Python's asyncio and aiohttp against our cluster endpoint. We measure Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT) under sustained stress.

import asyncio
import time
import json
import aiohttp
import numpy as np

URL = "http://<ALB-ENDPOINT>/v1/completions"
HEADERS = {"Content-Type": "application/json"}
PAYLOAD = {
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "prompt": "Explain Quantum Entanglement to a Senior Cloud Architect in 500 words.",
    "max_tokens": 256,
    "stream": True
}

async def send_request(session):
    start_time = time.time()
    ttft = None
    tokens = 0
    
    async with session.post(URL, headers=HEADERS, json=PAYLOAD) as response:
        async for chunk in response.content:
            if chunk:
                if ttft is None:
                    ttft = time.time() - start_time
                tokens += 1
                
    total_time = time.time() - start_time
    tpot = (total_time - ttft) / max(1, tokens - 1) if tokens > 1 else 0
    return ttft, tpot, total_time

async def main(concurrency=50):
    async with aiohttp.ClientSession() as session:
        tasks = [send_request(session) for _ in range(concurrency)]
        results = await asyncio.gather(*tasks)
        
    ttfts = [r[0] for r in results]
    tpots = [r[1] for r in results]
    
    print(f"--- Benchmark Results (Concurrency: {concurrency}) ---")
    print(f"TTFT  - P50: {np.percentile(ttfts, 50)*1000:.2f}ms | P99: {np.percentile(ttfts, 99)*1000:.2f}ms")
    print(f"TPOT  - P50: {np.percentile(tpots, 50)*1000:.2f}ms | P99: {np.percentile(tpots, 99)*1000:.2f}ms")

if __name__ == "__main__":
    asyncio.run(main(concurrency=50))

Operational Readiness Checklist

Before promoting this stack into production environments, ensure you have implemented the following operational controls:

• [ ] DaemonSet Tolerations: Exclude log collection agents and telemetry daemons from taking up GPU resources unless explicitly targeted.
• [ ] Warm Model Caching: Utilize AWS Elastic File System (EFS) or high-speed FSx for ONTAP/Lustre to pre-populate /root/.cache/huggingface across new GPU instances, skipping execution delays during cold boots.
• [ ] Termination Handlers: Deploy karpenter-core disruption controllers to gracefully catch Spot instance reclamation signals (ITN) and execute zero-downtime drain events.
• [ ] Concurrency Tuning: Set vLLM --max-num-seqs parameters proportionally to your target context size to prevent engine-level OOM exceptions during high burst activity.

Conclusion

Combining vLLM, AWS EKS, Karpenter, and KEDA changes the math for running generative AI in production. By switching from standard CPU/Memory auto-scaling to real-time engine queue tracking, you eliminate P99 latency spikes during traffic bursts. At the same time, delegating hardware provisioning to Karpenter allows you to safely utilize EC2 Spot instances and diverse GPU families, keeping compute costs closely tied to actual user demand.