Architecting High-Throughput LLM Inference Pipelines on AWS EKS with vLLM and Ray

Serving foundation models in production at scale presents a stark engineering paradox: while Large Language Models (LLMs) are exponentially transforming user experiences, their operational demands are brutal on infrastructure. Traditional stateless web services scale gracefully on CPU and RAM metrics. LLM workloads, by contrast, are fundamentally constrained by GPU VRAM capacity, High Bandwidth Memory (HBM) throughput, and complex context-state management.

When serving models like Llama-3-70B or Mixtral-8x22B to thousands of concurrent users, two primary performance metrics dictate success:

1. Time-to-First-Token (TTFT): The time taken to process the prompt context (Prefill phase) and output the initial token. TTFT is largely compute-bound.
2. Inter-Token Latency (ITL) / Throughput: The generation speed per user (Decode phase) and overall tokens per second across the cluster. ITL is strictly memory-bandwidth bound.

To maximize throughput and minimize TTFT without over-provisioning million-dollar GPU clusters, you must pair an intelligent inference engine with a resilient orchestrator.

In this guide, we will architect a production-ready, highly available, and auto-scaling LLM inference pipeline on AWS Elastic Kubernetes Service (EKS) using vLLM and Ray, managed via Terraform and KEDA.

Architectural Overview

The core architecture uses a decoupling strategy: AWS EKS handles cluster topology and hardware scheduling, KubeRay provisions and manages distributed Ray clusters, vLLM executes high-throughput inference using continuous batching and PagedAttention, and KEDA dynamically scales GPU worker nodes based on real-time inference queue telemetry.

flowchart TD
    Client[Client / Application Service] --> ALB[AWS Application Load Balancer]
    
    subgraph EKS["AWS EKS Cluster (Control Plane)"]
        Ingress[AWS Load Balancer Controller]
        KEDA[KEDA Scaler Controller]
        KubeRay[KubeRay Operator]
    end

    ALB --> Ingress
    Ingress --> RayHead[Ray Head Node Service]

    subgraph RayCluster["Distributed Ray Cluster (GPU Worknodes)"]
        RayHead -->|Orchestrates Workloads| Worker1["Ray Worker (g5.12xlarge / p4d.24xlarge)
        vLLM Engine Engine Instance (TP=4)"]
        RayHead -->|Orchestrates Workloads| Worker2["Ray Worker (g5.12xlarge / p4d.24xlarge)
        vLLM Engine Engine Instance (TP=4)"]
    end

    Prometheus[Prometheus Server] <--Scrapes Metrics-- Worker1
    Prometheus <--Scrapes Metrics-- Worker2
    Prometheus -->|Provides vllm:num_requests_waiting| KEDA
    KEDA -->|Triggers Scaling| KubeRay

    S3[AWS S3 / Mountpoint S3] -->|Fast Weight Loading| Worker1
    S3 -->|Fast Weight Loading| Worker2

Why vLLM and Ray?

• vLLM: Traditional inference engines suffer from extreme memory waste due to key-value (KV) cache fragmentation. vLLM implements PagedAttention, an algorithm inspired by virtual memory paging in operating systems. It allows KV caches to be stored in non-contiguous physical memory spaces, reducing VRAM waste from >60% to under 4%. Furthermore, vLLM utilizes continuous batching (iteration-level scheduling) rather than request-level batching, ensuring maximum GPU compute utilization.
• Ray & KubeRay: When models exceed the VRAM capacity of a single GPU (e.g., Llama-3-70B FP16 requires ~140GB VRAM just for weights), you must partition the model across multiple GPUs using Tensor Parallelism (TP) or Pipeline Parallelism (PP). Ray provides the distributed execution framework to coordinate processes across multiple GPUs and nodes seamlessly, while the KubeRay Operator translates Kubernetes custom resources into stable Ray clusters.

Infrastructure Provisioning with Terraform

We will provision an EKS cluster with two specialized node groups:

1. Core Node Group: CPU-only nodes running Bottlerocket for system components, KubeRay controllers, Prometheus, and KEDA.
2. GPU Node Group: Bottlerocket-based, GPU-optimized instances (e.g., g5.12xlarge with 4x NVIDIA A10G GPUs or p4d.24xlarge with 8x NVIDIA A100 GPUs) utilizing Elastic Fabric Adapter (EFA) for low-latency inter-node communication and local NVMe storage for model caching.

Terraform Module: EKS Node Group Configuration

# main.tf

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "production-llm-cluster"
  cluster_version = "1.29"

  cluster_endpoint_public_access = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Enable AWS VPC CNI Prefix Delegation for high pod density
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET      = "1"
        }
      })
    }
    aws-ebs-csi-driver = {
      most_recent = true
    }
  }

  eks_managed_node_groups = {
    # System workloads (Control Loop, KEDA, KubeRay Operator)
    system_nodes = {
      min_size     = 2
      max_size     = 4
      desired_size = 2

      instance_types = ["m6i.xlarge"]
      ami_type       = "BOTTLEROCKET_x86_64"

      labels = {
        "workload.type" = "system"
      }
    }

    # High-Throughput GPU Inference Nodes
    gpu_nodes_g5 = {
      min_size     = 1
      max_size     = 10
      desired_size = 2

      # 4x NVIDIA A10G (96GB VRAM Total), 48 vCPUs, 192GB RAM
      instance_types = ["g5.12xlarge"] 
      ami_type       = "BOTTLEROCKET_x86_64_NVIDIA"

      # Provision local NVMe ephemeral storage for fast model loading
      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 100
            volume_type           = "gp3"
            iops                  = 3000
            throughput            = 125
            delete_on_termination = true
          }
        }
      }

      labels = {
        "node.kubernetes.io/instance-type" = "g5.12xlarge"
        "accelerator"                      = "nvidia-a10g"
        "workload.type"                    = "llm-inference"
      }

      taints = [
        {
          key    = "nvidia.com/gpu"
          value  = "true"
          effect = "NO_SCHEDULE"
        }
      ]

      # Enable EFA for multi-node distributed training/inference
      enable_efa_support = true
    }
  }
}

Deploying Distributed vLLM via KubeRay

To deploy model endpoints reliably, we use the RayCluster Custom Resource Definition (CRD) managed by KubeRay. Below is an enterprise manifest setting up a Ray Cluster designed for serving Llama-3-70B-Instruct across 4x A10G GPUs (Tensor Parallelism = 4) on a single g5.12xlarge instance.

KubeRay Cluster & Service Manifest

# vllm-ray-cluster.yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: vllm-llama3-70b
  namespace: llm-serving
spec:
  rayVersion: '2.9.0'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
      num-gpus: '0' # Head node acts pure control plane
    template:
      spec:
        nodeSelector:
          workload.type: system
        containers:
          - name: ray-head
            image: rayproject/ray:2.9.0-py310
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
              requests:
                cpu: "2"
                memory: "8Gi"
            ports:
              - containerPort: 6379
                name: gcs
              - containerPort: 8265
                name: dashboard
              - containerPort: 8000
                name: serve
  workerGroupSpecs:
    - groupName: gpu-group
      replicas: 1
      minReplicas: 1
      maxReplicas: 5
      rayStartParams:
        block: 'true'
      template:
        spec:
          nodeSelector:
            workload.type: llm-inference
            node.kubernetes.io/instance-type: g5.12xlarge
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          volumes:
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 16Gi
            - name: model-cache
              hostPath:
                path: /mnt/k8s-disks/by-name/ephemeral0
                type: DirectoryOrCreate
          containers:
            - name: vllm-worker
              image: vllm/vllm-openai:v0.4.2
              securityContext:
                capabilities:
                  add: ["SYS_PTRACE"]
              resources:
                limits:
                  cpu: "40"
                  memory: "160Gi"
                  nvidia.com/gpu: "4"
                requests:
                  cpu: "32"
                  memory: "128Gi"
                  nvidia.com/gpu: "4"
              volumeMounts:
                - mountPath: /dev/shm
                  name: dshm
                - mountPath: /root/.cache/huggingface
                  name: model-cache
              env:
                - name: HUGGING_FACE_HUB_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: hf-token-secret
                      key: token
              # Command launching vLLM engine inside Ray worker context
              command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
              args:
                - "--model=meta-llama/Meta-Llama-3-70B-Instruct"
                - "--tensor-parallel-size=4"
                - "--gpu-memory-utilization=0.92"
                - "--max-model-len=8192"
                - "--max-num-batched-tokens=32768"
                - "--trust-remote-code"
                - "--port=8000"
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama3-service
  namespace: llm-serving
spec:
  ports:
    - name: http
      port: 8000
      targetPort: 8000
  selector:
    ray.io/node-type: head
  type: ClusterIP

Intelligent Scaling Strategy: KEDA + Custom vLLM Metrics

Standard horizontal pod autoscalers (HPA) scale on CPU or RAM—metrics that are completely decoupled from LLM demand. Scaling on raw GPU Duty Cycle (container_gpu_utilization) is equally flawed, as a GPU processing a single token will report 100% compute utilization, causing premature upscaling.

The standard metrics for scaling LLM inference engines are:

1. vllm:num_requests_waiting: The number of requests stuck in the iteration scheduler's queue awaiting processing. If this is $> 0$, your current capacity is fully saturated.
2. vllm:gpu_cache_usage_perc: The percentage of KV-cache memory blocks currently occupied. As this approaches $1.0$ (100%), continuous batching fails, forcing the engine to preempt or swap requests to CPU RAM, spiking TTFT massively.

1. Prometheus Scraping Configuration

vLLM exposes Prometheus metrics natively on port 8000 at /metrics. We deploy a PodMonitor or ServiceMonitor to aggregate these:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: vllm-prometheus-monitor
  namespace: llm-serving
spec:
  selector:
    matchLabels:
      ray.io/node-type: worker
  podMetricsEndpoints:
    - port: http
      path: /metrics
      interval: 5s

2. KEDA ScaledObject Configuration

We configure KEDA (Kubernetes Event-driven Autoscaling) to evaluate the vLLM request queue depth. If the pending request count exceeds an average of 5 per worker pod over a 30-second window, KEDA provisions additional GPU nodes via the RayCluster object.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: ray.io/v1
    kind: RayCluster
    name: vllm-llama3-70b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 600 # Conservative scale-down to prevent cache thrashing
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting{namespace="llm-serving"}) / count(vllm:num_requests_waiting{namespace="llm-serving"})
        threshold: '5'

Mitigating Cold Starts & Optimizing Storage I/O

A major bottleneck in auto-scaling high-throughput LLM clusters is Cold Start Latency. Pulling a 140GB unquantized Llama-3-70B model checkpoint from Hugging Face or standard EBS volumes over the network during a scale-up event can easily take 10 to 20 minutes.

To drive this down to sub-minute durations, combine Mountpoint for Amazon S3 with Local NVMe ephemeral cache pre-warming.

sequenceDiagram
    autonumber
    participant K8s as EKS Provisioner
    participant Node as GPU Node (Bottlerocket)
    participant NVMe as Local NVMe Array
    participant S3 as AWS S3 (Model Store)
    participant vLLM as vLLM Engine

    K8s->>Node: Spin up Instance (g5.12xlarge)
    Node->>NVMe: Format & Mount ephemeral disks (/mnt/k8s-disks)
    
    alt Model Cached in Local NVMe
        vLLM->>NVMe: Read Weights directly via Direct-I/O
        Note over vLLM,NVMe: Load time: ~12 Seconds
    else Cache Miss
        vLLM->>S3: Stream Weights via Mountpoint S3 Driver
        S3-->>vLLM: High Bandwidth Concurrent S3 Stream
        vLLM->>NVMe: Populate Local Cache for Future Runs
        Note over vLLM,S3: Load time: ~45 Seconds
    end
    
    vLLM->>vLLM: Initialize PagedAttention KV Cache VRAM
    vLLM-->>K8s: Ready to serve traffic

Storage Optimization Steps:

1. Store Weights in Safetensors Format: Convert legacy PyTorch .bin files to .safetensors. Safetensors maps directly to memory (zero-copy deserialization), enabling dramatically faster load times.
2. Mountpoint for Amazon S3 CSIDriver: Mount an S3 bucket containing your .safetensors files directly into the pod using the AWS S3 CSI driver. This leverages S3's massive aggregate throughput across multiple parallel HTTP requests, bypassing the IOPS caps of standard EBS volumes.

Production Benchmarking & Tuning Playbook

To validate architectural performance, run continuous synthetic benchmarks measuring TTFT and Throughput under varying concurrency using vllm-benchmark or Locust.

Running a Benchmark Test

Run the vLLM benchmark tool inside your cluster to simulate heavy multi-user context-length loads:

python3 benchmarks/benchmark_serving.py \
    --host vllm-llama3-service.llm-serving.svc.cluster.local \
    --port 8000 \
    --backend vllm \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --dataset-name sharegpt \
    --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 1000 \
    --request-rate 20 # 20 Requests Per Second

Inference Hyperparameter Tuning Matrix

Fine-tuning parameters in vLLM determines how aggressively your engine utilizes available VRAM for KV-cache versus batch compute sizing:

| Hyperparameter | Recommended Value | Operational Impact | | :--- | :--- | :--- | | --gpu-memory-utilization | 0.92 to 0.95 | Allocates up to 95% of total VRAM strictly for model weights and KV cache. Reserve ~5-8% for CUDA context overhead to prevent Out-Of-Memory (OOM) crashes. | | --max-model-len | 8192 (or target SLA context) | Restricts context window cap. Allocating max sequence context (e.g., 32k or 128k) when unnecessary linearly degrades total available KV-cache blocks for concurrent users. | | --max-num-batched-tokens | 32768 | Defines maximum tokens processed in a single continuous iteration across all sequences in Prefill phase. Higher values improve GPU throughput but spike TTFT for newly arrived requests. | | --max-num-seqs | 256 | Sets hard limit on concurrent streams processed per engine instance. Protects Inter-Token Latency (ITL) under sudden context spikes. | | --enable-chunked-prefill | true | Breaks down large prompts into smaller chunks, interleaving prefill compute with ongoing generation decode phases. Massively reduces TTFT spikes. |

Architectural Checklist for Production Readiness

Before routing production live user traffic to your self-hosted LLM EKS pipeline, verify the following baseline requirements:

• [ ] Prefix Delegation Enabled: Ensure the AWS VPC CNI uses prefix delegation (ENABLE_PREFIX_DELEGATION=true) to avoid IP exhaustion on multi-pod GPU instances.
• [ ] Shared Memory Allocations: Ensure /dev/shm is mounted as an emptyDir with medium: Memory sized to at least 16GB. Ray NCCL distributed processes will fail unexpectedly with low shared memory allocations.
• [ ] Cluster Autoscaler Graceful Termination: Configure KEDA and Karpenter/EKS Managed Node Groups to drain nodes cleanly, allowing current generation sequences to finish before terminating GPU instances.
• [ ] Chunked Prefill Activation: Enable vLLM's chunked prefill (--enable-chunked-prefill=true) for workloads with mixed long-context prompts and short-latency generations.
• [ ] Distributed Cache Warmup: Implement init-containers to fetch or map model .safetensors files from S3 to local NVMe SSDs before spinning up engine runtime logic.