Scaling vLLM on Kubernetes: Architecting Low-Latency LLM Inference at Cloud Scale

In the era of modern AI engineering, moving Large Language Models (LLMs) from experimental notebooks to high-throughput, low-latency production environments is one of the most complex infrastructure challenges enterprise cloud architects face.

Unlike traditional microservices that scale horizontally based on predictable CPU or memory saturation, LLM serving is aggressively bound by GPU VRAM capacity, memory bandwidth, and non-deterministic request lengths. Naively throwing GPUs behind a Kubernetes Service running standard HTTP load balancers yields abysmal resource utilization (often under 30%), catastrophic P99 latency spikes, and ballooning cloud bills.

To achieve cloud-scale efficiency, high-throughput LLM serving requires dynamic memory allocation frameworks like PagedAttention and distributed inference engines like vLLM. In this deep dive, we will architect, deploy, auto-scale, and optimize production-grade vLLM clusters on Kubernetes to maximize GPU compute efficiency while strictly maintaining sub-100ms Time-To-First-Token (TTFT) and low Time-Per-Output-Token (TPOT) targets.

1. The Core Engine: PagedAttention & vLLM Architecture

To understand how to scale vLLM on Kubernetes, we must first understand why traditional LLM inference frameworks choke under variable concurrent workloads.

The KV Cache Bottleneck

During autoregressive LLM inference, the model processes input tokens (Prefill phase) and generates output tokens sequentially (Decode phase). To avoid re-computing attention keys and values for previous tokens at every step, the engine caches these vectors in VRAM—this is the Key-Value (KV) Cache.

In traditional implementations (e.g., naive Hugging Face Transformers):

• KV cache allocations require contiguous memory.
• Memory is pre-allocated based on the maximum possible sequence length (e.g., 4096 or 8192 tokens) rather than the actual sequence length.
• Result: Up to 60–80% of GPU VRAM is wasted due to internal fragmentation (pre-allocated memory unused by short requests) and external fragmentation (inability to fit new requests despite having free scattered VRAM blocks).

Traditional KV Cache Allocation (Contiguous & Oversized)
[ Request 1: Tokens 1-500 | Unused Pre-allocated VRAM (3596 tokens) ] -> WASTED

PagedAttention Allocation (Virtual Memory Paging)
[ Logical Page 0 ] -> Physical Block 12 (VRAM)
[ Logical Page 1 ] -> Physical Block 45 (VRAM)
[ Logical Page 2 ] -> Physical Block 03 (VRAM)

How PagedAttention Solves Memory Bottlenecks

Inspired by virtual memory paging in classical Operating Systems, vLLM introduces PagedAttention. Key-Value matrices are broken down into fixed-size Physical Blocks (typically 16 or 32 tokens).

• Non-contiguous Memory: KV caches are stored in non-contiguous physical memory blocks.
• Dynamic Allocation: Blocks are allocated on-demand as tokens are generated.
• Block Tables: A logical-to-physical mapping layer translates sequence indices to physical GPU memory addresses.

By eliminating memory fragmentation, PagedAttention drops VRAM waste to under 4%, allowing you to increase batch sizes by 2x to 4x on the same hardware, drastically improving system throughput ($QPS$) without degrading individual request latency.

2. Architecting the Kubernetes vLLM Node Topology

When designing a Kubernetes cluster for high-performance vLLM pods, standard node groups are insufficient. You must tailor worker nodes for hardware acceleration and fast inter-process communication.

Prerequisites for GPU Nodes

• GPU Hardware: NVIDIA A10G, L40S, A100 (40GB/80GB), or H100 GPUs.
• Interconnect: NVLink/NVSwitch for multi-GPU setups on a single node; AWS EFA, GCP Fast Socket, or InfiniBand for multi-node setups.
• Host Provisioning: NVIDIA GPU Operator installed on K8s to expose standard driver libraries and nvidia.com/gpu extended resources.

Crucial Production Gotchas: Shared Memory (/dev/shm)

When running multi-GPU tensor parallelism (TP), vLLM leverages NCCL (NVIDIA Collective Communications Library) or PyTorch distributed backends. These processes require extensive Shared Memory (/dev/shm).

By default, Docker and Kubernetes assign a restrictive 64MB to /dev/shm, which causes NCCL processes to crash with SIGSEGV or silent execution hangs under batch load. You must mount an emptyDir backed by RAM (medium: Memory) to /dev/shm.

3. Production Deployment Specification

Below is a production-grade Kubernetes Deployment manifest designed for a single-node, 4-GPU configuration running a quantized Meta-Llama-3-70B-Instruct model using AWQ or FP8 quantization.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-70b
  namespace: llm-serving
  labels:
    app.kubernetes.io/name: vllm-llama3-70b
    app.kubernetes.io/part-of: ecstaticloud-ai
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-llama3-70b
  template:
    metadata:
      labels:
        app: vllm-llama3-70b
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: vllm-server
        image: vllm/vllm-openai:v0.6.3.post1
        imagePullPolicy: IfNotPresent
        command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
        args:
        - "--model=neuralmagic/Meta-Llama-3-70B-Instruct-FP8"
        - "--tensor-parallel-size=4"
        - "--max-model-len=8192"
        - "--gpu-memory-utilization=0.92"
        - "--max-num-batched-tokens=32768"
        - "--enable-chunked-prefill=true"
        - "--port=8000"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-secret
              key: token
        - name: NCCL_DEBUG
          value: "INFO"
        ports:
        - name: http
          containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: "4"
            memory: "128Gi"
            cpu: "16"
          requests:
            nvidia.com/gpu: "4"
            memory: "64Gi"
            cpu: "8"
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /root/.cache/huggingface
          name: model-cache
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 180
          periodSeconds: 15
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 16Gi
      - name: model-cache
        persistentVolumeClaim:
          claimName: nfs-model-cache-pvc
      nodeSelector:
        node.kubernetes.io/instance-type: g5.12xlarge # Example for AWS 4xA10G
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"

4. Intelligent Routing and Load Balancing

Standard Layer 4 (TCP) or naive Layer 7 HTTP round-robin load balancing (e.g., basic Kubernetes Services) is inefficient for LLM workloads.

Why Standard Round-Robin Fails

1. Variable Execution Time: Request processing time is proportional to $(Prompt Tokens + Completion Tokens)$. One request might take 50ms (short context), while another takes 15s (deep reasoning on 8k context).
2. State Cache Misses: Round-robin breaks continuous session efficiency. If the same user sends subsequent follow-up prompts, routing to a random pod invalidates potential prefix-cache hits in GPU VRAM.

The Solution: Metric-Aware Router or Proxy

Deploy a lightweight intermediate proxy (e.g., custom Envoy filter, SGLang Router, or a dedicated vLLM Router) that interacts directly with vLLM engine metrics.

                      [ Incoming API Requests ]
                                 │
                                 ▼
                     [ Smart LLM Routing Proxy ]
                         │               │
      (Least KV Cache Alloc)           (Least Active Requests)
                         │               │
                         ▼               ▼
                 ┌──────────────┐ ┌──────────────┐
                 │ vLLM Pod #1  │ │ vLLM Pod #2  │
                 └──────────────┘ └──────────────┘

The router should inspect the real-time status exported at /metrics:

• vllm:num_requests_waiting: Pending requests in the queue.
• vllm:gpu_cache_usage_perc: VRAM KV cache percentage in use.

Route incoming traffic to the pod with the lowest num_requests_waiting, and fallback to gpu_cache_usage_perc as a tie-breaker.

5. Metric-Driven Dynamic Autoscaling with KEDA

Autoscaling LLM pods on standard metrics like CPU or Memory utilization is useless—a GPU worker’s CPU usage remains flat regardless of whether the model is idling or actively decoding at maximum batch size.

To auto-scale properly, we utilize KEDA (Kubernetes Event-driven Autoscaling) targeting native vLLM Prometheus metrics.

Key Metrics to Track

1. vllm:num_requests_waiting: Immediate signal of backlog. If $>0$, your cluster is over capacity, and requests are queuing (increasing TTFT).
2. vllm:gpu_cache_usage_perc: Direct memory saturation. If this reaches $>0.85$, PagedAttention will start preempting or swapping sequences to CPU RAM, destroying TPOT.

KEDA ScaledObject Configuration

Here is a production KEDA ScaledObject that scales out when the queue backlog increases or KV cache approaches saturation:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama3-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-70b
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 300 # Prevent premature scale-down due to long model warmups
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600 # 10-min window to avoid thrashing
          policies:
          - type: Percent
            value: 25
            periodSeconds: 60
        scaleUp:
          stabilizationWindowSeconds: 0 # Fast scale-up on traffic spikes
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: |
        sum(vllm:num_requests_waiting{app="vllm-llama3-70b"}) 
        / 
        count(vllm:num_requests_waiting{app="vllm-llama3-70b"})
      threshold: '2.0' # Scale out if avg waiting requests per pod > 2
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_gpu_cache_usage_perc
      query: |
        max(vllm:gpu_cache_usage_perc{app="vllm-llama3-70b"})
      threshold: '0.85' # Scale out if any pod's VRAM cache hits 85%

6. Cold Start Mitigation Strategies

Scaling LLM pods dynamically introduces a severe bottleneck: Cold Start Latency. Pulling a 70B model checkpoint (~140GB in FP16, ~35GB in FP8) from Hugging Face or S3 onto a new node can take 5 to 15 minutes.

To maintain SLAs during autoscaling, apply these cluster infrastructure practices:

1. High-Performance Local NVMe Scratch Caching

Do not stream weights over the network directly into PyTorch at startup. Instead, use a daemonset or init-container to mirror high-demand weights to a fast local NVMe directory on the worker nodes (e.g., /mnt/nvme/models). Mount this local path into the vLLM pod.

2. Fast Weight Distribution with Spegel or Peer-to-Peer Layers

If using container images containing frozen weights, leverage image pre-warming via tools like Spegel (a stateless P2P image distribution system for Kubernetes). Spegel allows nodes within the same K8s cluster to download container image layers directly from peer nodes over local VPC network bandwidth (up to 100 Gbps), bypassing registry bottlenecks.

3. Pre-Warming Model Weight Engines

Execute an initContainer that warms up CUDA kernels and verifies GPU P2P links before declaring the main vLLM container ready to accept actual HTTP traffic.

7. Advanced Runtime Optimizations Checklist

To squeeze every drop of performance out of your vLLM deployment, ensure these runtime flags are enabled in your orchestration specs:

Enable Chunked Prefill (--enable-chunked-prefill=true)

By default, long prompt prefill phases block the processing of ongoing generation/decode steps, causing massive TPOT latency jitter. Chunked Prefill breaks large context prompts into smaller chunks, interleaving prefill execution steps smoothly alongside token generation cycles.

Speculative Decoding (--speculative-model)

Pair a large target model (e.g., Llama-3-70B) with a tiny, fast draft model (e.g., Llama-3-8B). The draft model speculatively generates multiple candidate tokens rapidly, and the larger target model verifies them in a single parallel forward-pass. This can yield a 1.5x to 2.8x speedup in generation latency.

# Example vLLM launch parameters with speculative decoding:
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --speculative-model meta-llama/Meta-Llama-3-8B-Instruct \
  --num-speculative-tokens 5 \
  --tensor-parallel-size 4

FP8 / AWQ Quantization

Switching from FP16 to FP8 precision on modern GPU architectures (NVIDIA H100, L40S, Ada Lovelace) reduces memory footprint by 50% with negligible loss in accuracy. This enables larger max sequence lengths, higher concurrent batch limits, and higher memory-bandwidth bound execution speeds.

Summary Architectural Architecture Blueprint

To operationalize low-latency, cloud-scale LLM inference with vLLM on Kubernetes:

1. Memory First: Maximize VRAM efficiency via vLLM PagedAttention and tune --gpu-memory-utilization (typically 0.90 to 0.95).
2. Correct Node Specs: Allocate ample /dev/shm backed by RAM for multi-GPU IPC communication.
3. Smart Metrics Scaling: Ignore CPU/RAM HPA. Scale horizontally via KEDA driven by vllm:num_requests_waiting and vllm:gpu_cache_usage_perc.
4. Queue & Cache Aware Routing: Place a smart proxy layer in front of vLLM pods to route traffic based on real-time backlogs and cache state.
5. Eradicate Cold Starts: Pre-cache model weights on local node NVMe drives or leverage cluster P2P layer sharing.

By moving away from static GPU sizing and embracing dynamic memory-centric Kubernetes orchestration, you can scale enterprise AI services seamlessly—achieving ultra-low latency while slashing cloud infrastructure costs.