Optimizing LLM Inference Latency on Kubernetes: A Guide to vLLM and Custom GPU Autoscaling | Ecstaticloud Blog

Deploying Large Language Models (LLMs) into production presents a fundamentally different set of infrastructure challenges than serving traditional microservices or classical machine learning models. When running models like Llama 3 or Mistral at scale, standard Kubernetes autoscaling patterns break down.

If you rely on CPU or GPU utilization to trigger standard Horizontal Pod Autoscalers (HPA), you will encounter two critical failures:

Severe Latency Spikes: Your GPU compute utilization will hit 90%+ instantly during the prefill phase, but incoming requests will queue up indefinitely while waiting for KV-Cache allocations, causing P99 latency to explode.
Resource Waste: Your GPU memory will remain 95% full simply holding KV-Cache pages even when zero compute is actively happening, preventing HPA from scaling down during traffic lulls.

To build an enterprise-grade LLM inference platform that balances sub-second latency with infrastructure unit economics, you need two things: vLLM as the execution engine and KEDA (Kubernetes Event-driven Autoscaling) driven by engine-level metrics.

The Physics of LLM Inference: Prefill vs. Decode

To understand why traditional scaling fails, we must first analyze the two distinct computational phases of autoregressive transformer inference.

+-----------------------------------------------------------------------------+
|                             INCOMING REQUEST                                |
+-----------------------------------------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 1. PREFILL PHASE (Compute-Bound)                                            |
| - Processes prompt tokens in parallel via GEMM (General Matrix Multiply).   |
| - Saturation: GPU Compute Engines (Tensor Cores).                           |
| - Primary Metric Impact: Time to First Token (TTFT).                        |
+-----------------------------------------------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 2. DECODE PHASE (Memory Bandwidth-Bound)                                    |
| - Generates tokens sequentially via GEMV (General Matrix-Vector Multiply).  |
| - Saturation: HBM Memory Bandwidth & Memory Capacity (KV Cache).            |
| - Primary Metric Impact: Time Per Output Token (TPOT).                      |
+-----------------------------------------------------------------------------+

1. The Prefill Phase (Compute-Bound)

What happens: The model takes the input prompt tokens and processes them in parallel to build the initial Key-Value (KV) cache.
Hardware bottleneck: Compute capability (FLOPs / Tensor Cores).
Metric impact: Time To First Token (TTFT).

2. The Decode Phase (Memory Bandwidth-Bound)

What happens: The model generates output tokens sequentially, one by one. Each new token requires reading the model weights plus the entire historical KV-cache out of High Bandwidth Memory (HBM) into SRAM.
Hardware bottleneck: HBM Bandwidth (GB/sec) and Memory Capacity.
Metric impact: Time Per Output Token (TPOT) or Inter-Token Latency (ITL).

Because the Decode phase requires fetching gigabytes of model weights from HBM to compute a single token for a single user, single-request execution on modern GPUs (like NVIDIA H100 or A100) severely underutilizes GPU Tensor Cores.

To maximize throughput, we must batch multiple requests together. However, traditional batching strategies force requests to wait for the slowest generation to complete, wasting massive amounts of compute.

Mechanics of vLLM: Why Architecture Matters

vLLM solves the throughput and latency bottleneck through three primary architectural innovations:

1. PagedAttention

Traditional framework implementations allocate contiguous physical GPU memory for a request's KV-cache based on the model's max_sequence_length. If a model supports 8,192 tokens, memory is reserved for 8,192 tokens even if the user request only generates 50 tokens. This results in 60% to 80% memory fragmentation and waste.

PagedAttention mimics Virtual Memory in operating systems:

It divides the KV-cache into fixed-size physical blocks (e.g., 16 tokens per block).
Blocks are allocated dynamically on-demand as tokens are generated.
Physical blocks do not need to be contiguous in GPU HBM.

Logical KV Cache (Request A): [ Block 0 ] [ Block 1 ] [ Block 2 ]
                                   |           |           |
                                   v           v           v
Physical GPU HBM Memory:       [ Block 1 ] [ Block 0 ] [ Block 2 ] (Non-contiguous)

By eliminating memory fragmentation, vLLM frees up massive amounts of HBM, allowing you to increase batch sizes by 2x–4x without increasing hardware costs.

2. Continuous Batching (Iteration-Level Batching)

Instead of waiting for an entire batch of requests to reach their <EOS> (End of Sequence) token before accepting new work, vLLM operates at the iteration level:

As soon as a request finishes generating, its allocated memory blocks are freed.
A waiting request from the queue is immediately inserted into the running batch on the very next token iteration.

3. Parallelism Strategies: Tensor Parallelism vs. Pipeline Parallelism

When model weights exceed the memory of a single GPU, you must distribute the model across multiple devices:

Tensor Parallelism (TP): Splits individual matrix multiplication operations within a transformer layer across multiple GPUs (e.g., splitting a 4096-dim projection across 4 GPUs).
- Constraint: Requires ultra-high intra-node inter-GPU bandwidth (NVIDIA NVLink @ 900 GB/s on H100). Do not run Tensor Parallelism across network nodes.
Pipeline Parallelism (PP): Splits model layers sequentially across different GPUs or nodes (e.g., Layers 1–16 on Node A, Layers 17–32 on Node B).
- Constraint: Introduces pipeline bubbles (idle time while waiting for previous stage results). Used primarily for massive multi-node deployments.

Kubernetes Cluster Architecture for GPU Inference

To serve production LLMs on Kubernetes, your infrastructure layer must be fine-tuned for high IPC (Inter-Process Communication) bandwidth and low-latency storage access.

+--------------------------------------------------------------------+
|                         KUBERNETES NODE                            |
|                                                                    |
|  +--------------------------------------------------------------+  |
|  |                        vLLM POD                              |  |
|  |                                                              |  |
|  |  +---------------------+        +-------------------------+  |  |
|  |  | Container Engine    |        | Inter-Process Comm      |  |  |
|  |  | (PyTorch + vLLM)    |        | /dev/shm (Shared Mem)   |  |  |
|  |  +----------+----------+        +------------+------------+  |  |
|  +-------------|--------------------------------|---------------+  |
|                |                                |                  |
|                v                                v                  |
|  +--------------------------------------------------------------+  |
|  |                     NVIDIA GPU OPERATOR                      |  |
|  |  +--------------------------------------------------------+  |  |
|  |  | NVIDIA Driver | CUDA Runtime | Container Toolkit       |  |  |
|  |  +--------------------------------------------------------+  |  |
|  +------------------------------+-------------------------------+  |
|                                 |                                  |
|                                 v                                  |
|  +--------------------------------------------------------------+  |
|  |                      PHYSICAL HARDWARE                       |  |
|  |  +-----------------+  NVLink  +---------------------------+  |  |
|  |  | GPU 0 (A100/H100) |========| GPU 1 (A100/H100)         |  |  |
|  |  +-----------------+          +---------------------------+  |  |
|  |  +--------------------------------------------------------+  |  |
|  |  | Local NVMe Array (/var/autoscale/model-cache)          |  |  |
|  |  +--------------------------------------------------------+  |  |
|  +--------------------------------------------------------------+  |
+--------------------------------------------------------------------+

Shared Memory Requirements (`/dev/shm`)

PyTorch uses shared memory for inter-process communications when running Tensor Parallelism across multiple processes on the same pod. The default Linux shared memory allocation inside a Kubernetes container is 64MB, which will instantly cause a SIGBUS or CUDA Out-Of-Memory error during initialization. You must mount an emptyDir volume with medium: Memory to /dev/shm.

GPU Device Allocation and Drivers

Deploy the NVIDIA GPU Operator to manage CUDA driver initialization, container runtime hooks, and device plugin exposure. Ensure your node pools isolate GPU workloads using taints and tolerations to avoid running control plane or CPU-heavy workloads on high-cost GPU nodes.

Deep Dive: Custom GPU Autoscaling with KEDA

Standard Kubernetes HPA calculates target metric averages across pods using standard CPU/Memory signals or basic GPU utilization exported by nvidia-dcgm-exporter.

Why standard GPU metrics fail for scaling:

GPU Compute Utilization (DCGM_FI_DEV_GPU_UTIL): Can hit 100% processing a single long-context prompt. Scaling out here would over-provision infrastructure pointlessly.
GPU Memory Utilization (DCGM_FI_DEV_FB_USED): vLLM pre-allocates almost all available GPU memory (typically --gpu-memory-utilization 0.90 or 90%) at container startup to form its physical PagedAttention block pool. HPA will see 90% memory usage continuously and keep the replica count pinned at max capacity, even if 0 requests are actively running.

The Solution: Exposing Internal Engine Metrics

vLLM exports native Prometheus metrics at :8000/metrics. The two critical metrics that dictate cluster health and autoscaling demand are:

vllm:num_requests_waiting: The number of requests sitting in vLLM's internal execution queue. If this number is greater than zero, your existing GPU instances are fully saturated and incoming requests are experiencing queuing latency.
vllm:gpu_cache_usage_perc: The percentage of physical PagedAttention memory blocks currently occupied by active request KV-caches.

KEDA Prometheus Trigger Architecture

We use KEDA to evaluate custom PromQL queries based on these native metrics and dynamically scale our vLLM Deployment.

Production-Grade Kubernetes Deployment & Scaling Manifests

Below is a complete, production-ready setup deploying Llama-3-8B-Instruct using Tensor Parallelism across 2 GPUs with custom KEDA autoscaling.

1. vLLM Kubernetes Deployment Manifest (`deployment.yaml`)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
  namespace: llm-serving
  labels:
    app.kubernetes.io/name: vllm-llama3-8b
    app.kubernetes.io/component: inference-engine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama3-8b
  template:
    metadata:
      labels:
        app: vllm-llama3-8b
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      nodeSelector:
        accelerator: nvidia-gpu
        gpu-type: a100-80gb
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: vllm-engine
        image: vllm/vllm-openai:v0.4.2
        imagePullPolicy: IfNotPresent
        command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
        args:
        - "--model=meta-llama/Meta-Llama-3-8B-Instruct"
        - "--tensor-parallel-size=2"
        - "--max-model-len=8192"
        - "--gpu-memory-utilization=0.90"
        - "--port=8000"
        - "--download-dir=/model-cache"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        - name: NCCL_DEBUG
          value: "INFO"
        ports:
        - name: http
          containerPort: 8000
          protocol: TCP
        resources:
          limits:
            cpu: "16"
            memory: "64Gi"
            nvidia.com/gpu: "2"
          requests:
            cpu: "8"
            memory: "32Gi"
            nvidia.com/gpu: "2"
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 15
          timeoutSeconds: 5
          failureThreshold: 5
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
        - name: model-cache-volume
          mountPath: /model-cache
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 16Gi
      - name: model-cache-volume
        hostPath:
          path: /var/autoscale/model-cache
          type: DirectoryOrCreate

2. KEDA Custom ScaledObject Manifest (`keda-scaledobject.yaml`)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama3-keda-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 25
            periodSeconds: 60
  triggers:
  # Trigger 1: Scale instantly if requests are queueing up in vLLM
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{namespace="llm-serving", app="vllm-llama3-8b"})
      threshold: '3'
  # Trigger 2: Scale out proactively if KV-Cache memory usage exceeds 80%
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_gpu_cache_usage_perc
      query: max(vllm:gpu_cache_usage_perc{namespace="llm-serving", app="vllm-llama3-8b"})
      threshold: '0.80'

3. Prometheus Alerting Rules for Slos (`prometheus-rules.yaml`)

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: vllm-performance-alerts
  namespace: llm-serving
spec:
  groups:
  - name: vllm.rules
    rules:
    - alert: HighRequestQueueTime
      expr: sum(vllm:num_requests_waiting{namespace="llm-serving"}) > 10
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "vLLM instance queue depth is critically high"
        description: "vLLM queue has exceeded 10 requests for > 2 minutes. Infrastructure capacity is insufficient for current traffic."
    - alert: KVCacheSaturation
      expr: max(vllm:gpu_cache_usage_perc{namespace="llm-serving"}) > 0.95
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "vLLM GPU KV-Cache near total exhaustion"
        description: "KV Cache usage is above 95%. Incoming requests will experience severe prefill queuing delays."

Cold Starts & Storage Engineering: Solving Model Pull Bottlenecks

Scale-out events are useless if a newly provisioned Pod spends 8 to 12 minutes downloading model weights from Hugging Face before passing readiness probes. To achieve true responsive scaling, you must optimize model layer ingestion speed.

+-------------------------------------------------------------------------------+
|                       COLD-START STORAGE ARCHITECTURES                        |
+-------------------------------------------------------------------------------+

[NAIVE] Cloud Object Storage Download (Standard S3)
 Pod Launch ---> Pull Image (1m) ---> S3 Model Download (8m) ---> Load to GPU (30s)
 Total Time to Ready: ~9m30s
 
[OPTIMIZED] Distributed Storage Engine (JuiceFS / Chunk Caching)
 Pod Launch ---> Pull Image (1m) ---> Parallel Distributed Fetch (45s) ---> Load to GPU (30s)
 Total Time to Ready: ~2m15s

[ULTRA-FAST] HostPath / Local NVMe Pre-Warming
 Pod Launch ---> Pull Image (1m) ---> Local NVMe PCI-e Bus Read (8s) ---> Load to GPU (30s)
 Total Time to Ready: ~1m38s

1. Local HostPath NVMe Caching

Mount high-performance local NVMe SSDs directly into the Kubernetes nodes. Use a DaemonSet or Node-Initialization script to pre-hydrate model weights onto /var/autoscale/model-cache. As shown in our deployment manifest above, mounting hostPath directly reduces disk read times to PCIe bus speed limits (~7–12 GB/s), bypassing the network entirely.

2. High-Performance Distributed File Systems (JuiceFS / Spegel)

If local HostPath storage is impractical due to node ephemeral lifecycles:

Deploy a distributed cache layer like JuiceFS backed by S3, mounting the cache as a ReadWriteMany Persistent Volume (PV).
Use peer-to-peer container image distribution engines like Spegel or Dragonfly to parallelize layer pulling across worker nodes.

3. Container Image Pre-Warming

For rigid production environments with static models, bake the model weights directly into a dedicated container image layer using git-lfs during CI/CD build stages, or use an init-container that executes a parallelized aria2c fetch directly from an internal high-throughput MinIO cluster over 100GbE network interfaces.

Real-World Benchmarks & Latency Profiling

Below is performance data captured under a simulated multi-tenant load test comparing a standard Hugging Face Text Generation Inference (TGI) default setup against our vLLM + KEDA Autoscaled Platform running on NVIDIA A100-SXM4-80GB GPUs serving Meta-Llama-3-8B-Instruct.

Concurrency Load: Ramp from 1 to 120 concurrent client connections over 10 minutes.
Input Context: Average 1,024 prompt tokens.
Generation Output: Average 256 output tokens.

Latency vs. Concurrency Breakdown

Latency (ms)
  ^
5000 |                                            / (Naive Pipeline TTFT Spikes)
4000 |                                           /
3000 |                                          /
2000 |                                         /
1000 |                                        /
 200 |_______________________________________/___ (vLLM + KEDA Dynamic Scaling)
   0 +---------------------------------------------------> Concurrency (RPS)
     1          20          40          80        120

Under heavy concurrency spikes, the naive pipeline runs out of contiguous memory allocations for its KV-cache, causing incoming requests to block waiting for active requests to run to completion.

Conversely, vLLM's continuous batching handles the load up to engine capacity, at which point the metric vllm:num_requests_waiting instantly breaches our KEDA threshold of 3, triggering an immediate cluster expansion before the customer experiences catastrophic SLA degradation.

Architectural Rules for Production LLM Deployment

Never scale LLMs on CPU or GPU Utilization percentage. Always drive horizontal pod autoscaling through application-layer signals: queue depth (vllm:num_requests_waiting) and KV-Cache saturation (vllm:gpu_cache_usage_perc).
Mount shared memory properly. Always configure /dev/shm as an emptyDir with medium: Memory when running multi-GPU Tensor Parallelism to avoid NCCL crashes.
Isolate intra-node inter-GPU communications. Ensure Tensor Parallelism (--tensor-parallel-size) is bounded to single nodes connected via NVLink. Do not stretch Tensor Parallelism over local Ethernet/IP networks.
Pre-hydrate storage. Mitigate cold starts by avoiding raw runtime downloads from external model hubs. Use local hostPath mounts, local NVMe caches, or high-speed distributed object layers.
Tune PagedAttention block size. Match vLLM block size configuration to your expected prompt payload lengths. Smaller block sizes (e.g., 8 or 16) reduce memory fragmentation, while larger block sizes optimize compute engine cache lines.