Deploying Large Language Models (LLMs) in enterprise production environments exposes a fundamental friction point: standard Kubernetes workload paradigms were designed for stateless microservices or coarse batch jobs, not the microsecond-level memory allocation and tensor-parallel operations required by state-of-the-art foundation models.
When running models like Llama 3 70B or Mixtral 8x22B at scale, traditional Kubernetes deployments relying on basic FastAPI wrappers and static GPU allocation hit a performance wall. High Time-To-First-Token (TTFT), low throughput under multi-tenant concurrent loads, and catastrophic GPU Out-Of-Memory (OOM) crashes during traffic spikes become recurring failure modes.
To overcome these constraints, modern AI platform engineering has converged on a powerful solution: combining vLLM's memory-efficient inference engine with Ray's distributed orchestration platform, running natively on Kubernetes via KubeRay.
This guide dives into the architecture, configuration, and implementation patterns required to achieve high-throughput, low-latency LLM serving on Kubernetes.
1. The Bottlenecks of Standard Kubernetes LLM Deployments
To understand why a vLLM + Ray architecture is necessary, we must examine where traditional Kubernetes serving patterns fail under high-throughput LLM workloads.
Standard K8s Deployments:
[ Request ] ──> [ Pod (FastAPI + Transformers) ] ──> Pre-allocated GPU Memory
├── Standard KV Cache (60-80% dynamic waste)
└── Static Batching (Gpu underutilized)
The KV Cache Memory Wall
During transformer autoregressive generation, the Key-Value (KV) cache stores past token states to prevent redundant re-computation. In standard implementations (like Hugging Face Transformers), memory for this cache must be allocated contiguously. Because sequence lengths are unpredictable, engines pre-allocate memory based on the worst-case scenario (max_model_len, e.g., 8192 or 32768 tokens).
This causes severe internal and external memory fragmentation. Up to 60% to 80% of costly VRAM lies unutilized, reserved for sequence capacity that is never used. This drastically limits the maximum concurrent batch size a single GPU can support.
Static vs. Continuous Batching
Traditional web servers aggregate requests into fixed-size batches (static batching) or wait for a timeout before pushing a batch to the GPU (dynamic batching). However, LLM token generation is non-deterministic—one request might produce 10 tokens while another produces 1,000. Under static batching, the GPU remains locked in memory and execution time until the longest request in the batch finishes, leaving compute units idle during short-sequence completions.
Naive Metrics for Horizontal Autoscaling
Standard Kubernetes Horizontal Pod Autoscalers (HPA) rely on CPU, RAM, or naive GPU compute utilization metrics (via NVIDIA DCGM). GPU utilization metrics are notoriously misleading for LLMs: a GPU running inference on a single sequence can report 99% compute utilization, yet possess massive headroom for additional batched sequences. Autoscaling based on raw GPU utilization leads to premature node provisioning, high infrastructure costs, or failure to scale when the request queue actually backs up.
2. Core Architecture: vLLM, PagedAttention, and Ray Serve
Resolving these bottlenecks requires fundamentally changing how memory is managed inside the GPU and how compute resources are orchestrated across pods.
vLLM + Ray Serve on K8s Architecture:
┌── Worker Pod 0 (Ray Actor) ──> vLLM Engine (GPU 0/1)
[ Ingress Controller ] ──> [ Ray Serve Head Pod ] ──┤
└── Worker Pod 1 (Ray Actor) ──> vLLM Engine (GPU 2/3)
▲
│ PagedAttention Engine
└── Non-contiguous Logical KV Blocks
vLLM & PagedAttention
vLLM solves the KV cache bottleneck through PagedAttention. Drawing from OS virtual memory concepts (paging), PagedAttention breaks the KV cache into small, fixed-size physical blocks (e.g., 16 tokens per block).
- Logical to Physical Mapping: Key and Value states are written into physical blocks non-contiguously. A block table translates virtual token sequences to actual physical GPU memory addresses.
- Dynamic Block Allocation: Blocks are allocated on-demand as tokens are generated. When a sequence completes, its blocks are immediately returned to the free memory pool.
- Continuous Batching (Iteration-level Scheduling): vLLM schedules work at the token iteration level rather than the request level. As soon as a request finishes generating, a new request from the queue is slotted into the active execution batch without waiting for the rest of the batch to complete.
Ray Serve & Distributed Orchestration
While vLLM efficiently manages memory on single nodes or discrete worker groups, Ray provides the distributed compute framework required to scale horizontally across multi-node, multi-GPU Kubernetes clusters.
- Model Parallelism (Tensor & Pipeline Parallelism): Models larger than a single GPU's VRAM (e.g., Llama 3 70B requiring ~140GB unquantized) must be split across multiple GPUs. Ray orchestrates model workers into unified Actor Groups spanning physical hosts using NCCL for inter-GPU communication.
- Dynamic Routing: Ray Serve routes requests directly to active worker actors based on target queue lengths rather than blunt round-robin algorithms, bypassing standard K8s service hop latency.
3. Infrastructure Blueprint: KubeRay on Kubernetes
To orchestrate this architecture, we use the KubeRay Operator, which manages the lifecycle of Ray Clusters natively within Kubernetes using Custom Resource Definitions (CRDs).
Architecture Layout
- RayHead Node: Runs the Ray Serve API Gateway, cluster autoscaler, and routing logic. Non-GPU workload.
- RayWorker NodeGroups: Hetrogeneous pools of GPU nodes (e.g., NVIDIA A100-80GB or H100s) isolated via node selectors and tolerations.
KubeRay RayCluster Manifest Configuration
Below is a production-grade RayCluster CRD spec optimized for serving a Tensor Parallelism (TP=2) deployment using NVIDIA A100 GPUs.
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: vllm-llama3-cluster
namespace: llm-serving
spec:
rayVersion: '2.35.0'
headGroupSpec:
rayStartParams:
dashboard-host: '0.0.0.0'
num-cpus: '0' # Head node routes traffic; offloads compute
template:
metadata:
labels:
ray.io/node-type: head
spec:
nodeSelector:
node.kubernetes.io/instance-type: c6i.2xlarge
containers:
- name: ray-head
image: rayproject/ray:2.35.0-py310
resources:
limits:
cpu: "7"
memory: "28Gi"
requests:
cpu: "4"
memory: "16Gi"
ports:
- containerPort: 6379
name: gcs
- containerPort: 8265
name: dashboard
- containerPort: 8000
name: serve
workerGroupSpecs:
- groupName: gpu-group-a100
replicas: 2
minReplicas: 1
maxReplicas: 8
rayStartParams:
num-gpus: '2'
template:
metadata:
labels:
ray.io/node-type: worker
spec:
nodeSelector:
accelerator/type: nvidia-a100-80gb
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: ray-worker
image: vllm/vllm-openai:v0.6.0
securityContext:
capabilities:
add: ["SYS_PTRACE"] # Required for efficient NCCL shared memory transfers
resources:
limits:
cpu: "16"
memory: "120Gi"
nvidia.com/gpu: "2"
requests:
cpu: "12"
memory: "90Gi"
nvidia.com/gpu: "2"
volumeMounts:
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cache/huggingface
name: model-cache
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 10Gi # Crucial for high-throughput PyTorch/NCCL IPC
- name: model-cache
persistentVolumeClaim:
claimName: efs-hf-cache-pvc
4. Production Deployment: Implementing the Ray Serve vLLM App
Rather than running vLLM via a simple CLI entrypoint inside containers, deploying vLLM via Ray Serve Python applications grants fine-grained control over dynamic request batching, model parallel actor placements, streaming response processing, and OpenAI API compatibility.
Production Ray Serve Python Implementation
This Python script defines an async vLLM worker deployment using Ray Serve:
import os
import typing
from typing import AsyncGenerator
from fastapi import FastAPI
import ray
from ray import serve
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
from vllm.utils import random_uuid
app = FastAPI()
@serve.deployment(
num_replicas=2,
ray_actor_options={"num_cpus": 4, "num_gpus": 2},
max_ongoing_requests=64
)
@serve.ingress(app)
class VLLMDeployment:
def __init__(self):
# Configure model parallelism and memory utilization parameters
engine_args = AsyncEngineArgs(
model="meta-llama/Meta-Llama-3-70B-Instruct",
tensor_parallel_size=2, # Distributed across the 2 assigned GPUs
pipeline_parallel_size=1,
gpu_memory_utilization=0.90, # Reserve 90% VRAM for KV cache/weights
max_model_len=8192,
enable_chunked_prefill=True, # Prevents prefill phase from starving decode steps
max_num_batched_tokens=2048,
trust_remote_code=True,
enforce_eager=False # Enable CUDA graphs for fast execution
)
self.engine = AsyncLLMEngine.from_engine_args(engine_args)
@app.post("/v1/completions")
async def generate(self, prompt: str, max_tokens: int = 512, temperature: float = 0.7):
"""Standard completions endpoint with token streaming capability."""
sampling_params = SamplingParams(
temperature=temperature,
max_tokens=max_tokens,
)
request_id = f"req-{random_uuid()}"
results_generator = self.engine.generate(prompt, sampling_params, request_id)
async def stream_results() -> AsyncGenerator[bytes, None]:
async for request_output in results_generator:
text_outputs = [output.text for output in request_output.outputs]
yield text_outputs[0].encode("utf-8")
from fastapi.responses import StreamingResponse
return StreamingResponse(stream_results(), media_type="text/plain")
@app.get("/health")
async def health_check(self):
"""Health check probe for Kubernetes readiness/liveness testing."""
if self.engine:
return {"status": "healthy"}
return {"status": "unhealthy"}, 503
# Deploy application to Ray cluster
entrypoint = VLLMDeployment.bind()
5. Advanced Autoscaling Strategy
Using standard CPU or GPU usage algorithms to scale LLM workloads causes oscillations and slow cold starts. A resilient deployment requires a two-tiered scaling framework:
Autoscaling Pipeline:
[ Client Traffic ] ──> [ Serve Queue Metrics ]
│
├── Tier 1: Ray Serve Replica Scaler (Scale Ray Actors up/down)
│
└── Tier 2: KubeRay + Cluster Autoscaler (Provision Cloud GPU Nodes)
1. Application Layer: Ray Serve Dynamic Autoscaler
Ray Serve continuously monitors the queue depth (max_ongoing_requests) for every deployment replica. It dynamically adds or removes Ray Actors based on actual execution saturation rather than raw hardware metrics.
# Ray Serve Deployment Configuration (serve_config.yaml)
applications:
- name: llama3_service
route_prefix: /
import_path: serve_app:entrypoint
runtime_env:
pip:
- vllm==0.6.0
autoscaling_config:
min_replicas: 1
max_replicas: 8
target_ongoing_requests: 16 # Target backpressure per replica before triggering scale-out
upscale_delay_s: 15
downscale_delay_s: 300 # Keep GPUs warm during dynamic load drops
2. Infrastructure Layer: KubeRay + Kubernetes Cluster Autoscaler
When the Ray Serve deployment requests more replicas than the current RayCluster capacity permits:
- Ray Serve requests new actors from the Ray Head node.
- Ray Head creates pending Ray Worker Pods that cannot be scheduled.
- KubeRay reflects these pending pods to standard Kubernetes scheduling APIs.
- The cloud provider's Kubernetes Cluster Autoscaler (or Karpenter) provisions physical host GPU nodes (e.g., AWS
g5.12xlargeorp4d.24xlargeinstances) to satisfy pod constraints.
6. Production Hardening & Optimization Tuning
Deploying LLMs at enterprise scale requires tuning low-level kernel and memory settings. Here is a production checklist:
Optimization Settings
| Parameter / Feature | Value | Architectural Impact |
| :--- | :--- | :--- |
| enable_chunked_prefill | True | Breaks long input prefill requests into chunked batches. Prevents large prompt ingestions from spiking Time-To-First-Token (TTFT) for existing active generation requests. |
| Automatic Prefix Caching | Enabled via vllm flags | Reuses existing KV Cache blocks for recurring system prompts or long document context across different requests. Drops prefill latency by up to 80%. |
| /dev/shm Size | >= 10 GiB | High-throughput inter-process communication (PyTorch distributed / NCCL ring reduce) requires large shared memory volumes. Default K8s emptyDir (64MB) will result in bus errors/OOMs. |
| SYS_PTRACE Capability | add: ["SYS_PTRACE"] | Allows NCCL to bypass user-space memory copies using IPC, maximizing inter-GPU bandwidth. |
| enforce_eager | False | Forces execution via compiled CUDA Graphs instead of PyTorch eager mode. Reduces GPU launch overhead on smaller generation steps. |
Observability Infrastructure
Standard metrics are inadequate for LLMs. Extract custom metrics directly from vLLM and Ray Serve via Prometheus and visualize them on Grafana using these key indicators:
# Core Prometheus Metrics to Monitor
vllm:num_requests_waiting # Ingress Queue Length (SLA Alerting trigger)
vllm:gpu_cache_usage_perc # Physical PagedAttention Memory Utilization
vllm:avg_prompt_throughput_toks # Prefill processing speed
vllm:avg_generation_throughput_toks # Decode token generation speed (TPOT)
vllm:time_to_first_token_seconds # Initial response latency (TTFT)
Configure your Kubernetes deployment with standard Prometheus annotations so scraping agents target the vLLM engine directly:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
Summary: Performance Gains Realized
By moving away from unmanaged, static GPU pod deployments to an integrated vLLM + Ray + KubeRay architecture, enterprise platforms consistently see remarkable structural improvements:
- 5x Throughput Increase: PagedAttention paired with Continuous Batching yields up to 500% higher token generation throughput per dollar spent on GPU infrastructure.
- Deterministic Memory Scaling: Dynamic KV cache page allocation eliminates premature GPU Out-Of-Memory crashes, permitting VRAM saturation rates upwards of 90%.
- Queue-Driven Autoscaling: Scaling dynamically based on actual request saturation rather than raw GPU metric signals reduces infrastructure burn rates during non-peak traffic windows.
This combination forms the state-of-the-art foundation for serving modern multi-billion parameter foundation models efficiently, stably, and cost-effectively on Kubernetes.