Scaling Distributed LLM Inference on Kubernetes: A Blueprint Using vLLM and Ray

In modern cloud infrastructure, scaling Large Language Models (LLMs) like Llama 3 70B, Mixtral 8x22B, or DeepSeek-V3 presents a radical departure from traditional microservices architecture. Standard web services scale horizontally along stateless boundaries. In contrast, LLM inference workloads are constrained by compute bounds during the prefill phase, memory bandwidth bounds during the decode phase, and strict VRAM footprint requirements dictated by model weights and Key-Value (KV) cache allocation.

Deploying high-throughput, low-latency LLM workloads at enterprise scale requires an orchestration layer capable of coordinating multi-GPU, multi-node Tensor Parallelism (TP) and Pipeline Parallelism (PP), while dynamically managing requests across a pool of ephemeral compute nodes.

This architectural blueprint outlines how to combine Kubernetes, KubeRay, and vLLM into a resilient, enterprise-grade LLM inference platform capable of serving multi-billion parameter models with sub-second time-to-first-token (TTFT) and optimized cost per token.

1. The Architectural Bottleneck: Why Standard Kubernetes Deployments Fail

To understand why custom orchestration is necessary, we must examine the memory dynamics of autoregressive LLM inference:

1. Static Memory (Model Weights): A FP16 model requires ~2GB of VRAM per billion parameters. A 70B parameter model requires ~140GB of VRAM just to load its weights into memory, instantly exceeding the standard 80GB capacity of an NVIDIA A100/H100 GPU.
2. Dynamic Memory (KV Cache): During inference, tokens generated in previous steps are cached to avoid recomputing attention matrices. The memory consumed by the KV cache scales linearly with batch size, context window length, and layer count. Under heavy load, unbounded KV cache allocation leads to Out-Of-Memory (OOM) faults or aggressive request dropping.
3. Inter-GPU Communication: Distributing a model across multiple GPUs requires continuous synchronization via High-Bandwidth Memory (HBM) interconnects (NVLink/NVSwitch) or cross-node fabrics (InfiniBand/RoCEv2). Standard Kubernetes scheduler primitives are unaware of topology constructs like NVLink domains or NUMA nodes.

+-----------------------------------------------------------------------+
|                              GPU VRAM                                 |
+------------------------------------+----------------------------------+
| Model Weights (FP16/INT8/FP8)      | KV Cache (Dynamic Allocation)    |
| e.g., 140GB for Llama-3-70B (FP16) | Scales with Batch Size & Context |
+------------------------------------+----------------------------------+

When standard Kubernetes deployments attempt to run single-container instances of large models using simple HTTP wrappers, they run into three primary failure modes:

• Fragmented VRAM Utilization: Static batching pre-allocates maximum sequence length memory, leaving up to 60-80% of VRAM idle.
• Inter-Node Latency Degradation: Naive scheduling places distributed inference pods across high-latency network boundaries without topology alignment.
• Cold Start Penalties: Fetching 100GB+ model artifacts from remote registries on pod startup causes multi-minute scale-up delays, breaking auto-scaling SLAs.

2. The Triad Stack: Kubernetes + Ray + vLLM

To resolve these bottlenecks, we combine three infrastructure layers into a cohesive engine:

+-----------------------------------------------------------------------+
|                           Kubernetes API                              |
|   (Cluster Lifecycle, Ingress/Gateway API, Cloud Provider Resources)   |
+-----------------------------------------------------------------------+
                                    |
                                    v
+-----------------------------------------------------------------------+
|                     KubeRay Operator (RayCluster)                     |
|   - Ray Head Pod (Orchestration, Routing, API Gateway)                |
|   - Ray Worker Pods (GPU Execution Nodes, Actor Placement Groups)     |
+-----------------------------------------------------------------------+
                                    |
                                    v
+-----------------------------------------------------------------------+
|                    Ray Serve Distributed Runtime                      |
|   - Handles Inter-Actor RPC, Dynamic Scaling, Request Queueing        |
+-----------------------------------------------------------------------+
                                    |
                                    v
+-----------------------------------------------------------------------+
|                         vLLM Engine Instances                         |
|   - PagedAttention (Virtual Memory Management for KV Cache)          |
|   - Continuous Batching (Iteration-level Scheduling)                  |
|   - Chunked Prefill & CUDA Graph Execution Engine                     |
+-----------------------------------------------------------------------+

vLLM: The High-Efficiency Inference Engine

vLLM fundamentally changes memory management via PagedAttention. Inspired by virtual memory paging in operating systems, PagedAttention breaks the KV cache down into physical memory blocks. Tokens are assigned non-contiguous physical blocks on demand, completely eliminating internal VRAM fragmentation and reducing memory waste to near zero. Furthermore, vLLM supports continuous batching (iteration-level scheduling) rather than request-level batching, maximizing GPU compute density during the generation phase.

Ray & Ray Serve: Distributed Compute Engine

While vLLM provides optimized single-node/multi-GPU performance, Ray supplies the orchestration layer for multi-actor distribution, intra-node/inter-node remote procedure calls (RPC), and worker pool execution. Ray Serve provides programmable HTTP routing, model placement primitives (PlacementGroups), and metric-driven replica auto-scaling tailored specifically for distributed actors.

Kubernetes & KubeRay Operator: Declarative Infrastructure

Kubernetes manages the physical underlying compute nodes, persistent storage, hardware passthrough (NVIDIA GPU Operator), and CNI networking. The KubeRay Operator translates declarative Kubernetes Custom Resource Definitions (CRDs) like RayCluster and RayService into physical Head and Worker Pods, automating node failure recovery and pod-level scaling lifecycle management.

3. High-Level Target Architecture

The blueprint architecture relies on a decoupled ingress and execution layer managed via Kubernetes and KubeRay.

graph TD
    Client[Client Request] --> Ingress[Kubernetes Gateway API / NGINX Ingress]
    Ingress --> RayHead[Ray Head Pod: Ray Serve Router]
    
    subgraph "Kubernetes Compute Node Pool (GPU Nodes)"
        subgraph "Ray Worker Pod 0 (Node A)"
            Worker0_GPU0[GPU 0: vLLM Worker - TP Rank 0]
            Worker0_GPU1[GPU 1: vLLM Worker - TP Rank 1]
        end
        
        subgraph "Ray Worker Pod 1 (Node B)"
            Worker1_GPU0[GPU 0: vLLM Worker - TP Rank 2]
            Worker1_GPU1[GPU 1: vLLM Worker - TP Rank 3]
        end
    end

    RayHead -->|Inter-Actor Ray RPC| Worker0_GPU0
    RayHead -->|Inter-Actor Ray RPC| Worker1_GPU0
    
    Worker0_GPU0 <-->|NVLink / PyTorch NCCL| Worker0_GPU1
    Worker1_GPU0 <-->|NVLink / PyTorch NCCL| Worker1_GPU1
    Worker0_GPU0 <-->|RoCEv2 / InfiniBand NCCL| Worker1_GPU0

    SharedStore[(Shared High-Speed Storage: NVMe LPV / JuiceFS / S3 Cache)]
    SharedStore -->|Direct Memory Read| Worker0_GPU0
    SharedStore -->|Direct Memory Read| Worker1_GPU0

4. Production Deployment Implementation

Let me walk through the end-to-end setup for a production deployment of Llama-3-70B-Instruct across multiple GPU instances using Tensor Parallelism (TP=4).

Step 1: Infrastructure & Storage Prerequisites

Before applying Ray manifests, ensure your Kubernetes cluster has the following components enabled:

• NVIDIA GPU Operator: Configured with driver.enabled=true and toolkit.enabled=true.
• Local Persistent Volumes / High-Throughput CSI: Loading 140GB weights from standard object stores over slow networks on cold boots is unacceptable. Use host-path local NVMe mounted volumes, or high-throughput file systems like JuiceFS / Lustre backed by persistent caches.

Step 2: Deploying the KubeRay Operator & RayCluster Definition

Below is the production manifest for a high-availability RayService CRD. It provisions a Ray Head node and a scalable pool of GPU Worker nodes equipped with 4 x NVIDIA A100/H100 GPUs each.

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: vllm-llama3-70b-service
  namespace: llm-serving
spec:
  serviceUnhealthyThreshold: 300
  rayClusterConfig:
    rayVersion: '2.35.0'
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
        num-gpus: '0' # Head node acts as orchestrator/router only
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.35.0-py310
              resources:
                limits:
                  cpu: "8"
                  memory: "32Gi"
                requests:
                  cpu: "4"
                  memory: "16Gi"
              ports:
                - containerPort: 6379
                  name: gcs
                - containerPort: 8265
                  name: dashboard
                - containerPort: 8000
                  name: serve
    workerGroupSpecs:
      - groupName: gpu-group
        replicas: 2
        minReplicas: 1
        maxReplicas: 4
        rayStartParams:
          num-gpus: '4'
        template:
          spec:
            containers:
              - name: ray-worker
                image: vllm/vllm-openai:v0.6.0
                securityContext:
                  capabilities:
                    add: ["SYS_PTRACE"] # Required for NCCL shared memory IPC
                resources:
                  limits:
                    cpu: "32"
                    memory: "128Gi"
                    nvidia.com/gpu: "4"
                  requests:
                    cpu: "16"
                    memory: "64Gi"
                    nvidia.com/gpu: "4"
                volumeMounts:
                  - mountPath: /dev/shm
                    name: dshm
                  - mountPath: /root/.cache/huggingface
                    name: model-cache
            volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 16Gi # Prevents SHM OOM during inter-process communication
              - name: model-cache
                persistentVolumeClaim:
                  claimName: nvme-model-cache-pvc

Step 3: Implementing the Ray Serve vLLM Deployment Script

Next, define the Ray Serve application deployment script (deploy_vllm.py). This script configures vLLM's AsyncEngine inside a distributed Ray Serve deployment, utilizing Ray's Placement Groups to lock GPU resources for Tensor Parallel execution across ranks.

import os
import asyncio
from typing import AsyncGenerator
from ray import serve
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
from vllm.utils import random_uuid

@serve.deployment(
    name="llama-3-70b-deployment",
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 4,
        "target_ongoing_requests": 40, # Triggers scale up when batch concurrency saturates
    },
    ray_actor_options={
        "num_cpus": 8,
        "num_gpus": 0, # Serve controller delegates GPU work to vLLM placement group
    }
)
class VLLMDeployment:
    def __init__(self):
        # Configuration for Tensor Parallelism across 4 GPUs
        engine_args = AsyncEngineArgs(
            model="/root/.cache/huggingface/hub/models--meta-llama--Meta-Llama-3-70B-Instruct/snapshots/main",
            tensor_parallel_size=4,
            pipeline_parallel_size=1,
            trust_remote_code=True,
            gpu_memory_utilization=0.90, # Reserve 90% VRAM for weights + KV cache
            max_model_len=8192,
            enable_chunked_prefill=True, # Prevents large prefill tasks from starving decodes
            max_num_batched_tokens=4096,
            engine_use_ray=True
        )
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    async def stream_results(self, results) -> AsyncGenerator[str, None]:
        async for request_output in results:
            text_outputs = [output.text for output in request_output.outputs]
            yield text_outputs[-1]

    @serve.batch(max_batch_size=16, batch_wait_timeout_s=0.05)
    async def __call__(self, prompt: str, max_tokens: int = 512, temperature: float = 0.7) -> AsyncGenerator[str, None]:
        request_id = random_uuid()
        sampling_params = SamplingParams(
            max_tokens=max_tokens,
            temperature=temperature,
            top_p=0.95
        )
        
        results_generator = self.engine.generate(
            prompt,
            sampling_params,
            request_id
        )
        
        return self.stream_results(results_generator)

# Expose Ray Serve deployment wrapper entrypoint
entrypoint = VLLMDeployment.bind()

5. Performance Tuning & Advanced Configuration

To extract true line-rate performance from your GPU topology, standard settings are not enough. The following architectural optimizations must be applied to the platform:

                  +-----------------------------------+
                  |      Optimization Strategy        |
                  +-----------------------------------+
                                    |
        +---------------------------+---------------------------+
        |                                                       |
        v                                                       v
+-------------------------------+               +-------------------------------+
|     Inference Engine Level    |               |  System & Orchestration Level |
+-------------------------------+               +-------------------------------+
| - Chunked Prefill             |               | - Unified Host Memory Shared  |
| - Custom KV Block Size        |               |   IPC (`/dev/shm` > 16GB)     |
| - CUDA Graph Warmup Execution |               | - NCCL Over RoCEv2/InfiniBand |
| - Dynamic Quantization (FP8)  |               | - Fast Node Prefill Caching   |
+-------------------------------+               +-------------------------------+

A. Prefill vs. Decode Optimization (Chunked Prefill)

Compute requirements vary dramatically between the prefill stage (processing the prompt, compute-bound) and the decode stage (generating tokens one-by-one, memory-bandwidth-bound).

Large prefill bursts create extreme tail latency spikes for concurrent decode requests. Enabling enable_chunked_prefill=True breaks large prompts into smaller chunks, interleaving chunked prefill executions with active generation iterations. This keeps time-to-first-token (TTFT) low while stabilizing inter-token latency (ITL).

B. Shared Memory Overhead & IPC Hardware Access

Distributed PyTorch executions (NCCL) require fast intra-pod Shared Memory (/dev/shm) to perform inter-process communications (IPC).

• If /dev/shm is unconstrained or left at default Kubernetes settings (64MB), workers will instantly crash with SIGBUS or PyTorch NCCL initialization timeouts.
• Requirement: Always mount an explicit emptyDir backed by memory (medium: Memory) with a minimum size of 16Gi to /dev/shm.

C. NCCL Inter-Node Tuning for Multi-Node Tensor Parallelism

If your compute topology spans across distinct Kubernetes nodes (e.g., using 2 nodes with 2 GPUs each instead of 1 node with 4 GPUs), intra-node NVLink speeds are no longer available for all tensor operations. Inter-node communications must default to network adapters. Ensure your deployment injects the proper NCCL environment variables into the Ray Worker specs:

env:
  - name: NCCL_DEBUG
    value: "INFO"
  - name: NCCL_IB_DISABLE
    value: "0" # Do NOT disable InfiniBand/RoCE if available
  - name: NCCL_NET_GDR_LEVEL
    value: "2" # Enable GPUDirect RDMA across supported network adapters

6. Metrics, Observability, and Dynamic Auto-Scaling

In traditional autoscaling, CPU or Memory threshold triggers (HPA) dictate scale events. For LLMs, CPU usage is an entirely decoupled metric that bears no relation to pipeline saturation.

Key Metrics to Monitor via Prometheus

The Ray/vLLM engine exposes explicit metrics that serve as indicators for cluster state:

1. vllm:num_requests_waiting: Requests sitting in the prefill/decode engine queue. This is your primary trigger metric for dynamic pod autoscaling.
2. vllm:gpu_cache_usage_perc: VRAM KV Cache block consumption percentage. If this stays consistently above 85-90%, it indicates high dynamic load and potential memory exhaustion.
3. vllm:avg_time_to_first_token_seconds: The latency between request ingress and initial generation stream token.
4. vllm:time_per_output_token_seconds: The generation speed per token across active users.

Architectural Scaling Decision Matrix

| Observability Metric State | Underlying Cause | Corrective Infrastructure Action | | :--- | :--- | :--- | | num_requests_waiting > Threshold | Ingress traffic exceeds engine throughput capacity. | Trigger KubeRay autoscaler to spin up additional RayWorker groups. | | gpu_cache_usage_perc > 90%, but CPU low | High context-length requests saturating KV block memory. | Tune vLLM max_num_batched_tokens or scale horizontal model replicas. | | Tail-latency TTFT Spike | Unbatched huge prompt prefill executions blocking decode engine. | Reduce chunk prefill size or split prefill/decode onto dedicated node pools. |

7. Architectural Blueprint Trade-offs

When building this infrastructure platform, architects must select parameters based on business priorities:

                        High Throughput
                              /\
                             /  \
                            /    \
                           /      \
                          /        \
                         /          \
  Low Latency (TTFT/ITL) ----------- Cost Efficiency

• Maximal Throughput (Batch Processing): Maximize max_num_batched_tokens (e.g., 8192+), set high engine concurrency, and utilize dynamic quantization (e.g., FP8 / AWQ). This maximizes GPU compute utilization at the expense of slightly elevated inter-token latency.
• Minimal Latency (Real-time Chat Agents): Keep tensor_parallel_size restricted to intra-node NVLink bounds (typically 4 or 8 GPUs on a single host), enable small prefill chunking sizes, and cap max ongoing requests per replica at lower values (e.g., 10-20).
• Cost Efficiency (Scale-to-Zero): Combine KubeRay autoscaling with cloud provider spot instances and local model caching, sacrificing cold start times for idle cost elimination.

Conclusion

Scaling enterprise-grade LLM inference isn't solved by wrapping model scripts in basic Docker containers. By layering vLLM for physical VRAM memory optimization via PagedAttention, Ray for actor distribution across heterogeneous GPU clusters, and Kubernetes for declarative orchestration, platform teams can run state-of-the-art multi-billion parameter models at production scale with low latency, deterministic failover, and optimal GPU cost efficiency.