Architecting High-Throughput LLM Inference Pipelines on Kubernetes with Ray and vLLM

Deploying Large Language Models (LLMs) into production presents a stark contrast to traditional microservices. While microservices are generally CPU-bound or memory-predictable, LLM inference workloads are exceptionally compute-heavy, memory-constrained, and sensitive to request sequence lengths.

A naive deployment of a 70B parameter model on standard GPU pods often leads to catastrophic resource underutilization. GPUs spend up to 70% of their operational cycles idling, waiting for sequential token generation or blocked by fragmented Key-Value (KV) cache memory.

To achieve enterprise-grade throughput and minimal latency, you must fundamentally change how memory and dynamic compute are managed. In this guide, we will architect a production-ready, high-throughput LLM inference pipeline on Kubernetes using KubeRay for distributed orchestration and vLLM for optimized engine execution.

The Root Causes of LLM Inference Inefficiencies

Understanding why standard serving stacks fail requires examining the execution mechanics of autoregressive transformer models.

       +-------------------------------------------------------------------+
       |                        GPU VRAM Allocation                        |
       +-------------------------------------------------------------------+
       |  Model Weights  |      KV Cache (PagedAttention)      |  Compute  |
       |  (FP16/INT4)    |  [Block 0] [Block 1] [Block 2] ...  | Workspace |
       +-------------------------------------------------------------------+

1. KV Cache Memory Overhead: During the decoding phase, previous token representations (Keys and Values) are cached to avoid redundant computation. Traditional engines pre-allocate contiguous memory spaces for this cache based on the maximum sequence length (e.g., 4096 tokens). If a request only generates 128 tokens, the remaining pre-allocated memory is wasted. This memory fragmentation leads to low batch sizes and underutilized tensor cores.
2. Static vs. Dynamic Batching: Classical batching waits for a set of requests to arrive, packages them together, and processes them concurrently. However, because text generation length varies per prompt, short sequences finish early and sit idle while waiting for long sequences to complete execution iteration loops.
3. Hardware-Unaware Scaling: Standard Kubernetes Horizontal Pod Autoscalers (HPA) relying on CPU/Memory metrics are completely blind to GPU internal bottlenecks such as KV Cache saturation, execution queue depth, or Time-To-First-Token (TTFT) degradation.

The Solution Architecture Stack

Our high-throughput pipeline brings together three core technology layers:

• Kubernetes: Enterprise control plane, providing physical infrastructure abstractions, storage mapping, and GPU scheduling via the NVIDIA Kubernetes Device Plugin.
• KubeRay (Ray Operator): Orchestrates distributed Ray clusters on top of Kubernetes. It manages master/worker worker groups, handles distributed process execution, and facilitates seamless Tensor Parallelism across multi-GPU nodes.
• vLLM Engine: Provides the core inference runtime using PagedAttention (virtual memory paging for KV cache) and Continuous Batching (iteration-level dynamic scheduling), yielding up to 24x higher throughput compared to unoptimized HuggingFace Transformers.

End-to-End System Architecture

graph TD
    Client[Client Requests] --> Ingress[Kubernetes Ingress / Gateway API]
    Ingress --> Router[Ray Serve Router Pod]
    
    subgraph RayCluster ["KubeRay Cluster Deployment"]
        Router --> HeadNode[Ray Head Node - Controller]
        
        subgraph WorkerGroup ["Ray Worker Pod Group (GPU Nodes)"]
            HeadNode --> Worker1[Ray Worker Pod 1\nvLLM Engine Engine (GPU 0..3)]
            HeadNode --> Worker2[Ray Worker Pod 2\nvLLM Engine Engine (GPU 4..7)]
        end
    end

    Worker1 --> NVLink1[(Local GPU VRAM\nKV Cache Paged)]
    Worker2 --> NVLink2[(Local GPU VRAM\nKV Cache Paged)]

Deep Dive: Memory & Batching Optimization Mechanisms

Before deploying manifests, let's configure vLLM's memory management engine correctly.

1. PagedAttention

PagedAttention brings OS-style virtual memory management to LLM serving. It breaks the KV cache into non-contiguous physical blocks (typically size 16 or 32 tokens). As sequences grow, blocks are dynamically allocated on-demand without requiring physical memory continuity.

• Result: Memory waste drops from ~60-80% to under 4%. This freed-up memory directly translates into larger dynamic batch sizes.

2. Continuous (Iteration-Level) Batching

Instead of waiting for an entire batch of requests to finish, continuous batching operates at the iteration level. As soon as a sequence completes generation, a new sequence from the queue is swapped into the batch space on the next forward pass.

Engine Configuration Strategy

To achieve optimum performance on NVIDIA H100/A100 GPUs, apply these critical deployment flags:

• --gpu-memory-utilization: Defines the fraction of GPU memory allocated to model weights and KV cache. Set this to 0.92 to leave headroom for CUDA execution contexts.
• --max-num-batched-tokens: Max total tokens processed in a single forward pass iteration. For high throughput, scale this to 32768 or higher.
• --max-num-seqs: Limits maximum concurrent sequences. Balance this value with your average prompt + output token lengths to avoid KV Cache OOM spillover.

Production Deployment Implementation

Let's build a production-grade KubeRay service that deploys a distributed vLLM instance serving Llama-3-70B-Instruct across 4x NVIDIA A100 (80GB) GPUs using Tensor Parallelism (tp=4).

Step 1: Ray Serve Application Wrapper

First, define the Python runtime entrypoint (app.py). This script initializes the distributed vLLM AsyncLLMEngine inside a Ray Serve deployment context.

import os
import asyncio
from typing import AsyncGenerator
from ray import serve
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
from vllm.utils import random_uuid

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_cpus": 8, "num_gpus": 4}
)
class VLLMDeployment:
    def __init__(self):
        model_id = os.getenv("MODEL_ID", "meta-llama/Meta-Llama-3-70B-Instruct")
        
        engine_args = AsyncEngineArgs(
            model=model_id,
            tensor_parallel_size=4,
            gpu_memory_utilization=0.92,
            max_model_len=8192,
            max_num_seqs=256,
            trust_remote_code=True,
            enforce_eager=False, # Use CUDA Graphs for max throughput
        )
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    async def generate(self, prompt: str, sampling_params_dict: dict) -> AsyncGenerator[str, None]:
        sampling_params = SamplingParams(**sampling_params_dict)
        request_id = f"req-{random_uuid()}"
        
        results_generator = self.engine.generate(prompt, sampling_params, request_id)
        
        async for request_output in results_generator:
            yield request_output.outputs[0].text

    async def __call__(self, request) -> dict:
        json_data = await request.json()
        prompt = json_data.get("prompt")
        sampling_params = json_data.get("sampling_params", {})
        
        # Stream response or aggregate output
        generator = self.generate(prompt, sampling_params)
        full_text = ""
        async for text_chunk in generator:
            full_text = text_chunk
            
        return {"text": full_text}

app = VLLMDeployment.bind()

Step 2: KubeRay RayService CRD Deployment

Now, write the custom resource definition (ray-service-vllm.yaml) that provisions the underlying infrastructure on Kubernetes, including shared memory volume mounts (/dev/shm) necessary for Fast Inter-Process Communication (IPC) during Tensor Parallelism execution.

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: vllm-llama3-70b
  namespace: llm-serving
spec:
  serviceUnhealthyThreshold: 300
  rayClusterConfig:
    rayVersion: '2.9.0'
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.9.0-py310-gpu
              resources:
                limits:
                  cpu: "8"
                  memory: "32Gi"
                requests:
                  cpu: "4"
                  memory: "16Gi"
              ports:
                - containerPort: 6379
                  name: gcs
                - containerPort: 8265
                  name: dashboard
                - containerPort: 8000
                  name: serve
    workerGroupSpecs:
      - groupName: gpu-group
        replicas: 2
        minReplicas: 1
        maxReplicas: 4
        rayStartParams: {}
        template:
          spec:
            containers:
              - name: ray-worker
                image: rayproject/ray:2.9.0-py310-gpu
                securityContext:
                  capabilities:
                    add: ["SYS_PTRACE"]
                resources:
                  limits:
                    cpu: "32"
                    memory: "256Gi"
                    nvidia.com/gpu: "4"
                  requests:
                    cpu: "16"
                    memory: "128Gi"
                    nvidia.com/gpu: "4"
                volumeMounts:
                  - mountPath: /dev/shm
                    name: dshm
            volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 16Gi
            nodeSelector:
              accelerator: nvidia-a100-80gb

Autoscaling & Cost Optimization Strategies

Scale management for GPU compute demands metrics that reflect model internal state rather than OS generic counters.

Metrics-Driven Autoscaling with KEDA

vLLM exports native Prometheus metrics via its HTTP runtime server. Standard metrics to build autoscaling policies (via KEDA - Kubernetes Event-driven Autoscaling) include:

1. vllm:num_requests_waiting: The number of requests sitting in queue due to full KV cache spaces.
2. vllm:gpu_cache_usage_perc: Percentage of physical KV cache blocks occupied.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-keda-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: ray.io/v1
    kind: RayService
    name: vllm-llama3-70b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting{namespace="llm-serving"})
        threshold: '10'

Spot/Preemptible Instance Strategy

GPUs like A100s/H100s are costly. You can leverage Spot Instances safely using Ray Serve's architectural resilience:

• Ray Head Node: Run on a On-Demand system node pool (no GPUs required).
• Ray Worker Nodes: Run on Spot/Preemptible GPU node pools.
• Drain Management: Ensure terminationGracePeriodSeconds is set to at least 60 seconds on worker pods. KubeRay will gracefully divert incoming API routing from terminating worker nodes to remaining active nodes while in-flight generations drain out.

Monitoring Key Performance Indicators (KPIs)

To validate pipeline efficiency, monitor the following production metrics using Grafana:

| Metric | Target | Optimization Action if Violating | | :--- | :--- | :--- | | TTFT (Time-To-First-Token) | < 200ms | Increase prefill processing capabilities or lower --max-num-batched-tokens. | | TPOT (Time-Per-Output-Token) | < 30ms / token | Adjust Tensor Parallelism or apply model quantization (e.g., AWQ/FP8). | | KV Cache Usage % | 70% – 85% steady state | If constantly >95%, scale out replicas or decrease max_model_len. | | Queue Depth | 0 spikes under normal load | Scale Ray Worker replicas horizontally using KEDA. |

Conclusion

By moving away from naive microservice paradigms and pairing KubeRay with vLLM, you eliminate memory fragmentation via PagedAttention and maximize hardware execution saturation via continuous iteration batching.

This setup transforms Kubernetes into an adaptive AI inference engine capable of serving high-throughput LLM traffic efficiently, reducing operational costs, and maintaining sub-second user latency SLAs at scale.