Scaling Local LLM Inference: Optimizing vLLM and Ray Clusters on Kubernetes | Ecstaticloud Blog

Serving open-weights Large Language Models (LLMs) like Meta's Llama 3, Mistral, and DeepSeek in enterprise environments has shifted from simple single-node experimentation to high-throughput, distributed cloud deployment. While single-GPU frameworks (such as Ollama or basic Transformers pipelines) work well for dev environments, production SLA requirements demand two conflicting goals: minimizing Time-To-First-Token (TTFT) for user interactivity and maximizing aggregate throughput (tokens/second) for batch workloads—all while keeping compute costs optimized.

To achieve this, modern AI platform engineering relies on a three-tiered stack:

vLLM as the high-performance inference engine leveraging dynamic memory allocation and efficient tensor ops.
Ray (specifically Ray Serve) to orchestrate model execution across multiple nodes and handle distributed execution graphs.
Kubernetes (via KubeRay) as the infrastructure substrate providing automated container orchestration, GPU resource allocation, and elastic autoscaling.

In this deep dive, we will architecture a production-ready, distributed LLM inference engine on Kubernetes that handles multi-node scaling, minimizes latency overhead, and dynamically adjusts compute footprint based on request queue depth.

Architectural Overview

Before diving into configurations, let's establish the flow of a request through a KubeRay-managed vLLM cluster.

                         [ Client / API Request ]
                                    │
                                    ▼
                      [ Ingress Controller (NGINX) ]
                                    │
                                    ▼
                 [ Ray Serve Head Node (Router Deployment) ]
                                    │
           ┌────────────────────────┴────────────────────────┐
           ▼                                                 ▼
[ Ray Worker Node 1 ]                             [ Ray Worker Node 2 ]
┌─────────────────────────┐                       ┌─────────────────────────┐
│ GPU 0        GPU 1      │  ◄── NCCL Inter-Node ──►│ GPU 0        GPU 1      │
│ ┌──────────┐ ┌──────────┐│     Tensor Parallel  │ ┌──────────┐ ┌──────────┐│
│ │vLLM Engine│ │vLLM Engine││      (Intra-Mesh)    │ │vLLM Engine│ │vLLM Engine││
│ └──────────┘ └──────────┘│                      │ └──────────┘ └──────────┘│
└─────────────────────────┘                       └─────────────────────────┘

The Ray Head Node runs a lightweight API router that receives HTTP requests and proxies them to available Ray Worker deployments. The Ray Worker Nodes host the physical vLLM engines running across allocated GPUs.

Deconstructing the Engine: vLLM Optimizations

To tune this stack effectively, we must first understand the bottlenecks at the engine layer. Standard HuggingFace pipelines suffer from significant memory inefficiency during the Decoding Phase due to key-value (KV) caching.

PagedAttention & Virtual Memory Management

Traditional LLM decoding pre-allocates contiguous memory for the KV cache based on the maximum sequence length ($L_{max}$). If a sequence only uses 512 tokens out of an 8192 token window, over 90% of allocated GPU memory is wasted, leading to severe batch-size limitations.

vLLM resolves this by introducing PagedAttention, inspired by virtual memory paging in operating systems:

Memory for KV cache is divided into fixed-size physical blocks (e.g., 16 or 32 tokens).
Pages are allocated dynamically as tokens are generated.
A logical-to-physical block table maintains mapping without requiring memory contiguity.

Logical KV Cache Pages:   [ Block 0 ] ──► [ Block 1 ] ──► [ Block 2 ]
                               │               │               │
                               ▼               ▼               ▼
Physical GPU Memory:      [ Block 17 ]    [ Block 03 ]    [ Block 89 ]

This reduces KV cache memory wastage to under 4%, allowing you to dramatically increase batch sizes and boost aggregate throughput by up to $3\times\text{--}4\times$.

Parallelism Strategies: Tensor vs. Pipeline

When a model’s parameters exceed single-GPU VRAM (e.g., Llama-3-70B requiring ~140GB in FP16), the workload must be split:

Tensor Parallelism (TP): Splits individual matrix multiplications across GPUs within the same layer. High communication overhead via AllReduce ops.
- Best Practice: Keep TP within a single node using high-bandwidth interconnects (NVLink/NVSwitch, up to 900 GB/s per GPU on H100).
Pipeline Parallelism (PP): Splits layers sequentially across GPUs or nodes. Lower communication bandwidth requirements (only activation tensors passed between layer boundaries).
- Best Practice: Use PP for inter-node communication when cross-node networking is limited to standard Ethernet or lower-bandwidth fabrics.

Formula for GPU Memory Sizing:

$$\text{Required VRAM} \approx \left( \frac{\text{Params (Billions)} \times \text{Bytes per Param}}{\text{TP Size} \times \text{PP Size}} \right) \times 1.2 + \text{KV Cache Allocation}$$

Deploying via KubeRay Operator

To run distributed Ray clusters natively on Kubernetes, we use the KubeRay Operator. KubeRay manages RayCluster and RayService Custom Resources (CRDs), translating cluster definitions into Kubernetes Pods and Services.

Production-Grade `RayService` Manifest

The following manifest deploys Llama-3-70B-Instruct using Tensor Parallelism ($TP=4$) across nodes using Ray Serve and vLLM.

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: vllm-llama3-70b
  namespace: llm-serving
spec:
  serviceUnhealthyThreshold: 300
  rayClusterConfig:
    rayVersion: '2.35.0'
    headGroupSpec:
      rayStartParams:
        dashboard-host: '0.0.0.0'
        num-gpus: '0' # Head node acts purely as an orchestrator/router
      template:
        spec:
          containers:
            - name: ray-head
              image: rayproject/ray:2.35.0-py310
              resources:
                limits:
                  cpu: "8"
                  memory: "32Gi"
                requests:
                  cpu: "4"
                  memory: "16Gi"
              ports:
                - containerPort: 6379
                  name: gcs
                - containerPort: 8265
                  name: dashboard
                - containerPort: 8000
                  name: serve
    workerGroupSpecs:
      - groupName: gpu-group-tp4
        replicas: 2
        minReplicas: 1
        maxReplicas: 4
        rayStartParams:
          node-ip-address: '$MY_POD_IP'
        template:
          spec:
            containers:
              - name: ray-worker
                image: vllm/vllm-openai:v0.6.0
                securityContext:
                  capabilities:
                    add: ["SYS_PTRACE"]
                resources:
                  limits:
                    cpu: "32"
                    memory: "128Gi"
                    nvidia.com/gpu: "4"
                  requests:
                    cpu: "16"
                    memory: "64Gi"
                    nvidia.com/gpu: "4"
                volumeMounts:
                  - mountPath: /dev/shm
                    name: dshm
                  - mountPath: /root/.cache/huggingface
                    name: hf-cache
            volumes:
              - name: dshm
                emptyDir:
                  medium: Memory
                  sizeLimit: 16Gi # Essential for NCCL inter-GPU shared memory ops
              - name: hf-cache
                persistentVolumeClaim:
                  claimName: hf-cache-pvc
  serveConfigV2: |
    applications:
      - name: vllm_app
        route_prefix: "/"
        import_path: ray_serve_vllm:deployment
        runtime_env:
          env_vars:
            MODEL_ID: "meta-llama/Meta-Llama-3-70B-Instruct"
            TENSOR_PARALLEL_SIZE: "4"

Ray Serve Entrypoint Python Script

Here is the corresponding ray_serve_vllm.py file referenced in the import path:

import os
from typing import AsyncGenerator
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import ray
from ray import serve
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
from vllm.utils import random_uuid

app = FastAPI()

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_cpus": 4, "num_gpus": 4}
)
@serve.ingress(app)
class VLLMDeployment:
    def __init__(self):
        model_id = os.getenv("MODEL_ID", "meta-llama/Meta-Llama-3-70B-Instruct")
        tp_size = int(os.getenv("TENSOR_PARALLEL_SIZE", "4"))

        engine_args = AsyncEngineArgs(
            model=model_id,
            tensor_parallel_size=tp_size,
            gpu_memory_utilization=0.90,
            max_model_len=8192,
            enable_chunked_prefill=True, # Critical for TTFT optimization under load
            max_num_batched_tokens=2048,
            trust_remote_code=True,
        )
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    @app.post("/v1/completions")
    async def generate(self, prompt: str, max_tokens: int = 256) -> StreamingResponse:
        sampling_params = SamplingParams(
            temperature=0.7,
            max_tokens=max_tokens,
        )
        request_id = f"cmpl-{random_uuid()}"
        results_generator = self.engine.generate(prompt, sampling_params, request_id)

        async def stream_results() -> AsyncGenerator[str, None]:
            async for request_output in results_generator:
                text_outputs = [output.text for output in request_output.outputs]
                yield f"data: {text_outputs[-1]}\n\n"

        return StreamingResponse(stream_results(), media_type="text/event-stream")

deployment = VLLMDeployment.bind()

Advanced Performance Tuning Matrix

Deploying the stack is only step one. Minimizing TTFT and maximizing throughput requires precise kernel-level and runtime configuration.

1. Minimizing Inter-Node Communication Overhead (NCCL)

When doing Tensor Parallelism across multi-GPU setups or Pipeline Parallelism across nodes, CUDA Inter-Process Communication (IPC) and NCCL performance dictate efficiency.

Shared Memory (/dev/shm): Always mount an emptyDir with medium: Memory to /dev/shm. NCCL uses shared memory for intra-node GPU transfers. Insufficient /dev/shm leads to immediate Bus Errors or silent degradation down to socket speeds.

NCCL Environment Flags:

env:
  - name: NCCL_DEBUG
    value: "INFO"
  - name: NCCL_IB_DISABLE
    value: "0" # Force InfiniBand / RoCE usage if present
  - name: NCCL_SOCKET_IFNAME
    value: "eth0" # Pin interface to prevent routing loops

2. Chunked Prefills (`enable_chunked_prefill=True`)

LLM request processing consists of two stages:

Prefill Phase: Computes KV cache for all prompt tokens simultaneously (Compute-bound).
Decode Phase: Generates tokens one by one (Memory bandwidth-bound).

Large prefill requests stall ongoing token generation, spiking overall TTFT and causing high latency variance.

Chunked Prefills slice large prompt processing into smaller chunks, interleaving them with decode operations from other concurrent requests.

Without Chunked Prefill:
[ Large Prompt Prefill (100ms delay) ] ──► [ Decode ] ──► [ Decode ]

With Chunked Prefill:
[ Chunk 1 ] ──► [ Decode Req B ] ──► [ Chunk 2 ] ──► [ Decode Req B ]

Recommended Setting: Set max_num_batched_tokens=2048 and enable enable_chunked_prefill=True.

3. Tuning Memory Budgets

Metric-Driven Autoscaling with KEDA & Prometheus

Standard Kubernetes Horizontal Pod Autoscaler (HPA) metrics based on CPU or Memory usage are useless for LLM workloads, as GPU memory allocation remains near-100% constant once the engine initialized its KV Cache.

To scale intelligently, we must scale based on queue metrics and KV Cache saturation exposed by vLLM’s /metrics endpoint:

vllm:num_requests_waiting: The number of requests sitting in the vLLM scheduler queue.
vllm:gpu_cache_usage_perc: Percentage of GPU KV cache blocks currently allocated.

Step 1: Prometheus Metrics Scraping

Ensure your ServiceMonitor or Prometheus Pod annotations scrape the vLLM ports (8000).

Step 2: KEDA ScaledObject Configuration

Deploy a KEDA ScaledObject that dynamically adjusts the minReplicas and maxReplicas of our Ray Worker groups based on real-time inference load:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ray-worker-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: ray.io/v1
    kind: RayService
    name: vllm-llama3-70b
  minReplicaCount: 1
  maxReplicaCount: 4
  cooldownPeriod: 300
  pollingInterval: 15
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        query: |
          sum(vllm:num_requests_waiting{namespace="llm-serving"})
        threshold: '5' # Scale up if more than 5 requests are queued globally
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_gpu_cache_usage_perc
        query: |
          avg(vllm:gpu_cache_usage_perc{namespace="llm-serving"})
        threshold: '0.85' # Scale up if average KV Cache utilization exceeds 85%

Production Readiness Checklist

Before moving your distributed vLLM Kubernetes setup to live production traffic, verify the following configuration checklist:

Memory Locking (ulimit): Ensure IPC limits are unrestricted for NCCL ring buffers.
```
securityContext:
  capabilities:
    add: ["IPC_LOCK"]
```
Readiness Probes: Configure proper health checks against Ray's health endpoint rather than TCP socket checks, preventing traffic routing during active model loading phases:
```
readinessProbe:
  httpGet:
    path: /-/healthz
    port: 8000
  initialDelaySeconds: 120
  periodSeconds: 10
```
Model Weight Caching: Mount high-performance persistent storage (RWO ReadWriteOnce fast NVMe PVCs) to pre-warm weights, cutting initial cold-start pod bootstrap times from tens of minutes down to seconds.

Summary

Scaling LLM inference natively on Kubernetes requires optimizing the full stack: using vLLM for internal KV Cache management and tensor parallelism, Ray for multi-node process orchestration, and KubeRay + KEDA for declarative lifecycle management and dynamic queue-based autoscaling.

By utilizing PagedAttention, tuning batch execution parameters, and exposing hardware-aware metrics for scale operations, you can maintain sub-second TTFT SLAs while scaling compute nodes efficiently to handle dynamic production traffic.