Zero-Downtime LLM Deployment: Scaling vLLM on Kubernetes with Dynamic GPU Partitioning

Deploying Large Language Models (LLMs) at enterprise scale introduces an excruciating trade-off: cost efficiency versus tail latency.

While frameworks like vLLM have revolutionized inference throughput using PagedAttention, managing the underlying infrastructure on Kubernetes remains a bottleneck. Standard Kubernetes Pod Autoscalers (HPA) rely on CPU or memory metrics—completely blind to LLM-specific bottlenecks like KV-cache saturation or request queue depth. Furthermore, assigning entire NVIDIA A100/H100 GPUs to small-to-medium parameter models (e.g., Llama-3-8B, Mistral-7B) wastes massive amounts of compute budget.

To achieve zero-downtime, sub-second latency, and maximum GPU density, you must combine NVIDIA Multi-Instance GPU (MIG) dynamic partitioning, custom Prometheus metrics, and KEDA (Kubernetes Event-driven Autoscaling).

Here is the blueprint for building a production-ready, auto-scaling vLLM infrastructure on Kubernetes.

End-to-End System Architecture

Before diving into manifests, let’s look at the request flow and control plane interaction:

                          [ Client Traffic ]
                                  │
                                  ▼
                        [ Ingress Controller ]
                                  │
           ┌──────────────────────┴──────────────────────┐
           ▼                                             ▼
  [ vLLM Pod (MIG 3g.40gb) ]           [ vLLM Pod (MIG 3g.40gb) ]
   ├── Prometheus Metrics               ├── Prometheus Metrics
   └── KV Cache Monitoring              └── KV Cache Monitoring
           │                                             │
           └──────────────────────┬──────────────────────┘
                                  │ (Scrape /metrics)
                                  ▼
                        [ Prometheus Server ]
                                  │
                                  ▼
                      [ KEDA Metrics Server ]
                                  │
                                  ▼
                     [ Kubernetes HPA Controller ]
                                  │
               ┌──────────────────┴──────────────────┐
               ▼                                     ▼
     (Scale Out vLLM Pods)               (Trigger Node Drain & 
                                          Re-partition GPU Profile)

Phase 1: Dynamic GPU Partitioning with NVIDIA MIG

NVIDIA’s Multi-Instance GPU (MIG) technology allows GPUs like the A100 (40GB/80GB) and H100 (80GB) to be partitioned into up to seven independent GPU instances. Unlike time-slicing, MIG provides hardware isolation for memory, cache, and compute cores (Streaming Multiprocessors).

1. Configuring NVIDIA GPU Operator with mig-parted

To dynamically change MIG configurations without destroying the host node, configure the NVIDIA GPU Operator using a ConfigMap that defines custom MIG profiles.

Here is an example definition (mig-config.yaml) splitting an A100-80GB into two 3g.40gb slices for medium models, or four 2g.20gb slices for smaller workloads:

apiVersion: v1
kind: ConfigMap
metadata:
  name: default-mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-disabled:
        - devices: ["all"]
          mig-enabled: false
      custom-vllm-balanced:
        - devices: ["0"]
          mig-enabled: true
          mig-devices:
            "3g.40gb": 2
      custom-vllm-high-density:
        - devices: ["0"]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7

Apply this configuration dynamically to host nodes by updating the node label managed by the GPU Operator:

kubectl label nodes gpk-node-01 nvidia.com/mig.config=custom-vllm-balanced --overwrite

The GPU Operator will re-configure the physical GPU, and the K8s kubelet will export new allocatable resources: nvidia.com/mig-3g.40gb.

Phase 2: Production-Grade vLLM Kubernetes Deployment

vLLM requires precise setup to manage GPU memory allocation correctly. The critical parameter here is --gpu-memory-utilization, which dictates how much memory vLLM reserves for weights and the PagedAttention KV cache.

Production Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
  namespace: llm-serving
  labels:
    app.kubernetes.io/name: vllm-llama3-8b
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: vllm-llama3-8b
  template:
    metadata:
      labels:
        app: vllm-llama3-8b
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: vllm-engine
        image: vllm/vllm-openai:v0.6.3
        args:
        - "--model"
        - "meta-llama/Meta-Llama-3-8B-Instruct"
        - "--port"
        - "8000"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-model-len"
        - "8192"
        - "--tensor-parallel-size"
        - "1"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        - name: VLLM_USAGE_SOURCE
          value: "production"
        ports:
        - containerPort: 8000
          name: http
        resources:
          limits:
            cpu: "8"
            memory: 32Gi
            nvidia.com/mig-3g.40gb: "1"
          requests:
            cpu: "4"
            memory: 16Gi
            nvidia.com/mig-3g.40gb: "1"
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
        volumeMounts:
        - mountPath: /root/.cache/huggingface
          name: model-cache
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache-pvc

Key Architecture Points:

1. Resource Boundary: Requesting nvidia.com/mig-3g.40gb: "1" binds the pod strictly to a single 40GB MIG slice.
2. Weight Pre-caching: Model weights are stored on a fast PersistentVolumeClaim (NVMe-backed) to prevent downloading 16GB+ weights over the public internet during auto-scaling events, reducing cold starts from 6 minutes to under 30 seconds.
3. Graceful PreStop Hook: sleep 15 allows the Ingress controller to remove the Pod IP from the endpoints list before the process terminates, enabling zero-dropped requests during dynamic updates.

Phase 3: Real-Time Metrics & Autoscaling via KEDA

Standard CPU scaling breaks down with LLMs. A pod can sit at 15% CPU load while fully saturated because its KV Cache is 100% full, causing incoming requests to queue up or error out.

We must scale based on exposed vLLM metrics:

• vllm:gpu_cache_usage_perc: Percentage of GPU KV-cache blocks currently utilized.
• vllm:num_requests_waiting: The number of requests sitting in the processing queue.

1. Configure Prometheus Scraping (ServiceMonitor)

If using Prometheus Operator, create a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm-metrics-monitor
  namespace: llm-serving
spec:
  selector:
    matchLabels:
      app: vllm-llama3-8b
  endpoints:
  - port: http
    path: /metrics
    interval: 5s

2. Configure KEDA ScaledObject

KEDA will poll Prometheus and adjust the deployment replicas dynamically. We trigger scaling when KV Cache exceeds 75% OR when queued requests exceed 5.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-keda-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 10
            periodSeconds: 60
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_gpu_cache_usage_perc
      query: sum(vllm:gpu_cache_usage_perc{namespace="llm-serving"}) / count(vllm:gpu_cache_usage_perc{namespace="llm-serving"})
      threshold: '0.75'
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{namespace="llm-serving"})
      threshold: '5'

Phase 4: Zero-Downtime Deployment & Rolling Updates

Updating model weights or vLLM container versions without dropping requests requires careful coordination between Kubernetes readiness probes and ingress controllers.

1. Readiness Probe Warm-up Problem

When vLLM boots, it initializes CUDA contexts, downloads missing layers, and pre-allocates the KV-cache. This process can take several minutes.

• If the readinessProbe succeeds too early, traffic hits the pod before CUDA memory allocation finishes, resulting in HTTP 500.
• If it succeeds too late, rollout operations stall out.

Using vLLM's /health endpoint natively checks engine initialization state. Kubernetes will not route traffic via the Endpoint/Service until this health check returns HTTP 200.

2. Draining In-Flight Requests on Termination

When an instance scales down or updates during a rollout:

1. Kubernetes sets pod status to Terminating.
2. Endpoint controller removes Pod IP from Service backend pool.
3. The preStop hook triggers, pausing execution for 15s to allow load balancers to update routing tables.
4. SIGTERM is issued to vLLM. vLLM finishes processing current tokens for existing active streams before completely exiting.

Phase 5: Dynamic GPU Re-Partitioning Workflow

What happens when demand shifts from a small model (e.g., Llama-3-8B) to a massive model requiring a full unpartitioned GPU (e.g., Llama-3-70B)?

You must implement an automated node drain and re-partition sequence:

# Step 1: Cordon and Drain Node to evict existing MIG workloads
kubectl cordon gpk-node-01
kubectl drain gpk-node-01 --ignore-daemonsets --delete-emptydir-data --force

# Step 2: Change MIG profile label on host node
kubectl label node gpk-node-01 nvidia.com/mig.config=all-disabled --overwrite

# Step 3: Wait for GPU Operator to re-apply physical profile (verified via GPU operator logs)
kubectl get node gpk-node-01 -o jsonpath='{.status.capacity}' | jq .

# Step 4: Uncordon node to accept non-MIG heavy workloads
kubectl uncordon gpk-node-01

By scripting this workflow into a custom operator or Argo Workflows, you can automatically adjust your physical cluster geometry based on peak enterprise usage cycles (e.g., daytime interactive micro-models vs. nighttime batch processing macro-models).

Production Operational Checklist

| Metric / Objective | Standard Benchmark | Optimization Target | | :--- | :--- | :--- | | Model Load Time | ~4-6 minutes (S3/HF Direct) | < 30 seconds (Pre-warmed PVC/NVMe) | | Scaling Metric Trigger | CPU / RAM Utilization | KV Cache % (vllm:gpu_cache_usage_perc) | | Isolation Mechanism | Software Time-Slicing | Hardware Isolation via NVIDIA MIG | | Rollout Strategy | Default maxUnavailable: 25% | maxUnavailable: 0 + preStop Delay | | Target Cache Limit | N/A | 75% Cache Threshold (Prevents OOM Queueing) |

Summary

By coupling vLLM with NVIDIA MIG and KEDA, you eliminate both financial inefficiency and tail-latency spikes. Hardware memory boundaries ensure predictable performance across multiple operational deployments on a single physical GPU, while real-time metrics tracking KV cache utilization guarantees that autoscaling events trigger before requests hit execution bottlenecks.