Optimizing LLM Inference on Kubernetes: Dynamic GPU Partitioning with vLLM and KEDA

Serving Large Language Models (LLMs) in enterprise production environments is a masterclass in capital inefficiency. In a typical naive deployment, platform teams bind entire $30,000+ NVIDIA A100 or H100 GPUs—or equivalent cloud instances costing upwards of $3.50/hour per GPU—to static Kubernetes deployments running a single LLM instance.

When user traffic drops off during off-peak hours, those GPUs sit idle, drawing full power and incurring flat-rate operational costs. Conversely, when traffic spikes, static deployments suffer from queue saturation, tail-latency spikes, and Out-Of-Memory (OOM) crashes due to unbounded Key-Value (KV) cache allocation.

To achieve enterprise-grade reliability, ultra-low latency, and sanity in your AI infrastructure bill, you need an architecture that dynamically allocates compute and scales strictly based on inference queue demand.

In this post, we will build a production-ready, low-latency LLM inference pipeline on Kubernetes that combines:

1. Dynamic GPU Partitioning (NVIDIA MIG and Time-Slicing) to increase resource density.
2. vLLM for state-of-the-art PagedAttention, continuous batching, and high-throughput serving.
3. KEDA (Kubernetes Event-driven Autoscaling) to scale pods rapidly using real-time inference telemetry.

By the end of this guide, you will understand how to implement this architecture to slash your LLM hosting infrastructure costs by up to 60%.

The Architectural Flaw in Naive LLM Deployments

Standard Kubernetes autoscaling using the Horizontal Pod Autoscaler (HPA) relies on CPU and memory utilization metrics. For LLM workloads, these metrics are completely blind indicators.

An idle vLLM pod pre-allocates almost the entirety of available GPU VRAM upon initialization to store the model weights and set aside space for the KV cache pool (vllm:gpu_cache_usage_perc). To Kubernetes, this pod appears to be using ~90%+ of its allocated memory continuously—even when serving zero requests per second (RPS). Standard HPA will either fail to scale or trigger endless, incorrect scaling loops.

+-----------------------------------------------------------------------+
|                         NAIVE APPROACH                                |
|  +-------------------+    +-------------------+                       |
|  | Single Pod        |    | Single Pod        |                       |
|  | vLLM (Llama-3-8B) |    | vLLM (Llama-3-8B) |  <-- Compute & VRAM  |
|  +-------------------+    +-------------------+      Wasted at Off-Peak|
|  | 1x Full A100 GPU  |    | 1x Full A100 GPU  |                       |
|  +-------------------+    +-------------------+                       |
+-----------------------------------------------------------------------+
                                  VS
+-----------------------------------------------------------------------+
|                    DYNAMIC PARTITIONED ARCHITECTURE                   |
|  +-----------------------------------------------------------------+  |
|  |                       Single Physical A100                      |  |
|  |  +------------------+  +------------------+  +---------------+  |  |
|  |  | MIG Instance 1   |  | MIG Instance 2   |  | Slice 3 (Idle)|  |  |
|  |  | (vLLM Instance A)|  | (vLLM Instance B)|  | (Reclaimed)   |  |  |
|  |  +------------------+  +------------------+  +---------------+  |  |
|  +-----------------------------------------------------------------+  |
|            ^ Autoscaled via KEDA using custom queue depth metrics     |
+-----------------------------------------------------------------------+

Furthermore, running medium-sized parameters models (such as Llama-3-8B or Mistral-7B in FP16/INT4) on a full 80GB A100 GPU wastes vast amounts of Tensor Core compute power. The model easily fits within 16-20 GB of VRAM, leaving the remaining 60GB and compute capacity severely underutilized unless multi-tenant slicing is implemented.

Core Infrastructure Components

1. Engine Layer: vLLM & PagedAttention

vLLM fundamentally transforms memory efficiency by addressing the primary bottleneck in LLM serving: non-contiguous, highly fragmented KV cache memory allocation. Through PagedAttention, vLLM manages KV cache tensors in virtual memory blocks, enabling near-zero memory waste. Key configurations for production include:

• --gpu-memory-utilization: Defines the strict cap of VRAM reserved for weights and KV cache.
• --max-num-seqs / --max-model-len: Prevents unbounded context growth from triggering CUDA OOMs.
• --enable-chunked-prefill: Decouples prompt processing (compute-bound) from token generation (memory-bound) to drastically stabilize time-to-first-token (TTFT) and inter-token latency (ITL).

2. Slicing Layer: MIG vs. Time-Slicing vs. MPS

To maximize density on Kubernetes nodes, we slice underlying physical GPUs using one of three strategies:

| Strategy | Hardware Support | Fault Isolation | Memory Isolation | Performance Overhead | Best Use Case | | :--- | :--- | :--- | :--- | :--- | :--- | | NVIDIA MIG | A100 / H100 | Hardware-level | Strict / Dedicated | Zero | Production multi-tenant, strict SLA guarantees | | NVIDIA Time-Slicing | All CUDA GPUs | Soft (Process-level)| None (Shared VRAM) | Context-switching overhead | High-density dev/test, uniform non-bursty loads | | NVIDIA MPS | Volta & Newer | Soft | Configurable limits | Minimal | Shared memory burstable inference |

For high-reliability production environments running enterprise LLMs, Multi-Instance GPU (MIG) is the gold standard because it guarantees dedicated compute pipelines, cross-bar crossbars, and memory bandwidth at the silicon hardware layer.

3. Scaling Layer: KEDA (Kubernetes Event-driven Autoscaling)

KEDA injects event-driven scalability into Kubernetes by reaching directly into vLLM’s Prometheus metrics endpoint. Instead of checking pod CPU, KEDA evaluates metrics critical to LLM performance:

• vllm:num_requests_waiting: The length of the continuous batching waiting queue.
• vllm:gpu_cache_usage_perc: Real-time saturation of the internal KV cache pool.

Production Implementation Guide

Let's build a fully partitioned, autoscaled inference pipeline step by step.

Step 1: Configure GPU Partitioning via NVIDIA GPU Operator

Deploy the NVIDIA GPU Operator with dynamic MIG profile slicing or time-slicing enabled. Below is an example custom configuration applying MIG strategies to partition an A100-80GB into multiple sub-allocations (e.g., mig-2g.20gb slices capable of running Llama-3-8B).

# gpu-operator-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: default-mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-balanced:
        - device-filter: ["0x20b010de-0x00000000"] # A100-PCIE-40GB/80GB PCI ID
          devices: [0]
          mig-enabled: true
          mig-devices:
            "2g.20gb": 4 # Slices single 80GB GPU into 4 x 20GB distinct hardware instances

Apply the profile via node labeling:

kubectl label nodes gke-node-a100-pool-1 nvidia.com/mig.config=all-balanced --overwrite

Step 2: Deploy vLLM with Partitioned GPU Boundaries

Now we construct the Kubernetes deployment. The pod requests exact MIG resource capacities (nvidia.com/mig-2g.20gb: "1"), enforcing rigid platform constraints.

# vllm-llama3-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
  namespace: llm-serving
  labels:
    app: vllm-llama3-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama3-8b
  template:
    metadata:
      labels:
        app: vllm-llama3-8b
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: vllm-server
        image: vllm/vllm-openai:v0.6.3
        args:
        - "--model"
        - "meta-llama/Meta-Llama-3-8B-Instruct"
        - "--port"
        - "8000"
        - "--gpu-memory-utilization"
        - "0.90"
        - "--max-model-len"
        - "8192"
        - "--enable-chunked-prefill"
        - "true"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        resources:
          limits:
            cpu: "8"
            memory: "32Gi"
            nvidia.com/mig-2g.20gb: "1"
          requests:
            cpu: "4"
            memory: "16Gi"
            nvidia.com/mig-2g.20gb: "1"
        ports:
        - containerPort: 8000
          name: http
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 15

Step 3: Implement Queue-Aware Scaling with KEDA

We configure KEDA to monitor the metrics exposed by vLLM via Prometheus. If the prompt processing queue builds up (vllm:num_requests_waiting > 5) or if VRAM KV cache saturation exceeds 85%, KEDA scales up replicas into adjacent idle MIG slices.

# keda-vllm-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama3-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 600
          policies:
          - type: Percent
            value: 10
            periodSeconds: 60
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{namespace="llm-serving", pod=~"vllm-llama3-8b-.*"})
      threshold: '5'
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_gpu_cache_usage_perc
      query: max(vllm:gpu_cache_usage_perc{namespace="llm-serving", pod=~"vllm-llama3-8b-.*"})
      threshold: '0.85'

Unit Economics & Cost Savings Analysis

To quantify the cost impact, let's analyze a production workload requiring multi-tenant capacity for continuous low-latency inference on Llama-3-8B.

Baseline: Traditional Deployment Model

• Hardware: 4x Dedicated Unpartitioned NVIDIA A100 (80GB) cloud instances.
• Cost per Instance: ~$3.67 / hour.
• Total Hourly Rate: $3.67 × 4 = $14.68 / hour.
• Monthly Footprint: $14.68 × 730 hours = $10,716.40 / month.
• Utilization Efficiency: Average GPU Compute utilization hovers between 12-18% during off-peak windows.

Optimized Architecture: MIG Partitioning + KEDA

• Hardware: 1x NVIDIA A100 (80GB) dynamic node partitioned via MIG into 4x 2g.20gb slices.
• Scaling Policy: KEDA scales active vLLM replicas dynamically between 1 and 4 based on queue depth.
• Off-Peak Profile (16 hours/day): Runs 1 active slice. Unused slices are yielded or pooled.
• Peak Profile (8 hours/day): Scales dynamically to 4 active slices on a single node.
• Effective Hourly Spend (Blended): ~$1.46 / hour average effective node cost due to dynamic resource scheduling and cloud spot/reclaimed capability.
• Monthly Footprint: ~$4,286.00 / month.

Financial Result: 59.98% direct infra cost reduction (~$6,430 monthly savings per single GPU cluster) while preserving low p99 latencies under peak traffic.

Production Pitfalls & Hardening Strategies

Building optimized LLM infrastructure requires avoiding common production failure modes:

1. Eliminating Cold-Start Latency Drops

Large model weights (e.g., 16GB+ safetensors) take a significant amount of time to pull over standard container network registries during container creation.

• Mitigation: Pre-warm nodes using dynamic volume mounts backed by distributed read-only storage (e.g., AWS EFS with provisioned throughput, GCP Filestore, or local NVMe daemonset caching). Mount weights via standard PVCs:

volumeMounts:
- name: model-weights
  mountPath: /root/.cache/huggingface
volumes:
- name: model-weights
  persistentVolumeClaim:
    claimName: llama3-weights-pvc

2. Tuning KEDA Cooldown Windows against Flapping

LLM prompt bursts are often swingy. If KEDA scales down too quickly (cooldownPeriod), Kubernetes terminates pods whose startup cost (CUDA initialization + weight loading into VRAM) takes up to 90 seconds.

• Mitigation: Set cooldownPeriod: 300 or higher, and use stabilization windows in the scale-down policies to allow pods to stay warm through minor traffic dips.

3. Graceful Termination & Queue Draining

Terminating a vLLM instance mid-generation drops client TCP streams, resulting in severe client-side execution errors.

• Mitigation: Define a preStop hook to trigger graceful connection draining, allowing active continuous batching loops to complete generation while marking the endpoint unready:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

Wrapping Up

Maximizing performance while controlling costs in generative AI infrastructure requires moving beyond simple CPU/Memory abstractions. By combining vLLM's memory management, NVIDIA GPU partitioning (MIG/Time-slicing), and queue-aware autoscaling with KEDA, platform engineers can build elastic, high-throughput inference environments that scale strictly with business demand.

Implementing these practices transforms GPU infrastructure from a static cost center into a dynamic, highly performant component of your cloud architecture.

Have you implemented GPU dynamic partitioning or vLLM autoscaling in your production stacks? Connect with the technical team at Ecstaticloud to share your performance benchmarks or architectural challenges.