Running Large Language Models (LLMs) in production presents a stark dilemma: maintain fixed, over-provisioned GPU capacity and watch your cloud invoice explode, or implement aggressive autoscaling and suffer from catastrophic 90-second cold starts that destroy user experience.
Standard Kubernetes Horizontal Pod Autoscalers (HPA) relying on CPU or RAM utilization fail spectacularly when orchestrating generative AI workloads. GPUs appear idle to standard host metrics while their VRAM KV-cache is completely saturated, or conversely, GPUs report 100% utilization while spending most of me-time stalled on I/O.
At Ecstaticloud, we re-architected our serverless AI inference infrastructure to achieve sub-second scaling responses, zero-downtime traffic shifts, and a sustained 60% reduction in cloud compute costs.
This post details our end-to-end blueprint for building high-throughput, low-latency LLM inference pipelines using Kubernetes, vLLM, KEDA, dynamic GPU slicing, and optimized model-streaming layers.
Technical Architecture Overview
Before diving into individual components, let us look at the complete request lifecycle and autoscaling topology.
+-----------------------+
| Incoming Traffic |
+-----------------------+
|
v
+-----------------------+
| Envoy / Gateway |
+-----------------------+
|
+---------------------------+---------------------------+
| |
v v
+-----------------------+ +-----------------------+
| Prometheus Metrics | | Distributed Cache Layer|
| (vLLM Queue & Cache) | | (NVMe / Mountpoint S3)|
+-----------------------+ +-----------------------+
| |
v v
+-----------------------+ +-----------------------+
| KEDA Scaler | | Fast Weight Streaming|
+-----------------------+ +-----------------------+
| |
+---------------------------+---------------------------+
|
v
+-------------------------------+
| Kubernetes Node Pool (NVIDIA) |
| +-------------------------+ |
| | vLLM Engine + MPS / MIG | |
| | [Pod 1] [Pod 2] [Pod 3] | |
| +-------------------------+ |
+-------------------------------+
The system relies on three core feedback loops:
- Metrics Aggregation: vLLM exposes deep engine telemetry (KV-cache usage, queue depth, token generation rates).
- Event-Driven Autoscaling: KEDA evaluates custom metrics continuously, triggering rapid horizontal expansion or node-scale events.
- Optimized Provisioning: Kubernetes nodes utilize warm NVMe local caches and GPU memory-slicing (NVIDIA MPS/MIG) to register new model instances instantly.
Pillar 1: Dynamic GPU Slicing & Memory Partitioning
Running a full Llama-3-8B model on a dedicated NVIDIA A100 (80GB) for low-QPS internal microservices results in atrocious ROI. Standard spatial allocation leaves vast amounts of CUDA cores idle.
To solve this, we implement fractional GPU provisioning using two distinct technologies depending on the tenant's operational profile:
1. CUDA Multi-Process Service (MPS) for High Throughput & Shared VRAM
For workloads sharing the same underlying hardware where hardware isolation isn't legally mandatory, CUDA MPS allows multiple containerized vLLM instances to multiplex over the same GPU engine simultaneously.
Unlike basic time-slicing (which introduces heavy context-switching overhead), MPS enables true spatial multiplexing.
# Snippet: Kubernetes DaemonSet enabling MPS on GPU Worker Nodes
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nvidia-mps-control-daemon
namespace: kube-system
spec:
selector:
matchLabels:
name: nvidia-mps-control-daemon
template:
metadata:
labels:
name: nvidia-mps-control-daemon
spec:
containers:
- name: nvidia-mps-control-daemon
image: nvidia/cuda:12.2.0-base-ubuntu22.04
command: ["/bin/sh", "-c"]
args:
- nvidia-cuda-mps-control -d && pipeName=$(mktemp -u) && mkfifo $pipeName && read < $pipeName
securityContext:
privileged: true
volumeMounts:
- mountPath: /tmp/nvidia-mps
name: mps-shm
volumes:
- name: mps-shm
hostPath:
path: /tmp/nvidia-mps
2. Multi-Instance GPU (MIG) for Hard Isolation
For high-SLA enterprise workloads requiring strict compute and memory boundary enforcement, we leverage hardware-level MIG on NVIDIA A100/H100 GPUs.
By partitioning an A100-80GB into seven 1g.10gb or two 3g.40gb profiles, we guarantee deterministic memory access and latency SLAs without cross-pod interference.
| Feature | Dynamic Time-Slicing | CUDA MPS | NVIDIA MIG | | :--- | :--- | :--- | :--- | | Isolation Level | Soft (Process) | Soft (IPC/Compute) | Hard (Hardware Partition) | | VRAM Limits Enforced| No (Risk of OOM) | Yes (via Pipe limits) | Yes (Strict Hardware Limits)| | Context Switch Overhead| High | Low | Zero | | Primary Use Case | Dev/Staging | High-Throughput Micro-Models | Multi-Tenant Enterprise Production |
Pillar 2: Custom vLLM Orchestration with KEDA
vLLM utilizes PagedAttention to manage key-value (KV) cache memory efficiently. Traditional metrics like CPU utilization tell you nothing about when vLLM is running out of context memory or queueing incoming token generation requests.
Scaling on vLLM Native Metrics
vLLM exposes a /metrics Prometheus endpoint containing critical engine state data:
vllm:num_requests_waiting: Requests sitting in queue waiting for GPU allocation.vllm:gpu_cache_usage_perc: Percentage of allocated GPU KV-cache blocks currently utilized.
If vllm:gpu_cache_usage_perc exceeds 0.85, performance degrades due to request preemption and CPU offloading.
Implementing KEDA Custom Metrics Scaler
We deploy a KEDA ScaledObject targeted directly at the vLLM internal metrics, allowing scaling actions to trigger before latency spikes occur.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-llama3-autoscaler
namespace: ai-inference
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: vllm-llama3-70b
minReplicaCount: 1
maxReplicaCount: 16
cooldownPeriod: 300
pollingInterval: 15
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
metricName: vllm_num_requests_waiting
query: sum(vllm:num_requests_waiting{deployment="vllm-llama3-70b"})
threshold: '5'
- type: prometheus
metadata:
serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
metricName: vllm_gpu_cache_usage_perc
query: max(vllm:gpu_cache_usage_perc{deployment="vllm-llama3-70b"})
threshold: '0.80'
Pillar 3: Eradicating Cold Starts in Serverless GPU Pipelines
Autoscaling from 1 to 16 replicas is useless if downloading an 80GB model binary over HTTP takes 8 minutes per new instance. To achieve zero-perceivable cold starts, we implement a multi-layer streaming model delivery system.
+---------------------------------------------------------------------------------+
| COLD START OPTIMIZATION LAYERS |
| |
| 1. Container Engine --> Base images pre-baked with CUDA/PyTorch dependencies |
| 2. Storage Tier --> Local NVMe array backed by Mountpoint for Amazon S3 |
| 3. Execution Engine --> vLLM loading via Safetensors & Direct mmap |
+---------------------------------------------------------------------------------+
1. High-Throughput I/O with Local NVMe Cache and S3 Mountpoint
Instead of mounting slow network block storage (EBS/Persistent Disk), we leverage ephemeral local NVMe RAID-0 arrays on host nodes backed by Mountpoint for Amazon S3 or JuiceFS.
Model weights are chunked and streamed directly from object storage to local host RAM/NVMe using parallelized HTTP GET requests, bypassing standard single-threaded bottlenecks.
# Custom Model Loader Script executed in vLLM initContainer
import os
import time
from concurrent.futures import ThreadPoolExecutor
from boto3 import client
s3_client = client('s3')
BUCKET = "ecstaticloud-model-registry"
MODEL_PREFIX = "llama-3-70b-instruct/safetensors/"
LOCAL_DIR = "/mnt/fast-nvme/llama-3-70b-instruct/"
def download_file(key):
rel_path = os.path.relpath(key, MODEL_PREFIX)
target_path = os.path.join(LOCAL_DIR, rel_path)
os.makedirs(os.path.dirname(target_path), exist_ok=True)
if not os.path.exists(target_path):
print(f"Streaming {key} -> {target_path}")
s3_client.download_file(BUCKET, key, target_path)
def sync_weights():
start_time = time.time()
paginator = s3_client.get_paginator('list_objects_v2')
keys = []
for page in paginator.paginate(Bucket=BUCKET, Prefix=MODEL_PREFIX):
for obj in page.get('Contents', []):
keys.append(obj['Key'])
with ThreadPoolExecutor(max_workers=32) as executor:
executor.map(download_file, keys)
print(f"Model sync completed in {time.time() - start_time:.2f} seconds.")
if __name__ == "__main__":
sync_weights()
2. Fast Weight Loading via Safetensors and Zero-Copy mmap
PyTorch .bin (pickle) checkpoints require executing Python code to deserialize tensors, forcing full allocation in system RAM before moving to GPU VRAM.
We convert all weights exclusively to Hugging Face safetensors. This format allows memory-mapping (mmap) of files directly from the host NVMe cache to GPU VRAM using cudaMemcpyAsync, bypassing host CPU processing entirely.
# vLLM launch parameters optimized for hyper-fast startup and memory efficiency
python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/fast-nvme/llama-3-70b-instruct/ \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.90 \
--max-model-len 8192 \
--enable-chunked-prefill \
--max-num-batched-tokens 2048 \
--load-format safetensors \
--enforce-eager
Note: Using --enforce-eager disables CUDA graph compilation warm-up steps during development/scaling phases, reducing initial vLLM pod boot time from 45 seconds to under 4 seconds.
Real-World Impact: Cost & Latency Benchmarks
We benchmarked this optimized serverless architecture against a standard static-provisioned cluster handling an enterprise application with dynamic, spiked usage patterns (15 million tokens processed per hour peak, dropping to near-zero overnight).
Cost Comparison (Monthly Production Run Rate)
| Setup | Infrastructure Description | Cost/Month | Savings | | :--- | :--- | :--- | :--- | | Legacy Baseline | 12x Dedicated A100 Nodes (Static Provisioning 24/7) | $26,400 | Baseline | | Standard K8s HPA | CPU-Based Autoscaling + standard EBS persistent volumes | $18,200 | ~31% | | Ecstaticloud Architecture | KEDA + vLLM Metrics + MPS Slicing + NVMe Model Streaming | $10,400 | 60.6% |
Latency Performance Under Scale Spikes
Request Latency (P99) during 10x Sudden Scale Event:
Legacy (Static): [====================] 140ms
Naive HPA (Cold): [===================================================>] 84,000ms (Failed/Timed out)
Ecstaticloud Stack: [======================] 165ms (Gracefully buffered via KEDA queue)
By switching scaling triggers to deep engine metrics (num_requests_waiting), incoming bursts are buffered at the Envoy ingress layer while fast-provisioning nodes spin up within 8-12 seconds max.
Engineering Checklist for Implementation
If you are migrating your generative AI infrastructure to cost-optimized serverless pools, ensure you execute the following checklist:
- [ ] Instrument Native Engine Telemetry: Stop scaling on CPU/RAM. Expose vLLM/TGI Prometheus metrics directly to your K8s custom metrics adapter.
- [ ] Format All Model Weights to Safetensors: Eliminate pickle processing overhead to enable zero-copy
mmapdirect memory loading. - [ ] Deploy Host-Level NVMe Caching: Pre-pull base container layers via Karpenter / Cluster Autoscaler user-data scripts and stream weight chunks directly from Object Storage.
- [ ] Evaluate CUDA MPS for Small Models: Consolidate small micro-services or embedding models using MPS to maximize compute density per GPU node.
- [ ] Tune Aggressive Downscaling Cooldowns: Keep a 300-second stabilization window on scale-down events to avoid thrashing (rapidly spinning nodes up and down).
Summary
Serving Large Language Models at scale does not require blank-check infrastructure spending. By understanding how models access hardware memory and coupling native engine metrics directly to Kubernetes event-driven scaling mechanisms, enterprise platforms can run production generative AI workloads reliably—slashing infrastructure overhead by 60% while maintaining rock-solid SLAs.