Architecting Cost-Efficient LLM Inference Pipelines on Kubernetes Using vLLM and Spot Instances | Ecstaticloud Blog

Serving Large Language Models (LLMs) in production presents a massive infrastructure challenge: high throughput and ultra-low latency requirements coupled with the astronomical costs of modern GPU compute. Deploying dedicated On-Demand GPU instances (such as NVIDIA A10G or H100 clusters) running standard inference servers often leads to low GPU utilization during off-peak hours and exorbitant monthly cloud bills.

To achieve up to 70% cost reduction without sacrificing service-level agreements (SLAs), platform teams must rethink their LLM serving strategy.

In this architectural guide, we will design and deploy a production-grade, highly available, and cost-efficient LLM inference pipeline on Kubernetes. We will combine vLLM (an open-source LLM serving engine optimized for high memory efficiency), Karpenter for heterogeneous GPU dynamic provisioning, KEDA for metrics-based auto-scaling, and a resilient Spot-to-On-Demand fallback architecture.

1. The Bottlenecks of LLM Inference & The vLLM Architecture

Traditional Transformer-based inference implementations suffer from severe memory bottlenecks. The primary issue stems from the dynamic memory footprint of the Key-Value (KV) Cache, which stores attention keys and values to avoid recomputing them for previous tokens during generation.

Standard frameworks allocate contiguous memory for the maximum sequence length per request. This creates two distinct problems:

Internal Fragmentation: Unused reserved slots when requests end early.
External Fragmentation: Virtual memory fragmentation preventing batching of incoming requests despite total free physical VRAM.

How vLLM Changes the Math

vLLM addresses these bottlenecks through two key innovations:

PagedAttention: Inspired by virtual memory and paging in operating systems, PagedAttention allows KV caches to be stored in non-contiguous physical block locations. VRAM is divided into physical blocks, mapping logical token blocks dynamically. This reduces memory waste to near zero (<1%), allowing you to double or triple your batch sizes on the same hardware.
Continuous Batching (Iteration-level Scheduling): Instead of waiting for an entire batch to finish generating (request-level batching), vLLM injects new requests into the batch as soon as an existing request outputs an <EOS> token.

+-----------------------------------------------------------------------+
|                             vLLM Engine                               |
|                                                                       |
|  +------------------------+        +-------------------------------+  |
|  | Iteration-Level        |        | PagedAttention Engine         |  |
|  | Continuous Batcher     |        | (Block Space Manager)         |  |
|  +-----------+------------+        +---------------+---------------+  |
|              |                                     |                  |
|              v                                     v                  |
|  +-----------------------------------------------------------------+  |
|  |                     NVIDIA GPU Virtual VRAM                     |  |
|  |  [ Block 0 ] [ Block 1 ] [ Block 2 ] ... [ Block N ]              |  |
|  +-----------------------------------------------------------------+  |
+-----------------------------------------------------------------------+

By significantly increasing request throughput per GPU, vLLM allows us to satisfy target SLAs using smaller or fewer GPU instances.

2. Infrastructure Architecture Blueprint

To run on Spot instances safely, we must assume any GPU node can be reclaimed with a 2-minute warning. The architecture must separate stateless inference workloads from underlying node lifecycles while providing instantaneous fallback mechanisms.

                           +------------------------+
                           |  API Gateway / Ingress |
                           +-----------+------------+
                                       |
                                       v
                           +------------------------+
                           |   K8s Service / Mesh   |
                           +-----------+------------+
                                       |
                   +-------------------+-------------------+
                   |                                       |
                   v                                       v
     +---------------------------+           +---------------------------+
     |   Spot GPU NodePool       |           |  On-Demand GPU NodePool   |
     |   (Primary Workloads)     |           |  (Fallback / Safety Net)  |
     |                           |           |                           |
     |  +---------------------+  |           |  +---------------------+  |
     |  | Pod: vLLM Instance  |  |           |  | Pod: vLLM Instance  |  |
     |  +---------------------+  |           |  +---------------------+  |
     +---------------------------+           +---------------------------+
                   ^                                       ^
                   |                                       |
                   +-------------------+-------------------+
                                       |
                           +-----------+------------+
                           |    KEDA / Prometheus   |
                           |  Custom Metric Scaler  |
                           +------------------------+

Key Components:

Dynamic Spot Provisioning: Managed by Karpenter, targeting flexible instance families (e.g., AWS g5.xlarge, g5.2xlarge, g5.12xlarge, g6.xlarge).
On-Demand Fallback: Karpenter NodePool prioritized with negative weights or taints, scaling up only when Spot capacity for all instance types across Availability Zones is completely exhausted.
Queue-Aware Autoscaling: KEDA driven by real-time engine telemetry (vllm:num_requests_waiting) rather than CPU/GPU duty cycle metrics.
Fast Storage Caching: Local NVMe or shared ReadOnlyMany persistent cache for pre-fetched Hugging Face model weights to eliminate startup overhead.

3. Provisioning Heterogeneous GPU Spot Pools with Karpenter

AWS Cluster Autoscaler can be sluggish when dealing with complex GPU node constraints across multi-AZ setups. Karpenter handles multi-instance, multi-AZ Spot allocations natively with dynamic node creation in seconds.

Below is the Karpenter manifest establishing our Spot and On-Demand NodePools:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: gpu-spot-nodepool
spec:
  template:
    metadata:
      labels:
        workload: vllm-inference
        capacity-type: spot
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: instance-family
          operator: In
          values: ["g5", "g6"] # Multi-family fallback: A10G, L4 GPUs
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      nodeClassRef:
        name: gpu-nodeclass
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s
    expireAfter: 7d
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: gpu-nodeclass
spec:
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster-name"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "my-cluster-name"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 200Gi # Large root disk for caching local models
        volumeType: gp3
        iops: 3000
        throughput: 125

4. Deploying vLLM with Model Pre-Caching & Resilience

When deploying vLLM on Spot instances, model cold-start times are your biggest operational enemy. Downloading a 14GB–140GB model weight dataset from Hugging Face every time a Spot node rotates introduces severe latency spikes.

To counter this, we use an initContainer backing onto a shared high-throughput cache or localized host paths, along with carefully configured Kubernetes liveness/readiness probes targeting vLLM’s endpoints.

Here is the complete production Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-mistral-7b
  namespace: llm-serving
  labels:
    app: vllm-mistral-7b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-mistral-7b
  template:
    metadata:
      labels:
        app: vllm-mistral-7b
    spec:
      priorityClassName: high-priority-preemption # Ensure core pods stay running
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: capacity-type
                    operator: In
                    values: ["spot"]
            - weight: 10
              preference:
                matchExpressions:
                  - key: capacity-type
                    operator: In
                    values: ["on-demand"]
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values: ["vllm-mistral-7b"]
                topologyKey: "kubernetes.io/hostname"
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.6.3
          args:
            - "--model"
            - "mistralai/Mistral-7B-Instruct-v0.2"
            - "--port"
            - "8000"
            - "--max-model-len"
            - "8192"
            - "--gpu-memory-utilization"
            - "0.90"
            - "--tensor-parallel-size"
            - "1"
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token-secret
                  key: token
            - name: HF_HOME
              value: "/root/.cache/huggingface"
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: 32Gi
              cpu: "8"
            requests:
              nvidia.com/gpu: "1"
              memory: 16Gi
              cpu: "4"
          ports:
            - containerPort: 8000
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 5
          volumeMounts:
            - mountPath: /root/.cache/huggingface
              name: model-cache
            - mountPath: /dev/shm
              name: dshm
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: efs-model-cache-pvc # Fast shared storage or AWS EFS
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi

5. Queue-Aware Metrics Auto-Scaling with KEDA

Standard Kubernetes Horizontal Pod Autoscaler (HPA) using CPU or GPU Utilization metrics is completely ineffective for LLM workloads.

Why standard metrics fail: A vLLM container running GPU inference will report near 100% GPU utilization whether it is processing 1 token/sec or has 500 requests backed up in its internal queue.
The Solution: Scale based on the real-time queue length using vLLM's exposed Prometheus metrics (vllm:num_requests_waiting).

Prometheus Setup for vLLM Metrics

vLLM exposes metrics on /metrics by default. We configure Prometheus to scrape these metrics, focusing on two parameters:

vllm:num_requests_waiting: The number of requests sitting idle in the processing queue.
vllm:gpu_cache_usage_factor: The percentage of GPU memory used for KV caching.

Implementing KEDA ScaledObject

We install KEDA (Kubernetes Event-driven Autoscaling) and define a ScaledObject that scales out our vLLM pods as soon as requests start queuing up.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-mistral-7b
  minReplicaCount: 2
  maxReplicaCount: 10
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 50
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 600 # Slow scale down to prevent thrashing
          policies:
            - type: Percent
              value: 10
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        threshold: '3' # Scale up if waiting requests > 3 across the pool
        query: |
          sum(vllm:num_requests_waiting{namespace="llm-serving", app="vllm-mistral-7b"})

6. Zero-Downtime Spot Interruption Handling

AWS provides a 2-minute notification via Amazon EventBridge and Metadata Endpoints prior to reclaiming a Spot instance. If an inference request streaming back tokens is abruptly killed, the user experiences a truncated output or a broken TCP connection.

To achieve zero downtime, we implement a multi-layered graceful termination strategy.

 AWS EventBridge               Node Termination             vLLM Pod Lifecycle
    Notice                        Handler (NTH)                 Hook/Drain
       |                                |                            |
       |--- Spot Interruption (2m) ---->|                            |
       |                                |--- Mark Node Cordoned ---->|
       |                                |--- Pod SIGTERM Sent ------>|
       |                                                             |--- 1. Set Readiness: Failed
       |                                                             |--- 2. Stop Ingress Routing
       |                                                             |--- 3. Drain Existing Batches
       |                                                             |--- 4. Graceful Shutdown (Exit 0)

Step 1: Install AWS Node Termination Handler (NTH)

Deploy NTH in Queue Processor mode to catch EC2 Spot Instance Interruption Warnings. NTH automatically cordons and drains the affected node when a termination notice is caught.

Step 2: Graceful Termination via Pod Lifecycle PreStop Hooks

When a node drains, Kubernetes sends a SIGTERM signal to the pod. We insert a preStop hook to alter our app's readiness status immediately. This allows the Kubernetes Service endpoint to stop routing new traffic to the pod while allowing current continuous-batch generation cycles to complete cleanly before SIGKILL.

spec:
  containers:
    - name: vllm-container
      lifecycle:
        preStop:
          exec:
            command:
              - "/bin/sh"
              - "-c"
              - |
                # 1. Force readiness probe failure by creating a local block file or calling an internal endpoint
                touch /tmp/pod-terminating
                # 2. Sleep for 15s to allow endpoints controller/ingress to update routing tables
                sleep 15
                # 3. Allow current inference stream queries to complete cleanly (vLLM handles ongoing streams naturally)

Make sure your readiness probe checks for this block file:

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - "! test -f /tmp/pod-terminating && curl -f http://localhost:8000/health"

7. Cost & Performance Benchmark Analysis

To evaluate the operational impact of this architecture, we ran a workload simulation generating 500,000 prompt/completion tokens per hour over a 30-day billing cycle.

Hardware Configurations Tested:

Baseline: 4x g5.12xlarge (On-Demand) – 4x NVIDIA A10G GPUs per instance.
Optimized Strategy: Heterogeneous Spot Pool (g5.2xlarge, g5.4xlarge, g6.2xlarge) managed by Karpenter, scaled via KEDA, with an On-Demand minimum fallback layer.

Cost & Throughput Summary

TTFT = Time To First Token

Architectural Takeaways for Platform Engineers

Decouple Dynamic Compute from Processing Latency: Never rely on raw CPU/GPU memory percentages for autoscaling inference metrics. Scaling on engine queue depth (vllm:num_requests_waiting) prevents latency spikes before requests pile up.
Mitigate Cold Starts Early: Ensure model weights live on local NVMe caches or ultra-fast shared storage (like high-throughput EFS or AWS FSx for LUSTRE). Pre-warming nodes ensures Spot replacements are live within seconds, not minutes.
Embrace Multi-Family Provisioning: Do not lock your Karpenter specs to a single instance type. Broadening constraints to encompass multiple instance types across different families (g5, g6, p4de) across all Availability Zones maximizes your chances of securing Spot capacity.

By marrying vLLM's memory-efficient inference engine with dynamic Kubernetes Spot automation, infrastructure teams can scale generative AI applications sustainably without breaking the bank.