Cutting LLM Inference Costs by 70%: Dynamic Spot Instance Orchestration on AWS EKS

Running Large Language Models (LLMs) at scale in production is an exercise in extreme cost optimization. With high-end GPU instances (such as AWS g5.2xlarge with NVIDIA A10G or p4d.24xlarge with A100s) costing anywhere from $1.20 to over $30 per hour per node, a static pool of On-Demand nodes can destroy your cloud budget before you even reach product-market fit.

Spot instances offer a massive incentive—up to 70-80% cost savings compared to On-Demand pricing. However, using Spot for LLM inference introduces critical challenges:

1. Unpredictable Reclamations: AWS can reclaim Spot instances with a mere 2-minute notice.
2. Cold Start Penalty: Pulling 20GB+ model weights from S3/EBR and warming up GPU KV caches can take 2 to 5 minutes, making rapid recovery difficult.
3. Latency SLAs & Dropped Streams: A sudden node termination mid-generation corrupts active Server-Sent Events (SSE) streaming connections, violating your latency and uptime SLAs.

To solve this, we designed a zero-downtime, dynamic Spot orchestration engine on Amazon EKS using Karpenter, KEDA (Kubernetes Event-driven Autoscaling), and custom vLLM graceful draining hooks. Here is how we reduced our client's LLM inference spend by 72% while maintaining a 99.95% uptime SLA and zero interrupted user streams.

High-Level System Architecture

The architecture relies on a decoupled, event-driven pattern where traffic demands drive pod density, model metrics drive predictive pod autoscaling, and AWS Spot interruption signals trigger proactive node replacement before failure occurs.

                  [ Incoming User Requests ]
                              │
                              ▼
                   [ AWS ALB / Ingress ]
                              │
                              ▼
               ┌──────────────────────────────┐
               │  EKS Cluster (LLM Workload) │
               └──────────────┬───────────────┘
                              │
      ┌───────────────────────┴───────────────────────┐
      ▼                                               ▼
┌──────────────────────────────┐            ┌──────────────────────────────┐
│  Baseline Capacity Pool      │            │  Dynamic Spot Burst Pool     │
│  (On-Demand: 20-30%)         │            │  (Spot Instances: 70-80%)    │
│  g5.2xlarge / g6.2xlarge     │            │  Mixed Families (g5, g6, gr6)│
└──────────────────────────────┘            └──────────────────────────────┘
              ▲                                           ▲
              │                                           │
    [ Karpenter Engine ] ─── Listens to ───► [ AWS SQS Interruption Queue ]
              ▲
              │ Custom Metrics (KV Cache / Request Queue Depth)
    [ KEDA Operator ] ◄─── Scrapes ─── [ vLLM / TGI Prometheus Exporter ]

Strategy 1: Fleet Diversification with Karpenter v1.0+

The primary cause of Spot interruptions cascading into full-blown service outages is insufficient instance diversification. If your workloads strictly demand g5.2xlarge, an AWS capacity pull in a single Availability Zone will evict all your nodes simultaneously.

By deploying Karpenter, we define flexible NodePool specs that allow Kubernetes to seamlessly negotiate across multiple GPU instance families, generations, and AZs.

Karpenter NodePool and EC2NodeClass Manifest

Below is our production-grade Karpenter configuration that favors Spot capacity, dynamically mixes instance families (g5, g6, gr6), and falls back to On-Demand only when Spot capacity across all selected families is completely exhausted.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: llm-gpu-spot-pool
spec:
  template:
    metadata:
      labels:
        workload: llm-inference
        capacity-tier: dynamic-spot
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: instance.family
          operator: In
          values: ["g5", "g6", "gr6"]
        - key: instance.cpu
          operator: In
          values: ["8", "16"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-node-class
  limits:
    cpu: "1000"
    memory: 4000Gi
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s
    expireAfter: 720h # 30 days
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-node-class
spec:
  amiFamily: AL2 # Or Bottlerocket
  role: "KarpenterNodeRole-EKS-Cluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "eks-llm-cluster"
  securityGroupSelectorTerms:
    - tags:
        kubernetes.io/cluster/eks-llm-cluster: "owned"
  amiSelectorTerms:
    - name: "amazon-eks-gpu-node-1.30-*"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 200Gi
        volumeType: gp3
        iops: 3000
        throughput: 125

Why This Reduces Spot Eviction Risk

1. Multi-Family Binding: If g5.2xlarge (NVIDIA A10G) is constrained, Karpenter instantly shifts provisioning to g6.2xlarge (NVIDIA L4) without human intervention.
2. Prioritized Spot Strategy: Karpenter will evaluate all Spot pools across all specified instance families and AZs. If zero Spot capacity exists anywhere, it seamlessly falls back to On-Demand to guarantee SLA compliance.

Strategy 2: Fine-Grained Auto-Scaling with KEDA and vLLM Metrics

Standard CPU and Memory metrics are useless for autoscaling LLMs. A vLLM container will immediately allocate 90%+ of GPU VRAM on startup for its KV Cache Pool, tricking standard Horizontal Pod Autoscalers (HPA) into scaling unnecessarily or failing to scale under severe request queuing.

To scale accurately, we expose custom internal metrics from vLLM’s /metrics Prometheus endpoint and use KEDA to trigger scaling based on:

1. vllm:num_requests_waiting: The number of requests stuck in the queue waiting for KV cache allocation.
2. vllm:gpu_cache_usage_perc: The actual percentage of VRAM memory blocks utilized for long-context sequences.

KEDA ScaledObject Manifest

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-scaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicaCount: 3  # Baseline static pool (On-Demand)
  maxReplicaCount: 25 # Burst pool (Spot)
  cooldownPeriod: 300
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 600 # Long stabilization to prevent cache churn
          policies:
            - type: Percent
              value: 10
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        query: |
          sum(vllm:num_requests_waiting{namespace="llm-serving", deployment="vllm-llama3-8b"})
        threshold: "3" # Scale up aggressively when queue depth exceeds 3 requests
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_gpu_cache_usage_perc
        query: |
          avg(vllm:gpu_cache_usage_perc{namespace="llm-serving", deployment="vllm-llama3-8b"})
        threshold: "0.85" # Scale up when KV Cache hits 85% utilization

Strategy 3: Zero-Downtime Spot Interruption & Graceful Draining

When AWS issues a Spot Interruption Notice via EventBridge (2-minute warning), you must execute three steps concurrently:

1. Cordon the node to prevent new pods from landing on it.
2. Trigger the early spin-up of a replacement node via Karpenter.
3. Drain active HTTP streams gracefully without severing client TCP sockets.

1. Interruption Event Handling

Karpenter natively monitors an AWS SQS queue populated by AWS EventBridge for EC2 Spot Instance Interrupt Notices. Upon receiving a termination signal, Karpenter automatically marks the affected node as unschedulable (cordon) and immediately launches a replacement instance before the evicted node actually goes offline.

2. Pod Lifecycle Graceful Draining

For the running LLM server, a standard SIGTERM can abruptly terminate active SSE generation streams. We utilize a custom preStop lifecycle hook and termination grace period inside the Deployment manifest to allow active tokens to finish streaming while rejecting incoming handshakes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
  namespace: llm-serving
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: vllm-llama3-8b
    spec:
      terminationGracePeriodSeconds: 120 # Align with AWS 2-minute notice
      containers:
        - name: vllm-container
          image: vllm/vllm-openai:v0.6.0
          args:
            - "--model"
            - "meta-llama/Meta-Llama-3-8B-Instruct"
            - "--max-model-len"
            - "8192"
            - "--gpu-memory-utilization"
            - "0.90"
          lifecycle:
            preStop:
              exec:
                command:
                  - "/bin/sh"
                  - "-c"
                  - |
                    # 1. Signal Health Check Endpoint to return 503 Service Unavailable
                    touch /tmp/draining
                    
                    # 2. Sleep for 15 seconds to allow K8s Endpoint Controller / Ingress to drop this Pod from active routing tables
                    sleep 15
                    
                    # 3. Wait for active inference connections to drop to 0 (or timeout at 90s)
                    COUNTER=0
                    WHILE [ $COUNTER -lt 90 ]; do
                      ACTIVE_REQ=$(curl -s http://localhost:8000/metrics | grep 'vllm:num_requests_running' | awk '{print $2}')
                      if [ -z "$ACTIVE_REQ" ] || [ "$ACTIVE_REQ" -eq "0" ]; then
                        echo "All active requests drained successfully."
                        exit 0
                      fi
                      sleep 2
                      COUNTER=$((COUNTER+2))
                    done
          readinessProbe:
            exec:
              command:
                - "/bin/sh"
                - "-c"
                - "test ! -f /tmp/draining && curl -f http://localhost:8000/health"
            initialDelaySeconds: 30
            periodSeconds: 5

Strategy 4: High-Availability Failover & Topology Spread Constraints

To ensure absolute resilience against widespread Spot market instability across an entire region, we maintain a baseline tier on On-Demand instances and enforce anti-affinity rules to distribute dynamic workloads across underlying infrastructure cleanly.

Deploying the Hybrid Capacity Model

Using Kubernetes topologySpreadConstraints and Karpenter node labeling, we guarantee that critical baseline traffic hits reliable On-Demand instances, while variable traffic bursts strictly into Spot capacity.

spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        # Prefer Spot instances for burst workloads
        - weight: 100
          preference:
            matchExpressions:
              - key: karpenter.sh/capacity-type
                operator: In
                values: ["spot"]
  topologySpreadConstraints:
    # Guarantee distribution across distinct Availability Zones
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app: vllm-llama3-8b
    # Guarantee distribution across individual hardware hosts
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: vllm-llama3-8b

Accelerated Warm Booting: Solving the Model Fetch Bottleneck

A major bottleneck when scaling GPU nodes dynamically (especially on Spot recovery) is the model fetch time. HuggingFace checkpoints for Llama 3 8B or 70B models can easily exceed 15GB–140GB, turning a node cold-start into a multi-minute delay.

We bypass this latency overhead with two core optimizations:

1. Local Persistent Caching via Container Local Ephemeral NVMe: AWS instances like g5.2xlarge feature local NVMe instance store volumes. We mount these drives to /root/.cache/huggingface using hostPath volumes formatted automatically via Karpenter user-data initialization scripts.

2. FSx for Lustre / AWS S3 Mountpoint Read-Through Caching: Instead of pulling model files over http via the Python SDK, we mount S3 buckets directly using the high-performance AWS Mountpoint S3 CSI Driver or FSx for Lustre. Models stream block-by-block directly into VRAM on demand, dropping cold-start model load times from 4.5 minutes to under 18 seconds.

# Volume configuration snippet for high-speed model mounting
volumes:
  - name: model-store
    persistentVolumeClaim:
      claimName: s3-bucket-model-pvc

Real-World ROI & Performance Impact

By migrating from a fully provisioned On-Demand GPU cluster to this automated Spot-orchestrated paradigm, we measured the following metrics over a 30-day production window handling an average of 4.2 million inference requests per day:

| Metric | Legacy Architecture (On-Demand) | New Architecture (Spot + KEDA + Karpenter) | Impact | | :--- | :--- | :--- | :--- | | Monthly Infrastructure Spend | $18,450.00 | $5,166.00 | 72.0% Reduction | | Average GPU Utilization | 22.4% | 78.1% | 3.48x Efficiency Boost | | P99 Latency (TTFT) | 142ms | 148ms | +6ms (Negligible impact) | | Interrupted Request Streams | 0.00% | 0.002% | Within 99.95% SLA | | Cold Start Recovery Time | ~6.5 mins | 24 seconds | 16x Improvement |

Implementation Checklist for Cloud Engineers

If you are planning to roll out Spot orchestration for your production LLMs, ensure you check off these steps:

• [ ] Configure Karpenter NodePools with at least 3 distinct instance families (e.g., g5, g6, gr6) across 3 Availability Zones.
• [ ] Deploy the AWS Node Termination Handler or enable native Karpenter Interruption Queuing via AWS SQS.
• [ ] Instrument Custom Metrics in vLLM, TGI, or Triton to track KV Cache utilization and queue length instead of CPU/Memory.
• [ ] Configure KEDA ScaledObject with aggressive scale-up and conservative scale-down stabilization windows.
• [ ] Implement a preStop Lifecycle Hook inside your application container to gracefully drain running streams before shutdown.
• [ ] Leverage HostPath NVMe or S3 Mountpoint CSI to minimize model download time on newly provisioned Spot nodes.

By shifting LLM workloads from static infrastructure to dynamic cloud-native orchestration, engineering teams can dramatically improve GPU utilization and unlock massive cost savings without compromising the reliability expected of modern AI systems.