Slashing LLM Inference Costs: Auto-Scaling vLLM Workloads on Kubernetes with Karpenter

Running Large Language Models (LLMs) in production is notoriously expensive. A standard cluster running Meta-Llama-3.1-70B-Instruct across multiple AWS g5.12xlarge or p4d.24xlarge instances can burn through tens of thousands of dollars per month—even during low-traffic off-peak hours.

Traditional Kubernetes autoscaling approaches fail here. Standard Horizontal Pod Autoscalers (HPA) relying on CPU/Memory metrics don't capture LLM queue depths, and Kubernetes Cluster Autoscaler suffers from rigid Node Group abstractions that cannot efficiently handle heterogeneous GPU allocations or rapid scale-to-zero requirements.

To solve this, we combine two best-in-class technologies:

1. vLLM: A high-throughput, low-latency LLM serving engine leveraging PagedAttention for optimal GPU memory management.
2. Karpenter: An open-source, group-less node auto-scaler designed for Kubernetes that provisions targeted AWS EC2 instances in seconds based on pending pod constraints.

By coupling metric-driven pod scaling (via KEDA) with Karpenter’s just-in-time compute provisioning, you can achieve up to 60% reduction in GPU infrastructure costs while maintaining your SLA for request latencies.

Architectural Overview

The core philosophy of this architecture is Demand-Driven Compute Provisioning. Instead of maintaining pre-provisioned GPU pools, compute resources are allocated strictly on demand and downscaled the moment queue traffic drops.

                  +-------------------+
                  |  Incoming Requests|
                  +---------+---------+
                            |
                            v
                 +---------------------+
                 | Ingress Controller  |
                 +----------+----------+
                            |
                            v
                 +---------------------+
                 |  vLLM Deployment    | <---+
                 |   (Pod Scale)       |     |
                 +----------+----------+     | KEDA Metrics Query
                            |                | (num_requests_waiting)
                            v                |
                 +---------------------+     |
                 |  Prometheus Metrics |-----+
                 +---------------------+
                            |
           (Pending Pods trigger Karpenter)
                            v
                 +---------------------+
                 |  Karpenter Controller|
                 +----------+----------+
                            |
             Provisions Just-in-Time EC2
                            v
     +-----------------------------------------------+
     | AWS EC2 Fleet (g5.xlarge / g5.12xlarge / Spot)|
     +-----------------------------------------------+

Key Workflow Components

1. Metric Exposing: vLLM exposes native Prometheus metrics detailing queue status, such as vllm:num_requests_waiting.
2. Pod Autoscaling: KEDA (Kubernetes Event-driven Autoscaling) monitors this metric and scales vLLM Pod deployments up or down.
3. Node Provisioning: Unscheduled "Pending" vLLM pods are intercepted by Karpenter, which analyzes the pod’s node selectors, resource limits (nvidia.com/gpu), and tolerations to launch the exact EC2 instance required.
4. Consolidation: When traffic recedes, KEDA scales down pod replicas, and Karpenter aggressively consolidates or terminates empty GPU nodes.

Step 1: Deploying vLLM Engine on Kubernetes

vLLM drastically outperforms standard HuggingFace Transformers by reducing memory waste through PagedAttention. When deploying vLLM, you must properly allocate Shared Memory (/dev/shm) to prevent PyTorch distributed communication crashes and set accurate memory utilization ratios.

Here is a production manifest for serving Mistral-7B-Instruct-v0.2 on GPU instances:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-mistral-7b
  namespace: llm-serving
  labels:
    app: vllm-mistral-7b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-mistral-7b
  template:
    metadata:
      labels:
        app: vllm-mistral-7b
    spec:
      # Allow scheduling on Karpenter-managed GPU nodes
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      - key: "ecstaticloud.com/llm-node"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: vllm-container
        image: vllm/vllm-openai:v0.6.2
        args:
        - "--model"
        - "mistralai/Mistral-7B-Instruct-v0.2"
        - "--port"
        - "8000"
        - "--max-model-len"
        - "4096"
        - "--gpu-memory-utilization"
        - "0.90"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        ports:
        - containerPort: 8000
          name: http
        resources:
          requests:
            cpu: "6"
            memory: "24Gi"
            nvidia.com/gpu: "1"
          limits:
            cpu: "8"
            memory: "32Gi"
            nvidia.com/gpu: "1"
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /root/.cache/huggingface
          name: model-cache
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 15
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 8Gi
      - name: model-cache
        persistentVolumeClaim:
          claimName: efs-model-cache-pvc

Step 2: Metric-Driven Pod Scaling with KEDA

Standard Kubernetes CPU and Memory metrics are ineffective for LLM autoscaling. An LLM server fully saturates GPU VRAM at boot time to store model weights and allocate KV cache space. Thus, CPU and Memory usage remain high even when the model is processing zero requests.

Instead, we scale based on vllm:num_requests_waiting using KEDA.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-keda-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-mistral-7b
  minReplicaCount: 0 # Scales to zero during prolonged idle periods
  maxReplicaCount: 10
  cooldownPeriod: 300 # Wait 5 minutes before scaling down to prevent thrashing
  pollingInterval: 15
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 10
            periodSeconds: 60
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{namespace="llm-serving", pod=~"vllm-mistral-7b-.*"})
      threshold: '3' # Trigger new pod creation if average queue depth exceeds 3 requests

Step 3: Just-In-Time Node Provisioning with Karpenter

Now that KEDA can scale pods dynamically based on real traffic queues, we need an autoscaler capable of provisioning expensive GPU nodes on AWS in seconds.

Unlike Kubernetes Cluster Autoscaler, Karpenter does not rely on EC2 Auto Scaling Groups (ASGs). It evaluates unscheduled pod specifications directly and dynamically calls the AWS EC2 API to launch the most optimal instance type.

Below are the Karpenter NodePool and EC2NodeClass manifests designed specifically for GPU inference workloads:

1. Karpenter EC2NodeClass Manifest

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-node-class
spec:
  amiSelectorTerms:
    - alias: al2023@latest # Or explicit EKS-optimized GPU AMIs
  role: "KarpenterNodeRole-EKSCluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "eks-production-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "eks-production-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
        encrypted: true
  tags:
    Environment: production
    Workload: llm-inference
    ManagedBy: Karpenter

2. Karpenter NodePool Manifest

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-nodepool
spec:
  template:
    metadata:
      labels:
        ecstaticloud.com/node-type: gpu-inference
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-node-class
      taints:
      - key: "nvidia.com/gpu"
        value: "true"
        effect: "NoSchedule"
      - key: "ecstaticloud.com/llm-node"
        value: "true"
        effect: "NoSchedule"
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"] # Prioritize spot, fallback to on-demand
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values:
            - "g5.xlarge"   # 1x A10G (24GB VRAM)
            - "g5.2xlarge"  # 1x A10G (24GB VRAM)
            - "g6.xlarge"   # 1x L4 (24GB VRAM)
            - "g6.2xlarge"  # 1x L4 (24GB VRAM)
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 2m
    expireAfter: 7d # Prevent node drift
  limits:
    cpu: "200"
    memory: "800Gi"
    nvidia.com/gpu: "16"

Mitigating Cold Starts and Optimization Strategies

When scaling GPU nodes from zero, instance provisioning and image pulling introduces cold start latencies (often 3 to 5 minutes). To deploy this setup in production without breaking client SLAs, implement these critical optimization techniques:

1. Warm Model Caching via Shared Filesystems or High-Speed Persistent Volumes

Downloading model weights from HuggingFace on every node launch causes significant overhead.

• EFS/Amazon S3 CSI Driver Mounts: Mount shared storage containing pre-downloaded weights directly into the container path /root/.cache/huggingface.
• Container Image Pre-warming: Build custom container images containing model weights directly inside the layer if model sizes are under 10GB.

2. Rapid Node Bootstrapping via DaemonSets

Ensure the NVIDIA Container Toolkit and K8s Device Plugin load seamlessly. Using AWS EKS GPU-optimized AMIs completely avoids runtime driver compilation delays during node boot.

3. Graceful Termination & Dynamic Traffic Routing

When Karpenter triggers node consolidation during decreased demand, standard pods risk terminating active client HTTP streaming sessions (text/event-stream).

To prevent interrupted text generation:

• Define a preStop hook inside the vLLM container to remove the endpoint from K8s Service discovery while allowing existing connections to drain:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15 && kill -SIGTERM 1"]

Cost Comparison: Static vs Auto-Scaled Compute

Let’s evaluate a typical enterprise scenario serving Mistral-7B or Llama-3-8B.

Scenario Parameters

• Base Load: 1 GPU Node required continuous 24/7.
• Peak Load: Up to 8 GPU Nodes required during business hours (8 hours/day).
• Instance Type: AWS g5.2xlarge On-Demand ($1.212 / hour).

| Strategy | Daily Fleet Node Hours | Monthly Cost (Approx.) | | :--- | :--- | :--- | | Static Provisioning (Peak Capacity) | 8 nodes × 24 hours = 192 hrs/day | $7,000 / month | | Cluster Autoscaler + Native HPA | ~112 hrs/day (Slow scaling/fragmentation) | $4,080 / month | | vLLM + KEDA + Karpenter (Spot + On-Demand) | ~64 hrs/day (Aggressive consolidation + Spot mix) | $1,850 / month |

Net Result: Moving to a vLLM + KEDA + Karpenter dynamic infrastructure achieves up to ~60-70% savings over traditional static setups while dynamically absorbing unexpected demand spikes.

Conclusion

Running generative AI workloads doesn't mean writing a blank check to cloud providers. By pairing vLLM's memory efficiency with Karpenter's hyper-fast, group-less node provisioning, you can build a resilient, multi-tenant inference platform that scales smoothly down to zero and rapidly expands when demand spikes.

Takeaways

1. Don't scale LLMs on standard CPU/RAM metrics: Use queue-based metrics like vllm:num_requests_waiting via Prometheus and KEDA.
2. Abandon ASGs for GPU Workloads: Leverage Karpenter's native NodePool constructs to evaluate pod constraints directly.
3. Solve the Cold Start: Cache model weights using persistent storage options (S3 Mountpoint/EFS) to reduce node-to-serving times from minutes to seconds.