Scaling DeepSeek-R1 on Amazon EKS: Architecting Cost-Effective Private LLM Inference with vLLM and Karpenter | Ecstaticloud Blog

The arrival of open-weights reasoning models like DeepSeek-R1 has fundamentally shifted the AI deployment playbook. By leveraging chain-of-thought (CoT) reinforcement learning, DeepSeek-R1 delivers performance matching proprietary frontier models like OpenAI’s o1 across mathematics, code, and complex reasoning tasks.

However, running enterprise-grade inference for high-parameter reasoning models presents unique infrastructure challenges:

Massive VRAM Requirements: DeepSeek-R1’s full architecture spans 671 Billion parameters (using a Mixture-of-Experts (MoE) design with 37B active parameters per token). Even distilled variants (8B to 70B) demand substantial GPU memory footprints.
Variable Latency & Token Expansion: CoT reasoning models generate significantly more output tokens before returning a final answer. This elongates request lifecycles and places severe pressure on Key-Value (KV) cache memory.
GPU Cloud Economics: Idle high-end NVIDIA GPUs (A100, H100, L40S, A10G) severely destroy infrastructure ROI.

In this guide, we will architect a production-ready, highly elastic private inference engine on Amazon EKS using vLLM for high-throughput inference serving and Karpenter for just-in-time GPU auto-provisioning.

Architectural Overview

To serve DeepSeek-R1 efficiently while minimizing costs, we combine micro-batching optimizations at the engine level with dynamic cloud capacity orchestrations at the Kubernetes layer.

                             [ Enterprise Clients / Apps ]
                                          │
                                          ▼
                             [ AWS ALB / Ingress Controller ]
                                          │
                                          ▼
                      [ EKS Service (vLLM Router / Engine) ]
                                          │
            ┌─────────────────────────────┴─────────────────────────────┐
            ▼                                                           ▼
┌───────────────────────────────┐           ┌────────────────────────────────┐
│  vLLM Pod 1 (Pod Specs)       │           │  vLLM Pod N (Pod Specs)        │
│  - Tensor Parallelism: 4/8    │           │  - Tensor Parallelism: 4/8     │
│  - Engine: vLLM + PagedAttn   │           │  - Engine: vLLM + PagedAttn    │
└───────────────┬───────────────┘           └────────────────┬───────────────┘
                │                                            │
                ▼                                            ▼
┌───────────────────────────────┐           ┌────────────────────────────────┐
│  Karpenter Provisioned Node   │           │  Karpenter Provisioned Node    │
│  g5.12xlarge / p4d.24xlarge   │           │  g5.12xlarge / Spot Fallback   │
└───────────────────────────────┘           └────────────────────────────────┘

Core Architecture Components

vLLM Engine: Utilizes PagedAttention to eliminate KV cache memory fragmentation, allowing concurrent request execution with continuous dynamic batching.
Karpenter: A high-performance, node-lifecycle Kubernetes autoscaler that bypasses static node groups. It evaluates unschedulable vLLM pod requirements (GPU quantity, CUDA capability, VRAM depth) and provisions optimal EC2 instances in under 60 seconds.
AWS S3 / Amazon FSx for OpenZFS: High-throughput storage layer for rapid model weight streaming into host NVMe scratch storage.

DeepSeek-R1 Sizing & Compute Hardware Selection

Understanding your target deployment variant determines your AWS compute topology. DeepSeek-R1 comes in two primary forms: the full 671B MoE model and Distilled Variants (fine-tuned on dense architectures like Llama and Qwen).

Model Variant Sizing Matrix

Note: For production environments processing enterprise concurrency, allocate at least 30-40% additional VRAM above base model weights strictly for the vLLM KV Cache allocation.

vLLM Engine Configuration Tactics

Standard PyTorch serving wrappers fail under high-concurrency LLM workloads. vLLM solves this by managing memory allocations similar to virtual memory in operating systems.

Key execution optimizations for DeepSeek-R1:

PagedAttention & KV Cache Allocation: Configured via --gpu-memory-utilization (typically 0.90 to 0.95).
Chunked Prefills (--enable-chunked-prefill): Crucial for reasoning models. CoT inference intermingles large prompt prefills with continuous long token generation. Chunking breaks large prefills into smaller chunks, preventing tail-latency spikes (Time To First Token - TTFT) for existing streams.
Tensor Parallelism (TP): Distributes individual layer execution across multiple local GPUs using NCCL over NVLink/NVSwitch interfaces.

Deploying Karpenter for GPU Auto-Provisioning

Traditional Cluster Autoscaler (CAS) is too slow and rigid for complex heterogenous GPU fleets. Karpenter natively understands GPU constraints, pod affinity, local storage, and EC2 Spot market fluctuations.

Step 1: Install Karpenter `EC2NodeClass`

The EC2NodeClass custom resource defines the AWS-specific configuration (AMIs, IAM roles, Block Device Mappings, Subnets, and NVMe Instance Storage formatting).

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-node-class
spec:
  amiFamily: AL2 # Use AWS EKS Optimized Accelerator AMI
  role: "KarpenterNodeRole-EKS-Cluster"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "production-eks-cluster"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "production-eks-cluster"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        iops: 3000
        throughput: 125
    # High-performance local ephemeral storage for Model Weights cache
    - deviceName: /dev/sdb
      ebs:
        volumeSize: 500Gi
        volumeType: gp3
        iops: 10000
        throughput: 500
  tags:
    Workload: "DeepSeek-R1-Inference"

Step 2: Define the Karpenter `NodePool`

This NodePool targets high-throughput GPU instances (g5, g6, p4d, p5), permitting dynamic allocation of Spot or On-Demand instances with rapid consolidated scaling policies.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference-nodepool
spec:
  template:
    metadata:
      labels:
        workload: deepseek-inference
        accelerator: nvidia
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-node-class
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: instance.k8s.aws/instance-family
          operator: In
          values: ["g5", "g6", "p4d", "p5"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  limits:
    nvidia.com/gpu: "64" # Limit cluster-wide GPUs managed by this pool
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s

Deploying DeepSeek-R1-Distill-70B on EKS

Below is an enterprise-grade Deployment manifest configured for DeepSeek-R1-Distill-Llama-70B requiring 4x NVIDIA A10G GPUs (TP=4) on a single g5.12xlarge instance.

Step 1: Create the vLLM Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-deepseek-r1-70b
  namespace: llm-inference
  labels:
    app: vllm-deepseek-r1-70b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-deepseek-r1-70b
  template:
    metadata:
      labels:
        app: vllm-deepseek-r1-70b
    spec:
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      nodeSelector:
        workload: deepseek-inference
      containers:
        - name: vllm-server
          image: vllm/vllm-openai:v0.7.1
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - "--model=deepseek-ai/DeepSeek-R1-Distill-Llama-70B"
            - "--tensor-parallel-size=4"
            - "--max-model-len=16384"
            - "--gpu-memory-utilization=0.92"
            - "--enable-chunked-prefill=true"
            - "--max-num-batched-tokens=8192"
            - "--trust-remote-code"
            - "--port=8000"
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: token
            - name: NCCL_DEBUG
              value: "INFO"
          ports:
            - containerPort: 8000
              name: http
          resources:
            requests:
              cpu: "16"
              memory: "64Gi"
              nvidia.com/gpu: "4"
            limits:
              cpu: "32"
              memory: "128Gi"
              nvidia.com/gpu: "4"
          volumeMounts:
            - mountPath: /dev/shm
              name: dshm
            - mountPath: /root/.cache/huggingface
              name: model-cache
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 15
      volumes:
        # Crucial: Shared memory setup for Inter-process & Inter-GPU Communication via PyTorch
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi
        # HostPath mount to persist HuggingFace downloaded weights across pod restarts
        - name: model-cache
          hostPath:
            path: /mnt/k8s-disks/by-name/scratch/huggingface
            type: DirectoryOrCreate

Step 2: Expose Service via AWS Load Balancer Controller

apiVersion: v1
kind: Service
metadata:
  name: vllm-deepseek-service
  namespace: llm-inference
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8000
      name: http
      protocol: TCP
  selector:
    app: vllm-deepseek-r1-70b

Weight Loading Acceleration Tactics

Downloading 140GB+ of model weights during pod creation leads to unacceptably high startup latencies (often 10–20 minutes), causing Kubernetes readiness probes to time out.

To achieve dynamic scaling, implement one of these two architectural patterns:

Pattern A: S3 Mountpoint CSI Driver

Mount an S3 bucket containing pre-downloaded safetensors directly into the container using the AWS S3 Mountpoint CSI Driver. This delivers high aggregate throughput without consuming local storage overhead.

Pattern B: EBS Snapshot Warm Pools / Custom AMI

Pre-bake the model weights into an Amazon EBS volume, take a snapshot, and configure Karpenter to instantiate EC2 instances with block devices initialized from this snapshot. This drops container cold-start times from 15 minutes to under 90 seconds.

Advanced Cost-Optimization Strategies

Scaling private LLMs natively on AWS requires fine-tuned financial control:

1. Mixed Capacity Overprovisioning & Spot Fallback

Configure Karpenter to attempt provisioning GPU instances under EC2 Spot Capacity. While Spot instances can be interrupted with a 2-minute warning, reasoning workloads running stateless behind an ingress controller can handle mid-stream drops gracefully:

# In NodePool configuration
- key: karpenter.sh/capacity-type
  operator: In
  values: ["spot", "on-demand"]

Karpenter prioritizes Spot instances automatically. If Spot capacity is unavailable, it immediately falls back to On-Demand.

2. Time-Based / Queue-Depth Driven Autoscaling

Combine KEDA (Kubernetes Event-driven Autoscaling) with Prometheus metrics exported directly by vLLM (e.g., vllm:num_requests_waiting and vllm:gpu_cache_usage_perc).

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-autoscaler
  namespace: llm-inference
spec:
  scaleTargetRef:
    name: vllm-deepseek-r1-70b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting{app="vllm-deepseek-r1-70b"})
        threshold: '5'

When the queued waiting requests exceed 5, KEDA triggers scaling events. Pods request GPU allocations, prompting Karpenter to provision matching compute resources immediately.

Production Verification & Monitoring Metrics

To ensure latency SLA targets for reasoning tasks (CoT processing), monitor the following metrics using vLLM's native /metrics endpoint:

Time to First Token (TTFT): Measures initial context processing speed. Spikes indicate prompt prefill bottlenecks; mitigate using --enable-chunked-prefill.
Inter-Token Latency (ITL): Measures continuous generation throughput. Spikes signal KV-cache memory saturation or inefficient Tensor Parallelism communication overhead across GPUs.
GPU KV Cache Usage Percentage (vllm:gpu_cache_usage_perc): If this metric consistently exceeds 0.85, queue build-ups are imminent. Trigger autoscaling before usage hits 1.0.

Summary

Deploying DeepSeek-R1 privately on Amazon EKS puts you in total control over data security, model execution, and inference costs. By pairing vLLM's PagedAttention with Karpenter's dynamic node provisioning, your cluster auto-scales compute on demand—ensuring sub-second generation latencies without burning cloud budget on idle GPUs.