Architecting Zero-Downtime LLM Fine-Tuning Pipelines on AWS EKS with Ray and Karpenter

Fine-tuning modern Large Language Models (LLMs) with tens or hundreds of billions of parameters is no longer an academic exercise—it is a core requirement for enterprise AI teams adapting foundation models to domain-specific datasets. However, executing multi-node distributed training (using techniques like PyTorch FSDP, DeepSpeed, or Megatron-LM) introduces severe operational challenges:

1. Astronomical Compute Costs: Running dedicated On-Demand GPU clusters (e.g., NVIDIA A100/H100 instances) for training runs costs tens of thousands of dollars per week.
2. Preemption Vulnerability: Leveraging AWS EC2 Spot instances reduces compute costs by up to 70–90%, but introduces sudden node reclaims. Unhandled preemptions ruin multi-node training state and cause total pipeline failure.
3. Inference Coexistence: Production clusters must host low-latency inference endpoints (e.g., vLLM, TGI) with strict SLA guarantees alongside elastic, resource-heavy fine-tuning jobs without noisy-neighbor degradation or cluster downtime.

To solve this, we will architect an enterprise-grade, zero-downtime distributed LLM fine-tuning and inference platform on Amazon EKS, powered by KubeRay for Ray cluster management, Karpenter for high-speed, just-in-time GPU provisioning, and AWS Elastic Fabric Adapter (EFA) for ultra-low latency interconnects.

Target Architecture Overview

The platform uses a decoupled control plane and data plane architecture:

                          +------------------------------------------+
                          |              AWS EKS Cluster             |
                          |                                          |
                          |  +------------------------------------+  |
                          |  |        Control Plane / System      |  |
                          |  |  [KubeRay Operator] [Karpenter]    |  |
                          |  |  [Inference Service (On-Demand)]   |  |
                          |  +------------------------------------+  |
                          +--------------------+---------------------+
                                               |
                     +-------------------------+-------------------------+
                     |                                                   |
                     v                                                   v
        +-------------------------+                         +-------------------------+
        |   Karpenter NodePool    |                         |   Karpenter NodePool    |
        |  (Spot - Ray Workers)   |                         |  (On-Demand - Ray Head) |
        |                         |                         |                         |
        |  +-------------------+  |                         |  +-------------------+  |
        |  |  g5.12xl / p4d /  |  |                         |  |   m6i.xlarge      |  |
        |  |  p5.48xlarge      |  |                         |  |   Ray Head Node   |  |
        |  +---------+---------+  |                         |  +-------------------+  |
        +------------|------------+                         +-------------------------+
                     |
                     +====== EFA (400 Gbps GPUDirect RDMA) ======+
                     |                                           |
                     v                                           v
        +-------------------------+                         +-------------------------+
        |   S3 / FSx for Lustre   |<=======================>|    Prometheus/Grafana   |
        |  (Dataset & Checkpoints)|                         |   (Metrics & Tracing)   |
        +-------------------------+                         +-------------------------+

Key Architectural Pillars

1. Ray & KubeRay Operator: Orchestrates Python-native distributed training (Ray Train, Ray Data). Automatically isolates worker failures and manages execution graphs.
2. Karpenter: Dynamically provisions high-performance GPU instances (p4d.24xlarge, p5.48xlarge, g5.12xlarge) in seconds directly handling Kubernetes Pending pod signals without the latency of cluster-autoscaler group abstractions.
3. Fault-Tolerant Checkpointing: Dynamic integration with AWS S3 or FSx for Lustre using PyTorch Distributed Checkpoint (DCP) format to allow seamless training resumption under 3 minutes after a Spot termination.
4. Network Optimization: EFA bindings wired directly into Ray pods to allow inter-node NCCL communications via GPUDirect RDMA.

Step 1: Configuring Karpenter for Enterprise GPU Compute

Standard Kubernetes Cluster Autoscaler is poorly suited for GPU workloads due to slow scaling loops, lack of support for multi-instance type pooling, and complex node-group combinations. Karpenter provisions compute directly based on pending Pod requests, choosing the optimal mix of EC2 instances, Availability Zones, and Capacity Types (Spot vs. On-Demand).

We define two distinct NodePool resources:

1. Ray Head Node Pool: Reliable, On-Demand CPU-only or low-cost GPU node.
2. Ray Worker Node Pool: Dynamic, Spot-heavy GPU nodes with Elastic Fabric Adapter (EFA) interfaces enabled.

Karpenter EC2NodeClass Definition

First, set up an EC2NodeClass configured for deep learning workloads, custom local NVMe storage initialization, and EFA device attachments:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-training-nodeclass
spec:
  amiSelectorTerms:
    - alias: al2023@latest # Or official EKS Optimized GPU AMI
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: eks-ai-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: eks-ai-cluster
  role: KarpenterNodeRole-eks-ai-cluster
  
  # Enable EFA drivers and devices automatically on supported instances (e.g., p4d, p5)
  efaEnabled: true

  # Expand ephemeral storage using local instance store NVMe SSDs
  userData: |
    #!/bin/bash
    set -o errexit
    set -o pipefail
    set -o nounset

    # Format local NVMe devices into RAID0 for fast temporary scratch space / PyTorch cache
    if ls /dev/nvme*n1 1> /dev/null 2>&1; then
      DEVICES=$(ls /dev/nvme*n1 | grep -v nvme0n1 || true)
      if [ -n "$DEVICES" ]; then
        mdadm --create --verbose /dev/md0 --level=0 --raid-devices=$(echo $DEVICES | wc -w) $DEVICES
        mkfs.ext4 -F /dev/md0
        mkdir -p /mnt/k8s-disks/by-name/local-nvme
        mount -o noatime,nodiratime /dev/md0 /mnt/k8s-disks/by-name/local-nvme
        chmod 777 /mnt/k8s-disks/by-name/local-nvme
      fi
    fi

  tags:
    Workload: RayTraining
    Environment: Production

Karpenter NodePool for Ray Spot Workers

Next, declare the NodePool. We leverage multiple GPU instance families (g5, p4d, p5) to maximize Spot availability across AWS pools and reduce preemption frequency:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ray-gpu-workers-spot
spec:
  template:
    metadata:
      labels:
        workload: ray-worker
        gpu-type: nvidia
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-training-nodeclass
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"] # Prefers Spot, falls back if completely exhausted
        - key: category
          operator: In
          values: ["g", "p"]
        - key: instance-family
          operator: In
          values: ["g5", "p4d", "p5"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
        - key: ray.io/node-type
          value: "worker"
          effect: NoSchedule
      limits:
        cpu: "1000"
        memory: 4000Gi
        nvidia.com/gpu: "64"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 5m

Step 2: KubeRay Cluster Setup with EFA Interconnects

With Karpenter ready to scale our dynamic infrastructure, we deploy the Ray Cluster using the KubeRay Operator.

The Ray Head runs on an On-Demand instance to ensure orchestrator longevity. Ray Workers launch in the Karpenter Spot pool, mounting host shared memory (/dev/shm) and EFA network interface devices for multi-node PyTorch Distributed training.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: llama3-70b-finetune
  namespace: ray-system
spec:
  rayVersion: '2.35.0'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
      num-gpus: '0' # Head node acts purely as coordinator
      block: 'true'
    template:
      spec:
        containers:
          - name: ray-head
            image: rayproject/ray-ml:2.35.0-py310-gpu
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
              requests:
                cpu: "4"
                memory: "16Gi"
            ports:
              - containerPort: 6379
                name: gcs
              - containerPort: 8265
                name: dashboard
              - containerPort: 10001
                name: client
  workerGroupSpecs:
    - groupName: gpu-workers
      replicas: 4
      minReplicas: 0
      maxReplicas: 16
      rayStartParams:
        block: 'true'
      template:
        metadata:
          labels:
            workload: ray-worker
        spec:
          nodeSelector:
            workload: ray-worker
          tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule
            - key: ray.io/node-type
              operator: Equal
              value: worker
              effect: NoSchedule
          containers:
            - name: ray-worker
              image: rayproject/ray-ml:2.35.0-py310-gpu
              securityContext:
                capabilities:
                  add: ["SYS_PTRACE", "IPC_LOCK"] # Required for EFA RDMA communication
              resources:
                limits:
                  cpu: "48"
                  memory: "192Gi"
                  nvidia.com/gpu: "4"
                  vpc.amazonaws.com/efa: "4" # Request Elastic Fabric Adapter interface
                requests:
                  cpu: "42"
                  memory: "180Gi"
                  nvidia.com/gpu: "4"
                  vpc.amazonaws.com/efa: "4"
              volumeMounts:
                - mountPath: /dev/shm
                  name: dshm
                - mountPath: /mnt/scratch
                  name: local-nvme
          volumes:
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 64Gi # Required for PyTorch shared tensor memory
            - name: local-nvme
              hostPath:
                path: /mnt/k8s-disks/by-name/local-nvme

Step 3: Zero-Downtime Resilience & Spot Preemption Mechanics

Spot instances give us high cost savings, but they can be reclaimed by AWS with a 2-minute warning. Achieving zero-downtime during fine-tuning requires a multi-tier recovery pipeline:

1. Spot Interruption Handling: Karpenter natively monitors AWS Health EventBridge metrics and instance metadata endpoints (http://169.254.169.254/latest/meta-data/spot/instance-action).
2. Graceful Drain: Upon receiving the 2-minute preemption signal, Karpenter taints the node, setting it to Ready,SchedulingDisabled, and fires a drain event.
3. Ray Engine Fault Tolerance: KubeRay catches worker eviction signals and invokes Ray Train’s actor recovery loop.
4. PyTorch Distributed Checkpoint (DCP): Workers state is recovered from the last synced S3 checkpoint without restarting the entire execution pipeline.

+-----------------------------------------------------------------------------------+
| 1. AWS sends 2-Minute Spot Interruption Warning                                   |
+-----------------------------------------------------------------------------------+
                                        |
                                        v
+-----------------------------------------------------------------------------------+
| 2. Karpenter intercepts event -> Taints Node -> Triggers Kubernetes Pod Drain     |
+-----------------------------------------------------------------------------------+
                                        |
                                        v
+-----------------------------------------------------------------------------------+
| 3. Ray Worker catches SIGTERM -> Triggers emergency checkpoint sync to S3        |
+-----------------------------------------------------------------------------------+
                                        |
                                        v
+-----------------------------------------------------------------------------------+
| 4. Pod terminates -> KubeRay requests replacement pod -> Karpenter provisions replacement |
+-----------------------------------------------------------------------------------+
                                        |
                                        v
+-----------------------------------------------------------------------------------+
| 5. Ray Job automatically resumes fine-tuning from last saved S3 step              |
+-----------------------------------------------------------------------------------+

Fault-Tolerant Distributed Training Python Script

Here is an enterprise PyTorch FSDP fine-tuning entrypoint built with ray.train and AWS S3 checkpointing logic:

import os
import tempfile
import torch
import torch.distributed as dist
import ray
import ray.train
from ray.train import ScalingConfig, Checkpoint
from ray.train.torch import TorchTrainer, TorchConfig
import boto3

def train_loop_per_worker(config: dict):
    """
    Main PyTorch Fine-Tuning Loop executed inside every Ray Worker
    """
    # Environment Setup for NCCL + EFA
    os.environ["FI_PROVIDER"] = "efa"
    os.environ["FI_EFA_USE_DEVICE_RDMA"] = "1"
    os.environ["NCCL_DEBUG"] = "INFO"

    # Initialize Distributed Process Group
    ray.train.torch.enable_reproducibility()
    
    # Load Model (e.g., Llama-3-70B via Hugging Face PEFT / LoRA)
    local_rank = ray.train.get_context().get_local_rank()
    world_rank = ray.train.get_context().get_world_rank()
    
    device = torch.device(f"cuda:{local_rank}")
    torch.cuda.set_device(device)

    # Initialize FSDP or DeepSpeed Engine
    model = create_fsdp_model(config["model_id"]).to(device)
    optimizer = torch.optim.AdamW(model.parameters(), lr=config["lr"])

    start_epoch = 0
    
    # Check for existing checkpoint (Recovery scenario from Spot eviction)
    checkpoint = ray.train.get_checkpoint()
    if checkpoint:
        with checkpoint.as_directory() as checkpoint_dir:
            print(f"[Rank {world_rank}] Restoring state from checkpoint: {checkpoint_dir}")
            model_state = torch.load(os.path.join(checkpoint_dir, "model.pt"), map_location=device)
            opt_state = torch.load(os.path.join(checkpoint_dir, "optimizer.pt"), map_location=device)
            
            model.load_state_dict(model_state)
            optimizer.load_state_dict(opt_state)
            
            metadata = torch.load(os.path.join(checkpoint_dir, "metadata.pt"))
            start_epoch = metadata["epoch"] + 1

    dataset = get_dataset_shard(config["data_path"], world_rank)

    # Training Loop
    for epoch in range(start_epoch, config["epochs"]):
        for step, batch in enumerate(dataset):
            optimizer.zero_grad()
            inputs = {k: v.to(device) for k, v in batch.items()}
            outputs = model(**inputs)
            loss = outputs.loss
            loss.backward()
            optimizer.step()

            # Save distributed checkpoint every N steps
            if step % config["checkpoint_frequency"] == 0 and step > 0:
                if local_rank == 0:
                    with tempfile.TemporaryDirectory() as temp_dir:
                        torch.save(model.state_dict(), os.path.join(temp_dir, "model.pt"))
                        torch.save(optimizer.state_dict(), os.path.join(temp_dir, "optimizer.pt"))
                        torch.save({"epoch": epoch, "step": step}, os.path.join(temp_dir, "metadata.pt"))
                        
                        # Save checkpoint to Ray context (Synced to S3 backend)
                        ray.train.report(
                            metrics={"loss": loss.item(), "epoch": epoch, "step": step},
                            checkpoint=Checkpoint.from_directory(temp_dir)
                        )

# Orchestrator Function
def run_fine_tuning():
    ray.init()

    # Configure Distributed Training Topology
    scaling_config = ScalingConfig(
        num_workers=8, # Across 2 x 4-GPU Nodes
        use_gpu=True,
        resources_per_worker={"CPU": 10, "GPU": 1},
    )

    # S3 Bucket Persistence for Checkpointing across Node Evictions
    run_config = ray.train.RunConfig(
        name="llama3-70b-qlora",
        storage_path="s3://my-enterprise-ai-checkpoints/ray_jobs/",
        failure_config=ray.train.FailureConfig(
            max_failures=5 # Allows up to 5 Spot preemptions per run without job termination
        )
    )

    trainer = TorchTrainer(
        train_loop_per_worker=train_loop_per_worker,
        train_loop_config={
            "model_id": "meta-llama/Meta-Llama-3-70B",
            "lr": 2e-5,
            "epochs": 3,
            "checkpoint_frequency": 100,
            "data_path": "s3://my-enterprise-ai-checkpoints/datasets/domain_specific.arrow"
        },
        torch_config=TorchConfig(backend="nccl"),
        scaling_config=scaling_config,
        run_config=run_config
    )

    result = trainer.fit()
    print(f"Training Completed. Best Checkpoint: {result.checkpoint}")

if __name__ == "__main__":
    run_fine_tuning()

Step 4: Interconnect Optimization with AWS EFA & NCCL Tuning

When executing FSDP/DeepSpeed across multiple nodes, network throughput is the primary bottleneck. AWS Elastic Fabric Adapter (EFA) delivers custom OS-bypass hardware interfaces that scale intra-cluster bandwidth up to 400 Gbps per node.

To ensure your KubeRay worker containers take full advantage of EFA, inject these environment variables globally across your manifests:

env:
  # Enable EFA for AWS GPUDirect RDMA
  - name: FI_PROVIDER
    value: "efa"
  - name: FI_EFA_USE_DEVICE_RDMA
    value: "1"
  - name: FI_EFA_FORK_SAFE
    value: "1"
  
  # Tuning NCCL communication primitives
  - name: NCCL_DEBUG
    value: "WARN"
  - name: NCCL_BUFFSIZE
    value: "8388608" # Increase ring buffer size to 8MB for high-parameter LLM synchronization
  - name: NCCL_P2P_DISABLE
    value: "0"

Validating EFA Operational Status

Verify that EFA is active inside the running Ray worker container using fi_info:

kubectl exec -it ray-worker-pod -n ray-system -- fi_info -p efa

Expected output confirms the system provider is selected properly:

provider: efa
    fabric: efa
    domain: efa_0-rdm
    version: 120.10
    type: FI_EP_RDM

Day-2 Operations & Production Best Practices

To safely run multi-tenant fine-tuning alongside mission-critical inference, enforce these battle-tested architectural guidelines:

1. Hard Resource Isolation

Never allow fine-tuning jobs to share nodes with live inference workloads (such as vLLM or TGI pods). Use explicit Kubernetes Taints and Tolerations in conjunction with Karpenter NodePool definitions to enforce strict physical isolation.

2. Monitoring & Metrics Observability

Deploy the Prometheus Operator and scrape metrics exposed on KubeRay Head http://<head-pod>:8265/metrics alongside Karpenter operational metrics (karpenter_nodeclaims_created, karpenter_nodes_disrupted_total).

Track the following critical key performance indicators (KPIs):

• EFA Transceiver Drops / Retries: Detects underlying network degradation between GPU nodes.
• Ray Checkpoint Latency: Ensures time spent writing model checkpoints to S3 does not exceed 5% of overall step execution time.
• Node Recovery Time: Measures the time delta from Spot eviction signal to standard PyTorch state resumption (Goal: $< 180$ seconds).

3. Storage Subsystem Selection

For LLM fine-tuning pipelines:

• Use S3 with mountpoint-s3 or native boto3 stream loading for dataset fetching.
• Use FSx for Lustre (linked to S3) if training datasets consist of hundreds of millions of small files where low-latency IOPS are necessary.
• Utilize local NVMe instance storage mapped to /tmp / /mnt/scratch for temporary layer caching and local gradient accumulation buffers.

Conclusion

By combining AWS EKS, Karpenter, KubeRay, and EFA, enterprise teams can build an infrastructure platform that lowers the cost of fine-tuning large language models. Spot preemptions change from critical failures into routine, auto-remediated events handled by Ray’s fault tolerance and PyTorch Distributed Checkpoints.

This architecture enables cost-optimized training on dynamic GPU resources while keeping production inference workloads isolated and stable on On-Demand nodes.