Architecting Cost-Optimized GPU Clusters on Kubernetes for Distributed LLM Fine-Tuning | Ecstaticloud Blog

Fine-tuning large language models (LLMs) like Llama 3 70B, Qwen 2.5, or Mistral Large is an expensive endeavor. Dedicated cloud GPU nodes—such as AWS p4d.24xlarge (8x NVIDIA A100 40GB/80GB) or GCP a2-ultragpu-8gb—can quickly drain cloud budgets when running continuously at $30+ per hour per node.

When running distributed fine-tuning workloads with Fully Sharded Data Parallel (FSDP), DeepSpeed Stage 3, or Megatron-LM, naive setups suffer from three major cost traps:

Idle Over-Provisioning: Allocating full 80GB GPUs for auxiliary tasks like data preprocessing, validation, or evaluation adapters.
On-Demand Premium: Paying top-tier prices for static compute instead of leveraging spot/ephemeral instances.
Bandwidth & Boot Latency Penalties: Pulling massive 140GB+ model checkpoints over external networks on every pod restart, wasting high-cost GPU compute time during initialization.

By combining dynamic GPU partitioning, spot instance orchestration with graceful preemption handling, and a distributed local-NVMe model caching layer, you can cut infrastructure costs by up to 60% without compromising training stability.

Here is the exact architecture and implementation blueprint to achieve this on Kubernetes.

The Target Cost-Optimized Architecture

Before diving into configurations, let’s look at the topology of our cloud-agnostic Kubernetes cluster tailored for distributed LLM training:

                  +-------------------------------------------------------+
                  |               Kubernetes Control Plane                |
                  |     (Karpenter Auto-Scaler + Custom Controllers)     |
                  +---------------------------+---------------------------+
                                              |
        +-------------------------------------+-------------------------------------+
        |                                                                           |
        v                                                                           v
+-------------------------------+                                 +-------------------------------+
|     Spot GPU Node Pool        |                                 |    On-Demand / System Pool    |
|   (Heterogeneous p4d/p5/g5)   |                                 |       (c6i.xlarge Nodes)      |
+-------------------------------+                                 +-------------------------------+
|  +-------------------------+  |                                 |  +-------------------------+  |
|  | Torchrun Pod (Rank 0)   |  |                                 |  | Torchrun Master / Ops   |  |
|  | - Direct HostPath NVMe  |  |                                 |  | - Prometheus / Grafana  |  |
|  | - JuiceFS Distributed   |  |                                 |  | - Model Registry Sync   |  |
|  |   Weight Cache          |  |                                 |  +-------------------------+  |
|  +-------------------------+  |                                 +-------------------------------+
|  +-------------------------+  |
|  | NVIDIA GPU Operator     |  |
|  | - Hardware (8x A100)    |  |
|  | - Dynamic MIG/MPS Split |  |
|  +-------------------------+  |
+-------------------------------+

Pillar 1: Dynamic GPU Partitioning (MIG vs. Time-Slicing vs. MPS)

Not all stages of an LLM pipeline require full GPU access. While core matrix multiplication during backpropagation demands unconstrained hardware and high NVLink bandwidth, tasks like dataset tokenization, guardrail validation, and reward model evaluation run efficiently on partial GPUs.

To maximize density and stop wasting full GPUs on non-training tasks, we use three partitioning primitives supported by the NVIDIA GPU Operator:

Implementing Dynamic Slice Profiles via NVIDIA GPU Operator

Instead of statically slicing nodes at boot, deploy a ConfigMap to let the NVIDIA Device Plugin dynamically reconfigure profiles using mig-parted or MPS slicing based on pod annotations.

Below is a production ConfigMap configuring heterogeneous slicing for an 8x A100 80GB node:

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  a100-80gb-mixed: |
    version: v1
    flags:
      migStrategy: mixed
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4
    mig:
      changes:
        - name: mixed-eval-and-train
          devices: [0,1,2,3,4,5] # Whole GPUs for FSDP Distributed Training
          migEnabled: false
        - name: mixed-eval-and-train
          devices: [6,7]         # Sliced GPUs for Validation / Evaluation Tasks
          migEnabled: true
          migDevices:
            "1g.10gb": 7
            "2g.20gb": 3

Apply this configuration through your ClusterPolicy or NvidiaDriver CRD:

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: cluster-policy
spec:
  devicePlugin:
    config:
      name: device-plugin-config
      default: a100-80gb-mixed

Architectural Benefit: This releases up to 25% of your total GPU capacity by packing secondary evaluation and pipeline steps into fractional MIG profiles (1g.10gb), reserving dedicated NVLink-connected GPUs exclusively for the primary distributed training ranks.

Pillar 2: Spot Instance Orchestration & Fault-Tolerant Distributed Training

Spot instances offer 60% to 70% discounts compared to On-Demand pricing. However, distributed training frameworks like PyTorch FSDP or DeepSpeed traditionally fail if a single node disappears, causing all ranks to crash and lose uncheckpointed progress.

To run fine-tuning on Spot instances safely, you need three components:

Flexible Auto-scaling via Karpenter configured across multiple GPU instance types.
Fast Spot Interruption Handling to intercept Cloud 2-minute preemption notices.
Resilient Elastic Training Operators configured to save localized state and rejoin ranks automatically.

1. Provisioning Heterogeneous Spot Pools with Karpenter

Create a Karpenter NodePool that allows fallback across multiple GPU hardware configurations (e.g., A100, H100, L40S) to minimize spot reclamation risks.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-spot-nodepool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: category
          operator: In
          values: ["c"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p4d.24xlarge", "p5.48xlarge", "g5.12xlarge"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-node-class
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 5m

2. Signal-Aware Asynchronous Checkpointing in PyTorch FSDP

When AWS or GCP sends a termination notice, the node receives a 2-minute grace window. We catch this signal using a custom UNIX POSIX signal handler inside our PyTorch wrapper script.

Instead of writing massive multi-gigabyte checkpoints directly to cloud object storage (S3/GCS)—which takes longer than the 2-minute preemption window—write the distributed checkpoint to the local host's NVMe array (/tmp/scratch) first, notify the coordinator, and offload asynchronously.

Here is the fault-tolerant checkpointing hook:

import signal
import sys
import torch
import torch.distributed as dist
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp import StateDictType, FullStateDictConfig

class SpotTerminationHandler:
    def __init__(self, model, optimizer, local_nvme_path="/tmp/scratch"):
        self.model = model
        self.optimizer = optimizer
        self.local_nvme_path = local_nvme_path
        self.interrupted = False
        
        # Register POSIX SIGTERM handler sent by K8s pod eviction
        signal.signal(signal.SIGTERM, self._handle_sigterm)

    def _handle_sigterm(self, signum, frame):
        print("[WARNING] SIGTERM Received! Executing Emergency Checkpoint...")
        self.interrupted = True
        self.save_emergency_checkpoint()
        sys.exit(0)

    def save_emergency_checkpoint(self):
        # Save local FSDP sharded state (fastest method, no rank gathering needed)
        with FSDP.state_dict_type(self.model, StateDictType.SHARDED_STATE_DICT):
            state_dict = {
                "model": self.model.state_dict(),
                "optimizer": FSDP.optim_state_dict(self.model, self.optimizer),
            }
            rank = dist.get_rank()
            checkpoint_file = f"{self.local_nvme_path}/emergency_rank_{rank}.pt"
            torch.save(state_dict, checkpoint_file)
            print(f"[SUCCESS] Emergency checkpoint rank {rank} saved to NVMe: {checkpoint_file}")

        # Trigger background process to stream NVMe file to S3
        import subprocess
        subprocess.Popen(["aws", "s3", "cp", checkpoint_file, f"s3://my-llm-checkpoints/run-01/rank_{rank}.pt"])

# Usage inside PyTorch training loop
# handler = SpotTerminationHandler(fsdp_model, optimizer)
# for epoch in range(epochs):
#     ...

3. Elastic Pod Restoration with Kubeflow PyTorchJob

Use the Kubeflow Training Operator (PyTorchJob) configured with elasticPolicy to automatically replace preempted spot pods and resume training from the latest checkpoint without manual intervention.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama3-70b-finetune
  namespace: ml-training
spec:
  elasticPolicy:
    minReplicas: 2
    maxReplicas: 4
    rdzvBackend: etcd
    rdzvEndpoint: etcd-service.ml-training.svc.cluster.local:2379
    maxRestarts: 10
  pytorchReplicaSpecs:
    Worker:
      replicas: 2
      restartPolicy: OnFailure
      template:
        spec:
          nodeSelector:
            karpenter.sh/capacity-type: spot
          containers:
            - name: pytorch
              image: my-registry/llm-finetune:v1.2
              command:
                - torchrun
                - --nnodes=2
                - --nproc_per_node=8
                - train.py
                - --checkpoint-dir=/mnt/fast-cache/checkpoints
              resources:
                limits:
                  nvidia.com/gpu: "8"
                  ephemeral-storage: "500Gi"
              volumeMounts:
                - mountPath: /mnt/fast-cache
                  name: nvme-volume
          volumes:
            - name: nvme-volume
              hostPath:
                path: /mnt/k8s-local-nvme

Pillar 3: Automated Model Caching & High-Throughput I/O

When launching dozens of transient spot instances, downloading raw safetensors (e.g., Llama-3-70B requires ~140GB in bfloat16) on every pod cold-start causes significant issues:

Network throttling at the cloud provider NAT gateway.
High bandwidth egress/ingress charges.
10–15 minutes of idle GPU time per pod boot, costing money while waiting for I/O.

To achieve zero-copy, sub-minute warm-starts, deploy a distributed POSIX read-through caching engine using JuiceFS or Fluid backed by local NVMe drives installed directly on the physical nodes.

+-----------------------------------------------------------------------+
|                             JuiceFS Engine                            |
|             (Metadata Engine: Redis / Object Store: S3)               |
+-----------------------------------------------------------------------+
                                   |
         +-------------------------+-------------------------+
         |                                                   |
         v                                                   v
+-------------------------------+   Read Cache   +-------------------------------+
|       Worker Node A           | <------------> |       Worker Node B           |
|  Local NVMe Drive (/mnt/nvme) |  Peer-to-Peer  |  Local NVMe Drive (/mnt/nvme) |
+-------------------------------+   Prefetch     +-------------------------------+
         |                                                   |
         +-------------------------+-------------------------+
                                   |
                                   v
             +-------------------------------------------+
             | Central Weight Registry (S3 / HuggingFace)|
             +-------------------------------------------+

JuiceFS CSI Configuration for Local NVMe Speed

Set up a JuiceFS persistent volume that mounts the local host NVMe scratch space as an ephemeral read cache. Subsequent pod launches on any node in the cluster fetch pre-cached chunks at local disk speed (up to 6.8 GB/s read throughput).

apiVersion: v1
kind: PersistentVolume
metadata:
  name: juicefs-pv
spec:
  capacity:
    storage: 10Ti
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: csi.juicefs.org
    volumeHandle: llm-cache-volume
    volumeAttributes:
      name: llm-cache
      metaurl: redis://redis-service.ml-cache.svc.cluster.local:6379/1
      storage: s3
      bucket: https://my-llm-weights-bucket.s3.us-west-2.amazonaws.com
      # Enable agresive local NVMe caching
      cache-dir: /mnt/k8s-local-nvme/juicefs-cache
      cache-size: "500000" # 500 GB local NVMe cache per node
      free-space-ratio: "0.1"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  namespace: ml-training
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Ti
  volumeName: juicefs-pv

Loading Weights in PyTorch without Memory Duplication

Combine the high-throughput JuiceFS mount with HuggingFace's zero-copy accelerate initialization. This ensures you read directly from local cached files without allocating excess CPU RAM before streaming to GPU VRAM:

import torch
from transformers import AutoModelForCausalLM, AutoConfig
from accelerate import init_empty_weights

model_path = "/mnt/fast-cache/huggingface/hub/models--meta-llama--Meta-Llama-3-70B"

# Step 1: Initialize model with empty weights (0 seconds, 0 VRAM)
config = AutoConfig.from_pretrained(model_path)
with init_empty_weights():
    model = AutoModelForCausalLM.from_config(config)

# Step 2: Tie tensors directly to GPU VRAM from local high-speed NVMe/JuiceFS mount
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    low_cpu_mem_usage=True
)

print("[SUCCESS] Model loaded to VRAM in <12 seconds!")

Cost Comparison & Real-World Benchmarks

Here is an analysis of fine-tuning Llama 3 70B across a 32x A100 (80GB) cluster over 100 total hours of training compute time:

Standard Setup vs. Optimized Architecture

Final Cost Reduction: 66.3% Cost Savings ($8,696 saved per training run) with zero degradation in training convergence or floating-point precision TFLOPS throughput.

Conclusion & Architectural Blueprint Checklist

To implement this cost-optimized GPU infrastructure in your cluster, follow this rollout strategy:

Deploy NVIDIA GPU Operator using dynamic ClusterPolicy configurations. Slices auxiliary pods into MIG/MPS instances while leaving primary training channels unconstrained.
Configure Karpenter NodePools to request Spot capacity across multiple GPU instance families (p4d, p5, g5) to maximize spot fill rates.
Wrap your PyTorch FSDP / DeepSpeed scripts with POSIX SIGTERM handlers to push sharded emergency checkpoints to local NVMe before cloud reclamation occurs.
Deploy Kubeflow PyTorchJob alongside an etcd rendezvous backend to enable auto-healing rank reassignment upon node termination.
Implement JuiceFS / Fluid CSI Drivers over local NVMe arrays to eliminate model weight download bottlenecks during pod boot cycles.

By moving away from static, On-Demand GPU deployments and embracing automated, dynamic, and fault-tolerant Kubernetes orchestration, you can scale AI capabilities without overwhelming your infrastructure budget.