Optimizing LLM Inference Latency on AWS EKS Using vLLM and TensorRT-LLM

When serving 70B+ parameter Large Language Models (LLMs) like Llama 3 or Mixtral 8x7B at enterprise scale, naive deployment architectures quickly hit a wall. Infrastructure teams face a brutal trade-off: either tolerate P99 latencies stretching into seconds—ruining user experience—or over-provision expensive NVIDIA A100/H100 GPU instances, sending cloud costs skyrocketing.

At Ecstaticloud, we recently re-engineered the LLM serving infrastructure for a high-throughput enterprise platform running on Amazon Elastic Kubernetes Service (AWS EKS). By moving away from conventional Hugging Face Transformers-based endpoints and implementing a dual-engine strategy utilizing vLLM and NVIDIA TensorRT-LLM, we achieved a 60% reduction in P99 inference latency while cutting total GPU node expenditure by 42%.

In this technical deep dive, we walk through the bottleneck mechanics of LLM serving, the Kubernetes infrastructure blueprint on AWS EKS, and the production-ready configurations for both vLLM and TensorRT-LLM.

1. Anatomy of the LLM Inference Bottleneck

To optimize LLM inference, you must first understand why standard model serving architectures fail under load. LLM inference runs in two distinct phases with entirely different computational profiles:

+-----------------------------------------------------------------------+
|                            INFERENCE PHASES                           |
+-----------------------------------------------------------------------+
|  1. Prefill Phase (Context Processing)                               |
|     - Input prompt processing in parallel                            |
|     - Compute-bound (GEMM operations)                                |
|     - High FLOPS utilization                                         |
+-----------------------------------------------------------------------+
                                   |
                                   v
+-----------------------------------------------------------------------+
|  2. Generation Phase (Autoregressive Decoding)                       |
|     - Token-by-token generation (sequential)                        |
|     - Memory-bandwidth bound (GEMV operations)                       |
|     - Low compute utilization, high VRAM transfers                   |
+-----------------------------------------------------------------------+

The Root Causes of High Latency

1. KV Cache Memory Fragmentation: Standard PyTorch runtimes pre-allocate static contiguous memory blocks for the Key-Value (KV) cache based on maximum sequence lengths. This leads to 60% to 80% memory waste due to internal and external fragmentation, severely limiting batch sizes.
2. Naive Dynamic Batching: Traditional serving frames batch incoming requests at the iteration level. If one request in a batch generates 500 tokens while another generates 10, the short request waits for the long one to complete, wasting GPU cycles (padding tokens).
3. Unoptimized Kernels: Default PyTorch execution routes layer operations through multiple non-fused CUDA kernels, causing excessive memory traffic between GPU High-Bandwidth Memory (HBM) and SRAM.

2. Infrastructure Blueprint: AWS EKS for Distributed GPU Workloads

A high-performance inference engine requires an underlying Kubernetes cluster tuned for low-latency inter-GPU communication and rapid elasticity.

                           +-----------------------------------+
                           |        AWS EKS Control Plane      |
                           +-----------------------------------+
                                             |
                   +-------------------------+-------------------------+
                   |                                                   |
                   v                                                   v
      +-------------------------+                         +-------------------------+
      | Karpenter NodePool (g5) |                         | Karpenter NodePool (p4d)|
      |  (NVIDIA A10G - 24GB)   |                         |  (NVIDIA A100 - 40GB)   |
      +-------------------------+                         +-------------------------+
      |  AWS VPC CNI (Prefix)   |                         |   EFA Driver + NVLink   |
      | NVIDIA GPU Operator     |                         | NVIDIA GPU Operator     |
      +-------------------------+                         +-------------------------+
                   |                                                   |
                   v                                                   v
        +---------------------+                             +---------------------+
        |  vLLM Pod (Llama 8B)|                             | TRT-LLM Pod (70B)   |
        +---------------------+                             +---------------------+

GPU Instance Selection Matrix

For optimal cost-performance efficiency, we categorized workloads onto target AWS EC2 instance types:

| Instance Type | GPUs / VRAM | Interconnect | Primary Use Case | | :--- | :--- | :--- | :--- | | g5.12xlarge | 4x A10G (96GB total) | PCIe Gen4 | Small/Medium Models (Llama 3 8B, Qwen 14B) | | p4d.24xlarge | 8x A100 (320GB total) | NVLink (600 GB/s) + NVSwitch | Large Models (Llama 3 70B FP16, Mixtral) | | p5.48xlarge | 8x H100 (640GB total) | NVLink (900 GB/s) + NVSwitch | Ultra-low latency enterprise workloads (FP8) |

Karpenter NodePool for Provisioning Accelerated Nodes

We leverage Karpenter (v1.0+) to scale GPU nodes dynamically. The following manifest configures a NodePool targeting high-memory g5 and p4d instances with pre-loaded NVIDIA drivers and NVMe storage initialization.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference-nodepool
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: category
          operator: In
          values: ["g", "p"]
        - key: instance-family
          operator: In
          values: ["g5", "p4d"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-node-class
      limits:
        nvidia.com/gpu: 64
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-node-class
spec:
  amiFamily: AL2
  amiSelectorTerms:
    - alias: al2@latest
  role: KarpenterNodeRole-EKS-Cluster
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ecstaticloud-eks-cluster
  securityGroupSelectorTerms:
    - tags:
        kubernetes.io/cluster/ecstaticloud-eks-cluster: owned
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 200Gi
        volumeType: gp3
        iops: 3000
        throughput: 125

3. Deep Dive 1: High-Throughput Serving with vLLM

vLLM fundamentally solves memory fragmentation via PagedAttention. By allocating KV cache memory in physical pages (similar to virtual memory in operating systems), vLLM reduces memory waste to under 1%, allowing dramatically larger batch sizes.

Deploying vLLM on EKS (Llama 3 70B across 4 GPUs)

The following manifest deploys a distributed vLLM instance using Tensor Parallelism (--tensor-parallel-size 4) on a g5.12xlarge or p4d.24xlarge node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-70b
  namespace: llm-serving
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-llama3-70b
  template:
    metadata:
      labels:
        app: vllm-llama3-70b
    spec:
      containers:
      - name: vllm-engine
        image: vllm/vllm-openai:v0.6.2
        args:
        - "--model"
        - "meta-llama/Meta-Llama-3-70B-Instruct"
        - "--tensor-parallel-size"
        - "4"
        - "--gpu-memory-utilization"
        - "0.92"
        - "--max-model-len"
        - "8192"
        - "--max-num-batched-tokens"
        - "8192"
        - "--enable-chunked-prefill"
        - "true"
        - "--port"
        - "8000"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        - name: NCCL_DEBUG
          value: "INFO"
        ports:
        - containerPort: 8000
          name: http
        resources:
          limits:
            nvidia.com/gpu: "4"
            memory: "180Gi"
            cpu: "32"
          requests:
            nvidia.com/gpu: "4"
            memory: "120Gi"
            cpu: "16"
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 16Gi

Key Architectural Tuning:

1. --enable-chunked-prefill=true: Prevents large context prefill operations from starving ongoing generation requests, significantly flattening Time-To-First-Token (TTFT) spikes.
2. Shared Memory (/dev/shm): PyTorch Distributed / NCCL requires large shared memory volumes for inter-process communication. Using a RAM-backed emptyDir prevents random worker deadlocks.

4. Deep Dive 2: Ultra-Low Latency with TensorRT-LLM & Triton

For critical user-facing paths requiring absolute minimum latency, NVIDIA TensorRT-LLM outperforms generic frameworks. TensorRT-LLM compiles models into heavily optimized CUDA engines with fused multi-head attention (FMHA), in-flight batching, and native FP8/INT4 quantization.

Engine Compilation Pipeline

Before deploying to EKS, compile the raw PyTorch model into a TensorRT-LLM engine. The snippet below demonstrates building an INT4 AWQ quantized engine for Llama 3 70B:

# 1. Convert Weights to TensorRT-LLM Checkpoint format
python3 /app/TensorRT-LLM/examples/llama/convert_checkpoint.py \
    --model_dir ./Meta-Llama-3-70B-Instruct \
    --output_dir ./ttrt_ckpt/llama3_70b_tp4 \
    --dtype float16 \
    --tp_size 4 \
    --use_weight_only \
    --plugin_weight_only_quant_type int4_awq

# 2. Build the optimized TensorRT Engine
trtllm-build \
    --checkpoint_dir ./ttrt_ckpt/llama3_70b_tp4 \
    --output_dir ./engines/llama3_70b_tp4 \
    --gemm_plugin float16 \
    --gpt_attention_plugin float16 \
    --tokens_per_block 64 \
    --paged_kv_cache enable \
    --remove_input_padding enable \
    --use_custom_all_reduce enable \
    --max_batch_size 128 \
    --max_input_len 4096 \
    --max_output_len 2048

Triton Deployment Manifest with TensorRT-LLM Backend

Deploy the compiled engine stored in Amazon S3 to EKS using Triton Inference Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-trtllm-llama70b
  namespace: llm-serving
spec:
  replicas: 2
  selector:
    matchLabels:
      app: triton-trtllm
  template:
    metadata:
      labels:
        app: triton-trtllm
    spec:
      initContainers:
      - name: download-engine
        image: amazon/aws-cli:latest
        command: ["aws", "s3", "sync", "s3://ecstaticloud-model-store/llama3_70b_tp4/", "/model-store/llama3_70b/1/"]
        volumeMounts:
        - mountPath: /model-store
          name: model-volume
      containers:
      - name: triton-server
        image: nvcr.io/nvidia/tritonserver:24.08-trtllm-python-py3
        command: ["tritonserver"]
        args:
        - "--model-repository=/model-store"
        - "--disable-auto-complete-config"
        - "--backend-config=python,shm-region-prefix-name=prefix_"
        ports:
        - containerPort: 8000
          name: http
        - containerPort: 8001
          name: grpc
        - containerPort: 8002
          name: metrics
        resources:
          limits:
            nvidia.com/gpu: "4"
            memory: "200Gi"
          requests:
            nvidia.com/gpu: "4"
            memory: "150Gi"
        volumeMounts:
        - mountPath: /model-store
          name: model-volume
        - mountPath: /dev/shm
          name: dshm
      volumes:
      - name: model-volume
        emptyDir: {}
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 32Gi

5. Benchmarking & Real-World Performance Results

We executed load testing using Locust against three infrastructure patterns serving Llama 3 70B on p4d.24xlarge (8x A100 40GB) instances under identical concurrency workloads (100 concurrent clients, dynamic prompt/completion length: 512 input / 256 output tokens).

Performance Metrics Comparison

| Architecture Pattern | TTFT (P99) | Inter-Token Latency (ITL) | Max Throughput | Cost per 1M Tokens | | :--- | :--- | :--- | :--- | :--- | | Baseline: Hugging Face TGI (FP16) | 1,450 ms | 42.5 ms/tok | 210 tok/sec | $4.85 | | vLLM (PagedAttention + Prefill Chunking) | 480 ms | 18.2 ms/tok | 680 tok/sec | $2.10 | | TensorRT-LLM (INT4 AWQ + Triton) | 210 ms | 11.4 ms/tok | 1,150 tok/sec | $1.18 |

Latency Profiles (Lower is Better)
-----------------------------------------------------------------------------
TGI Baseline   : [==================================================] 1450ms
vLLM           : [=================>] 480ms (-66.8%)
TRT-LLM (AWQ)  : [========>] 210ms (-85.5%)
-----------------------------------------------------------------------------

Architectural Key Takeaways

• vLLM proved optimal for dynamic environments where models change frequently. It yielded a 3x increase in total throughput over the baseline with minimal configuration overhead.
• TensorRT-LLM + Triton achieved absolute performance dominance, cutting Time-To-First-Token (TTFT) by 85% and cost-per-million-tokens by 75%, making it the premier choice for production-locked models.

6. Target-Driven Autoscaling with KEDA

Standard Kubernetes Horizontal Pod Autoscaler (HPA) targets CPU or static Memory utilization—metrics that are completely ineffective for GPUs (where VRAM remains 100% allocated regardless of request traffic).

We implemented Kubernetes Event-driven Autoscaling (KEDA) watching real-time queue metrics exposed by vLLM/Triton to Prometheus.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-autoscaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-70b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{namespace="llm-serving"})
      threshold: '10'

How the Scaling Loop Functions:

1. When traffic surges, vLLM queues pending requests, raising vllm_num_requests_waiting.
2. KEDA triggers the deployment of new vLLM Pods when waiting requests exceed 10.
3. Karpenter intercepts the unallocatable Pods (requesting nvidia.com/gpu: 4) and provisions p4d.24xlarge nodes in under 90 seconds using warm-pool custom AMIs.

Conclusion & Architectural Recommendation

Optimizing LLM inference latency on AWS EKS requires shifting focus from raw compute allocation to memory layout and execution efficiency.

Decision Framework:

• Choose vLLM if: You require rapid model deployments, native OpenAI API compatibility, high flexibility, and multi-tenant dynamic models with minimal operational build pipelines.
• Choose TensorRT-LLM + Triton if: You run static models in high-volume production, require ultra-low latency (TTFT < 250ms), and have the engineering bandwidth to manage dynamic engine build pipelines.

By combining EKS, Karpenter, vLLM, and TensorRT-LLM, our infrastructure team transformed a fragile, expensive deployment into an elastic, low-latency inferencing powerhouse capable of processing millions of daily requests at a fraction of the cost.