EKS for AI Workloads: Run GPU Jobs on Kubernetes

Running AI training and inference on Amazon EKS gives DevOps teams the best of both worlds: the elasticity of Kubernetes and the raw compute of NVIDIA GPUs. But GPU nodes are expensive and behave differently from standard CPU pools. This guide walks through the specific steps to schedule, isolate, and autoscale EKS GPU workloads so you pay only for the acceleration you actually use.

Why Kubernetes for GPU jobs?

If your organization already runs services on EKS, extending that platform to AI workloads avoids a second silo of tooling. You reuse existing IAM roles, VPC networking, observability stacks, and CI/CD pipelines. Kubernetes also handles the two dominant AI patterns cleanly:

• Batch training jobs that run to completion, benefit from spot capacity, and need to be requeued on interruption.
• Long-running inference services that scale with request volume and demand low-latency GPU access.

The challenge is that GPUs are a scarce, costly resource. A single p5 or g6 node can dwarf the cost of an entire CPU fleet, so scheduling discipline matters more here than anywhere else in your cluster.

Setting up GPU nodes on EKS

Pick the right instance family

Match the instance to the job. For cost-sensitive inference and smaller fine-tuning runs, the G6 and G6e families (NVIDIA L4 and L40S) offer strong price-performance. For large-scale training, the P5 family (NVIDIA H100) and P5e/P5en variants provide high-bandwidth interconnect. Don't over-provision: an L4-backed node is often plenty for serving a quantized model.

Install the NVIDIA device plugin

Kubernetes doesn't see GPUs by default. You need the NVIDIA device plugin, which advertises nvidia.com/gpu as a schedulable resource. The cleanest path in 2026 is the NVIDIA GPU Operator, installed via Helm. It manages the device plugin, the container toolkit, driver compatibility, and DCGM metrics exporters as a single unit.

If you use EKS-optimized accelerated AMIs or Bottlerocket's NVIDIA variant, the driver and container toolkit ship preinstalled, so you can run the device plugin standalone and skip driver management entirely.

Request GPUs in your pod spec

Once the plugin is running, pods request GPUs through resource limits:

resources:
  limits:
    nvidia.com/gpu: 1

GPUs are only expressed as limits, not requests, and they cannot be overcommitted. One pod gets exclusive access to each whole GPU unless you enable time-slicing or MIG (see below).

Keeping GPU nodes for GPU work

The most common mistake is letting CPU pods land on expensive GPU nodes. Prevent this with a combination of taints and node labels.

Taint your GPU node group with something like nvidia.com/gpu=true:NoSchedule so only pods with a matching toleration are admitted. Then add a nodeSelector or node affinity rule targeting a label such as node.kubernetes.io/instance-type or a custom workload=gpu label. This guarantees your GPU fleet stays reserved for accelerated jobs and idle capacity doesn't get wasted on background daemons.

Autoscaling GPU capacity with Karpenter

For 2026 deployments, Karpenter is the recommended autoscaler over the older Cluster Autoscaler. Instead of pre-defining fixed node groups, Karpenter provisions nodes just-in-time based on pending pod requirements, which is ideal for bursty AI workloads.

Define a GPU NodePool

Create a Karpenter NodePool and EC2NodeClass scoped to GPU instance families. Set requirements that allow the g6 and p5 families, and add the appropriate taints so the nodes come up ready for GPU pods only. When a training job is submitted, Karpenter reads the nvidia.com/gpu request, finds a matching instance type, and launches it in roughly a minute.

Scale to zero

The biggest cost win is consolidation and scale-to-zero. Set a short ttlSecondsAfterEmpty or use Karpenter's consolidation policy so GPU nodes terminate as soon as jobs finish. A cluster with no pending GPU pods should hold zero GPU nodes. This turns your idle bill from hundreds of dollars per day to nothing.

Use Spot for training

Batch training tolerates interruption well. Configure your NodePool to prefer Spot capacity with On-Demand fallback. Pair this with checkpointing in your training code so an interrupted job resumes from the last saved state rather than restarting. Savings of 60–70% over On-Demand are common for GPU spot instances.

Squeezing more from each GPU

Whole-GPU allocation is wasteful for lightweight inference. Two techniques help you pack more pods onto each card:

• Time-slicing: The NVIDIA device plugin can advertise a single physical GPU as multiple logical replicas. Pods share the GPU in a round-robin fashion. Good for low-traffic inference and dev environments, but there is no hard memory isolation.
• Multi-Instance GPU (MIG): Available on A100, H100, and newer data-center GPUs, MIG partitions one physical GPU into isolated instances with dedicated memory and compute slices. Use this for production inference where predictable performance matters.

Scheduling batch jobs at scale

Kubernetes' native Job object works for simple runs, but multi-node training needs gang scheduling — all pods start together or none do. For that, add a batch scheduler such as Volcano or Kueue. Kueue in particular has matured into a strong choice for quota management and job queuing, letting teams share a GPU pool fairly without one team starving another.

For distributed training frameworks, the Kubeflow Training Operator provides custom resources like PyTorchJob that manage worker coordination, restarts, and rendezvous automatically.

Observability and cost control

Ship DCGM metrics (GPU utilization, memory, temperature, power) to Prometheus and visualize them in Grafana. Watch GPU utilization closely: a node running at 15% utilization is a signal to consolidate with time-slicing or downsize the instance.

Tag GPU node groups and use Kubecost or AWS Cost Explorer with cost allocation tags to attribute spend per team or model. Because GPU costs dominate AI budgets, even small utilization gains translate into large monthly savings.

A practical rollout checklist

Bring it all together in this order:

• Deploy the NVIDIA GPU Operator via Helm.
• Create a Karpenter NodePool for GPU families with Spot preference and taints.
• Add tolerations, nodeSelectors, and GPU resource limits to your workload manifests.
• Enable MIG or time-slicing based on inference vs. training needs.
• Add Kueue or Volcano for batch queuing and fair sharing.
• Wire up DCGM, Prometheus, and cost tags before you scale up.

With this foundation, DevOps engineers can offer their data science teams on-demand GPU capacity that spins up in minutes, scales to zero when idle, and stays within budget — all on the same EKS platform they already operate.

Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.