Terraform for MLOps: Provision ML Infra as Code
Machine learning teams love notebooks and models, but they often treat infrastructure as an afterthought — a pile of manually clicked cloud consoles that nobody can reproduce six months later. If you already know Terraform, you're holding one of the most valuable skills for modern MLOps. The same declarative, version-controlled workflow you use for web services works beautifully for training clusters, feature stores, model registries, and inference endpoints.
This guide shows cloud and DevOps engineers how to apply familiar Infrastructure as Code (IaC) practices to build reproducible, auditable ML environments.
Why Terraform Fits MLOps So Well
ML infrastructure has the same problems Terraform was built to solve: sprawling resources, environment drift, and painful onboarding. But ML adds unique pressure points.
Reproducibility is a first-class requirement
In ML, reproducibility isn't just nice to have — it's often a compliance and debugging necessity. When a model behaves differently in staging versus production, the infrastructure (GPU type, CUDA drivers, data access paths, network config) is frequently the culprit. Declaring that infrastructure in code means you can recreate an identical environment on demand and diff changes over time.
ML resources are expensive and ephemeral
GPU instances cost real money. Many ML workloads — hyperparameter sweeps, batch training, periodic retraining — are bursty. Terraform lets you spin up expensive compute on demand and tear it down cleanly, so you're not paying for idle A100s over a weekend.
Multiple personas touch the same stack
Data scientists, ML engineers, and platform teams all depend on shared infrastructure. Codifying it in a reviewed pull request creates a single source of truth instead of tribal knowledge.
Core Building Blocks to Provision
A typical MLOps platform provisioned with Terraform includes several layers. Structure your modules around these concerns:
1. Compute for training
Managed training services — Amazon SageMaker, Google Vertex AI, or Azure Machine Learning — expose Terraform resources or can be wrapped with their provider APIs. For self-managed workloads, you'll provision GPU node pools on Kubernetes (EKS, GKE, or AKS) using the respective provider modules, then let a scheduler like Kubeflow or Ray handle job placement.
2. Storage and feature stores
Model artifacts, datasets, and features need durable, versioned storage. Provision object storage buckets (S3, GCS, Azure Blob) with lifecycle policies, plus managed offline/online feature stores. Set bucket versioning and encryption in code so every environment inherits the same guarantees.
3. Model registry and serving
Whether you use a managed registry or MLflow on your own infrastructure, the backing database, artifact store, and serving endpoints all belong in Terraform. Inference endpoints — with autoscaling policies and instance types — are prime candidates for parameterized modules.
4. Networking, identity, and observability
VPCs, private endpoints for data access, IAM roles scoped to least privilege, and monitoring stacks round out the platform. ML jobs frequently need tight, auditable data access — encoding those IAM policies in code prevents scope creep.
A Practical Module Layout
Resist the urge to write one giant configuration. Split responsibilities into composable modules and separate state per environment.
A clean layout looks like this:
modules/ — reusable components such as gpu-node-pool, feature-store, model-endpoint, and training-bucket.
environments/ — thin root configs for dev, staging, and prod that call the modules with environment-specific variables.
Here's a minimal example of consuming a GPU node pool module:
module "training_pool" {
source = "../../modules/gpu-node-pool"
instance_type = "g5.2xlarge"
min_nodes = 0
max_nodes = 8
spot_enabled = true
environment = "staging"
}
Setting min_nodes to zero lets the cluster scale to nothing when no training jobs are queued — a simple but powerful cost lever.
State, Secrets, and Safety
ML infrastructure touches sensitive data, so treat state and secrets carefully.
Use remote state with locking
Store state in a remote backend (S3 with DynamoDB locking, GCS, or Terraform Cloud) so concurrent applies don't corrupt your platform. Never commit state to Git — it can contain resource metadata and, occasionally, secrets.
Keep credentials out of code
Data access keys, registry tokens, and API secrets belong in a secrets manager, referenced at runtime — not hardcoded in .tf files. Use dynamic provider credentials or workload identity federation where possible so short-lived tokens replace long-lived keys.
Plan before you apply
Run terraform plan in CI on every pull request and require review. Destroying or resizing a training cluster mid-job can waste hours of compute, so make destructive changes visible before they land.
Wiring Terraform Into ML Pipelines
The real payoff comes when infrastructure provisioning becomes part of your ML workflow rather than a separate manual step.
A common pattern: your CI/CD pipeline runs terraform apply to stand up a training environment, triggers the training job, promotes the resulting model to the registry, then updates the serving endpoint — all in code. Some teams pair Terraform (for durable, long-lived infrastructure) with a workflow orchestrator like Airflow, Kubeflow Pipelines, or Prefect (for job-level orchestration). Keep the boundary clear: Terraform manages infrastructure lifecycle, orchestrators manage job lifecycle.
For ephemeral experiment environments, consider using Terraform workspaces or dedicated short-lived state so a data scientist can request an isolated stack, run experiments, and destroy it without touching shared resources.
Common Pitfalls to Avoid
Mixing job state into infra state. Don't try to manage individual training runs as Terraform resources — they're too dynamic. Provision the platform; let purpose-built tools run the jobs.
Ignoring quota and capacity limits. GPU capacity is constrained in many regions. Bake in graceful handling and document required quota increases rather than discovering limits during a critical retrain.
Forgetting drift detection. Data teams sometimes tweak resources by hand during incidents. Schedule periodic terraform plan runs to catch drift before it becomes a mystery.
One-size-fits-all instance types. Parameterize instance types and accelerator counts so training and inference can use appropriately sized hardware instead of overpaying for both.
Skills That Transfer — and the New Ones You'll Add
If you already write Terraform for traditional workloads, you're most of the way there. The new muscles you'll build are ML-specific: understanding GPU instance families and drivers, feature store architecture, model registry patterns, and the cost dynamics of bursty training workloads. These are exactly the platform-engineering skills that make DevOps professionals so valuable on AI teams in 2026.
Start small: pick one manual piece of your ML stack — perhaps the artifact bucket or the inference endpoint — and codify it. Then expand module by module until your entire platform can be rebuilt from a clean clone of your repository.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
