S3 Data Lakes for AI Training: Layout Best Practices
If you already run infrastructure on AWS, you know that a poorly organized S3 bucket becomes a tax you pay forever — in slow jobs, surprise bills, and brittle pipelines. When that bucket feeds machine learning, the stakes get higher. Training reads the same data thousands of times across epochs and experiments, so every inefficiency in layout, format, or partitioning gets multiplied.
This guide walks through how to structure a S3 data lake for AI training so that Amazon SageMaker reads it fast and cheap. The focus is practical: bucket design, prefix and partition strategy, file formats, and the storage decisions that quietly control your throughput.
Start With a Bucket Layout That Reflects the ML Lifecycle
The most common mistake is treating an ML bucket like a dumping ground. Instead, model your prefixes around the stages data actually moves through. A durable layout looks like this:
- raw/ — immutable landing zone for ingested source data, never modified.
- processed/ — cleaned, validated, and typed data ready for feature work.
- curated/ or training-ready/ — the exact datasets your training jobs consume.
- artifacts/ — model checkpoints, metrics, and outputs written back by SageMaker.
Keep raw data separate from training-ready data so you can reprocess without re-ingesting, and so lifecycle policies can treat each zone differently. Enable versioning on the training-ready prefix so a dataset used for a specific run is reproducible even after you overwrite it.
One bucket or many?
For most teams, a single bucket with clear top-level prefixes is easier to govern than a sprawl of buckets. Use separate buckets only when you need distinct regions, hard billing separation, or different compliance boundaries. Remember that S3 scales per prefix, not per bucket, so splitting buckets for performance is unnecessary.
Partition for the Queries and Reads You Actually Make
Partitioning means encoding filter dimensions into the object key path so that engines and training loaders skip data they do not need. A typical scheme:
curated/dataset=images/version=2026-03-01/split=train/class=cat/part-0001.parquet
Good partitions share a few traits:
- Align with how you filter. If you always train on a single split at a time, make
split=train|val|testa partition so you never scan the others. - Avoid tiny partitions. Thousands of directories holding a few KB each destroy throughput because each object is a separate GET. Aim for file sizes in the 128 MB to 512 MB range for columnar data.
- Version explicitly. A
version=or dated partition lets you pin a run to an exact snapshot and roll experiments forward without mutating history.
Be careful not to over-partition. A partition on a high-cardinality field like user ID creates millions of near-empty prefixes and makes listing painfully slow. Partition on low-cardinality dimensions you filter on; keep everything else inside the files.
Choose Formats Based on the Access Pattern
Format choice is where most performance is won or lost. Match the format to the data type and how the training loop consumes it.
Tabular and structured data: Parquet
Apache Parquet is the default for structured training data. It is columnar, compressed, and lets engines read only the columns a feature set needs. Combined with partition pruning, Parquet dramatically cuts bytes scanned versus CSV or JSON. Use Snappy or Zstd compression and target the 128–512 MB file size band so readers get large, sequential requests instead of many small ones.
Images, audio, and unstructured samples: shard, don't scatter
Storing millions of individual image files is the classic anti-pattern. Each file is a separate S3 GET, and the per-request overhead dominates, starving your GPUs. Instead, pack samples into sharded archives — the WebDataset tar format or TFRecord are both well supported. A few thousand shards of a few hundred MB each stream far faster than millions of loose objects, and they play nicely with sequential, high-throughput reads.
Large tables that change: open table formats
If your training data is continuously updated and you need schema evolution, time travel, or upserts, consider Apache Iceberg on top of Parquet. It gives you snapshot isolation so a training job reads a consistent view even while new data lands.
Match SageMaker Input Modes to Your Layout
How SageMaker reads from S3 matters as much as how you store it. The main choices in 2026:
- Fast File Mode streams objects on demand and is a strong default for large datasets that don't fit on disk. It works best when files are large and read sequentially — exactly what sharded archives and right-sized Parquet give you.
- File Mode downloads the full dataset to the instance before training starts. Simple, but the download delay hurts for large corpora.
- Pipe Mode streams data as a Unix pipe; largely superseded by Fast File Mode for new work.
- Amazon S3 Express One Zone directory buckets deliver single-digit-millisecond access for latency-sensitive, high-request workloads. If small-file random reads are unavoidable, this tier cuts request latency significantly — at a higher storage cost that you offset with faster, shorter GPU jobs.
Keep your training data and your SageMaker compute in the same AWS Region. Cross-region reads add latency and data transfer charges that add up fast across epochs.
Control Cost Without Slowing Training
A data lake for training generates cost in three places: storage, requests, and transfer. Manage each deliberately.
- Lifecycle policies: move
raw/data to S3 Glacier tiers after it has been processed, while keepingtraining-ready/in S3 Standard for fast, repeated reads. Never put actively-trained data in a retrieval-delayed tier. - Fewer, larger objects: consolidating small files reduces both request charges and latency. This single change often delivers the biggest win.
- Compression: Zstd offers excellent ratios with fast decompression, reducing both storage and bytes transferred to the training instance.
- VPC gateway endpoints for S3: route reads over the endpoint to avoid NAT gateway data processing costs and keep traffic on the AWS backbone.
Make Datasets Reproducible and Discoverable
Training data that can't be reproduced can't be trusted. Register your training-ready datasets in the AWS Glue Data Catalog so schemas and partitions are queryable, and record the exact S3 prefix and version each training job used in its metadata. Pairing an immutable, versioned prefix with a catalog entry means any model can be traced back to the precise bytes that produced it — essential for debugging, audits, and regulatory review.
Add a lightweight manifest per dataset version listing shard paths, record counts, and checksums. Training loaders can read the manifest to shuffle and shard across GPUs deterministically, which improves both reproducibility and multi-node scaling.
A Quick Checklist Before Your Next Training Run
- Files sized 128 MB–512 MB, not millions of tiny objects.
- Parquet for tabular, sharded archives for unstructured data.
- Partitions match your filters and are low-cardinality.
- Compute and data in the same Region, reads over a VPC endpoint.
- Fast File Mode by default; S3 Express One Zone for latency-bound jobs.
- Versioned, cataloged, manifest-backed datasets for reproducibility.
Get these fundamentals right and your GPUs spend their time computing instead of waiting on storage — which is the whole point of paying for them.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
