The Filesystem Your Multi-GPU Rig Actually Needs

You install four PCIe Gen5 NVMe drives in a multi-GPU server, bring up eight H100s, launch torchrun, and notice an irritating pattern: GPU utilization fluctuates between 40% and 95%.

A quick look at nvidia-smi suggests that the GPUs are waiting for data. Profiling shows that CPU time is being spent not only on preprocessing, but also on filesystem activity: metadata operations, lock contention, and page-cache writeback during checkpointing.

In a modern deep-learning system, the filesystem is not merely a passive storage layer. It is part of the data pipeline, and its concurrency characteristics can affect end-to-end training throughput.

This article looks at two common local-storage workloads:

1. Parallel data ingestion: dozens of PyTorch DataLoader workers issue concurrent reads and metadata requests.
2. Large checkpoint writes: training processes serialize model weights, optimizer states, and tensor data into files that can range from 10 GB to more than 200 GB per rank.

For these workloads, XFS is often a strong choice for local NVMe scratch storage. Its allocation-group architecture, extent-based allocation, and delayed allocation can make it scale well under concurrent access. That does not mean XFS wins every workload or that filesystem choice alone fixes an input pipeline; the data layout, access pattern, kernel, mount options, and hardware all matter.

At a glance

1. The concurrency problem

PyTorch's torch.utils.data.DataLoader can run multiple worker processes using num_workers. On a multi-GPU node, the total number of workers can become substantial. For example, eight training processes with 16 workers each can create 128 independent workers.

Those workers may concurrently:

• Open and close files.
• Read file and directory metadata.
• Resolve paths and look up inodes.
• Issue data reads.
• Update access times, depending on mount options.

The resulting workload is not simply a stream of large reads. It can also be a high-rate stream of small metadata operations.

A simplified view of the I/O path

PyTorch DataLoader workers Worker 1 Worker 2 Worker 3 ... Worker N \ | | / \ | | / +-------+---------+--------------+ | v Linux Virtual Filesystem | +----------+----------+ | | v v ext4 XFS Block groups and Allocation groups journal coordination with per-AG metadata 

This is a conceptual diagram, not a claim that every operation is serialized by one global lock in ext4 or that XFS eliminates locking. Both filesystems use sophisticated concurrency mechanisms. The important distinction is that XFS can distribute many allocation and metadata operations across allocation groups.

2. Why ext4 can become a bottleneck

ext4 is a mature filesystem. It performs very well across a wide range of workloads. Under highly concurrent metadata workloads, however, contention can arise around shared filesystem structures and journal activity.

Two areas are worth understanding:

• Journal activity: metadata updates are coordinated through JBD2 transactions. A workload that creates, renames, or updates large numbers of files can generate substantial journal traffic.
• Allocation and metadata coordination: concurrent inode and block allocation can contend on shared structures, depending on the access pattern and filesystem layout.

The practical point is not that ext4 has a single global mutex. It does not. Rather, the cost of coordinating metadata operations can become visible when many processes issue small, concurrent filesystem requests.

3. How XFS uses allocation groups

XFS was designed for large filesystems and high-throughput workloads. One of its central design features is the allocation group (AG).

An XFS filesystem is divided into allocation groups. Each AG maintains its own allocation metadata, including structures used to track free space and inodes. These structures are commonly implemented using B+ trees.

An allocation group maintains, among other things:

• Free-space information, indexed by block location and extent size.
• Inode allocation information.
• AG-local metadata and coordination structures.

When multiple workers perform allocation or metadata operations, XFS can distribute work across different AGs rather than forcing all such activity through one shared allocation structure.

This does not make XFS lock-free. It means its architecture provides multiple allocation domains, which can reduce contention for workloads that exercise them effectively.

Practical implication: XFS is worth benchmarking when a node has many concurrent workers accessing a local NVMe filesystem, particularly when metadata activity is a measurable part of the bottleneck.

4. Checkpointing

Saving a modern deep-learning checkpoint can involve writing tens or hundreds of gigabytes of tensor data. Examples include:

• PyTorch state dictionaries.
• Optimizer state.
• Distributed training or ZeRO partitions.
• BF16 or FP8 tensor dumps.

The goal is to complete checkpoint writes quickly and predictably, minimizing the time training is interrupted or competing with storage activity.

Extent-based allocation

XFS uses extents to describe ranges of filesystem blocks. Instead of representing a file as a long list of individual block pointers, an extent can describe a contiguous range.

For large sequential writes, this can keep file metadata compact and help the filesystem allocate large runs of blocks. The actual extent layout still depends on free space, concurrent activity, write patterns, and filesystem state; contiguity is not guaranteed.

Delayed allocation

With buffered I/O, XFS can delay choosing the physical blocks for newly written data until writeback. This gives the allocator more information about the write and can improve the chance of allocating larger, more contiguous extents.

The allocsize mount option controls the default speculative preallocation size for buffered writes. It can influence allocation behavior, but it is not a guarantee that every write will be allocated in fixed-size chunks or that a checkpoint will consist of only a few extents.

What about Btrfs and ZFS?

Btrfs and ZFS provide valuable features such as snapshots, checksumming, and copy-on-write semantics. Those features can have costs for write-heavy workloads.

For example, when data blocks are shared through reflinks or snapshots, modifying them can require copy-on-write allocation and associated metadata updates. Fragmentation and write amplification can become concerns in some workloads.

However, it is too broad to say that every write on Btrfs or ZFS necessarily causes severe fragmentation or that every transaction causes problematic latency spikes. The outcome depends on workload, record and block sizes, pool or filesystem configuration, and the state of the storage.

There is also an important XFS detail: XFS supports reflink-based copy-on-write when reflink is enabled and extents are shared. Ordinary, unshared file writes are not automatically copy-on-write. If the objective is a scratch filesystem with no need for reflink cloning, decide whether reflink support is useful before formatting the volume.

5. Filesystem comparison for deep-learning workloads

The table below is a qualitative engineering guide, not a universal benchmark result. Actual performance depends on kernel version, filesystem configuration, device topology, dataset layout, and concurrency.

Do not interpret this table as proof that XFS is always faster. Benchmark the access pattern that your training job actually uses.

6. The small-file caveat

A common argument against XFS is that ext4 can perform very well with directories containing large numbers of small files. Consider a dataset containing:

• 1.2 million JPEG files, each around 20 KB.
• Several million WAV clips, each around 15 KB.

In this scenario, directory lookup, inode access, path resolution, and file-open overhead can become a significant part of the input pipeline.

But the more important design question is whether millions of individual files should be served directly to a multi-GPU training job in the first place.

Every sample fetched through a loose-file layout may involve operations such as open(), metadata lookup, read(), and close(). At scale, those operations create kernel transitions and metadata-cache pressure that can limit throughput regardless of the filesystem.

Anti-pattern

class LooseAudioDataset(torch.utils.data.Dataset): def __getitem__(self, idx): path = self.file_paths[idx] waveform, sample_rate = torchaudio.load(path) return waveform 

Better pattern

import webdataset as wds dataset = ( wds.WebDataset( "/mnt/fast_data/shards/train-{000000..000499}.tar" ) .decode("torch") .to_tuple("wav", "cls") ) 

Instead of opening a separate file for every sample, the pipeline reads samples from larger sequential containers.

Common options include:

• WebDataset / TAR shards
• HDF5
• Zarr
• Memory-mapped Arrow
• LMDB

A shard size in the range of hundreds of megabytes to a few gigabytes can be a useful starting point, but the right size depends on the number of workers, randomization strategy, storage capacity, and recovery requirements.

Once data is packed into shards, the pipeline can perform larger, more sequential reads. That reduces per-sample metadata overhead and makes it easier to use the bandwidth of fast NVMe storage.

7. Production tuning guide for XFS on NVMe

The following commands are starting points for a dedicated local scratch volume. They are not universal settings. Test them on the target kernel, NVMe device, and workload before deploying them on a production training node.

7.1 Format with an appropriate allocation-group count

For example:

sudo mkfs.xfs -f \ -d agcount=32 \ -m reflink=1,crc=1 \ -b size=4096 \ /dev/nvme0n1 

agcount=32 is an example, not a rule. More allocation groups do not automatically mean more performance. The appropriate count depends on filesystem size, CPU concurrency, and workload. Start with the mkfs.xfs defaults unless you have benchmark evidence that an explicit AG count helps.

The example enables reflink support. If this is a scratch volume and you do not need reflink cloning, consider whether enabling it is necessary. Reflink support does not make every ordinary write CoW, but writes to shared extents do use CoW semantics.

7.2 Mount options

An example /etc/fstab entry:

UUID=xxxx-xxxx-xxxx-xxxx /mnt/fast_data xfs noatime,nodiratime,logbufs=8,logbsize=256k,allocsize=64M,inode64 0 0 

Review each option before applying it:

Do not copy mount options blindly. In particular, options such as logbufs, logbsize, and allocsize should be justified by measurements rather than treated as mandatory tuning.

7.3 Kernel and block-device tuning

NVMe devices use multiple hardware queues. On many modern systems, the none I/O scheduler is a sensible baseline:

# Inspect the active scheduler first cat /sys/block/nvme0n1/queue/scheduler # Select the none scheduler echo none | sudo tee /sys/block/nvme0n1/queue/scheduler 

For sequential shard reads, you can test a larger read-ahead value:

# Example: 4096 sectors, typically 2 MiB with 512-byte sectors sudo blockdev --setra 4096 /dev/nvme0n1 

You can also experiment with dirty-page thresholds:

sudo sysctl -w vm.dirty_background_ratio=5 sudo sysctl -w vm.dirty_ratio=10 

These settings affect system-wide writeback behavior. They can change when dirty pages are flushed and when processes are throttled; they do not guarantee that checkpointing will become faster. Test them under realistic concurrent training and checkpoint workloads, and make any persistent configuration changes deliberately.

8. Diagnosing storage bottlenecks

Before changing filesystems or kernel parameters, establish what the storage path is actually doing.

8.1 Inspect checkpoint extent layout

Use xfs_bmap to inspect the extent map of a checkpoint:

xfs_bmap -v /mnt/fast_data/checkpoints/model_epoch_10.pt 

A large number of extents can indicate fragmentation, but extent count alone does not establish that fragmentation is hurting performance. Compare the layout with sequential read and write measurements.

8.2 Monitor device utilization and latency

iostat -xz 1 /dev/nvme0n1 

Pay attention to:

• rMB/s and wMB/s: observed throughput.
• await: average I/O wait time.
• aqu-sz: average queue size.
• %util: device utilization as reported by iostat.

A device showing high utilization and low throughput may be handling many small or latency-sensitive requests rather than large sequential transfers. It can also indicate queueing, CPU limitations, or another bottleneck. Compare the results with the device's specifications and with a controlled benchmark; %util alone is not a reliable measure of NVMe saturation.

8.3 Profile CPU-side filesystem activity

Use perf to identify where CPU time is going:

sudo perf top 

You can also inspect XFS-related symbols and lock activity where available in your kernel's profiling environment. Symbol names and tracepoint availability vary by kernel, so first inspect the events exposed by the running system rather than assuming a particular event name exists.

The goal is to determine whether the workers are spending meaningful time in filesystem metadata paths, block I/O, decompression, Python overhead, or data transformations.

9. Practical recommendations

For a multi-GPU training node using local NVMe scratch storage:

1. Benchmark XFS and ext4 with the real workload. Include both concurrent sample reads and large checkpoint writes.
2. Prefer sharded datasets over millions of loose files. This often produces a larger improvement than changing filesystems.
3. Use XFS when its allocation-group and large-file behavior fit the workload. Do not assume it wins every small-file or metadata benchmark.
4. Treat mount and kernel parameters as experiments. Change one variable at a time and measure throughput, latency, CPU utilization, and training-step time.
5. Measure the entire pipeline. GPU utilization is an outcome, not a diagnosis. Confirm that storage—not decoding, augmentation, Python, networking, or CPU scheduling—is the limiting factor.

Conclusion

For large local NVMe volumes serving concurrent deep-learning workloads, XFS is a strong filesystem candidate. Its allocation groups can help distribute metadata and allocation work, while extent-based allocation and delayed allocation are useful for large buffered writes.

The biggest gains, however, usually come from treating storage as part of the training architecture. Shard datasets, measure metadata pressure, profile checkpoint behavior, and tune the filesystem against the actual access pattern.

A well-designed input pipeline on a properly benchmarked filesystem is more valuable than a collection of aggressive mount flags. XFS is a good place to start—not a substitute for measurement.