Sharing a Fabric with Collective Communication: Two Storage Penalties in Deep Learning Training

How shared network fabric between NCCL/RCCL collectives and parallel filesystem I/O imposes heavy-tailed DataLoader stalls and traffic-class contention, and how node-local NVMe staging via DYAD eliminates both.

Chen Wang; Wenzhao Wu; Hyojin Kim; Jae-Sung Yeom· 2026· DOI 10.48550/arXiv.2609.06506

The core problem

Distributed deep learning (DL) training on high-performance computing (HPC) systems increasingly relies on a single network fabric to carry both collective communication (e.g., NCCL/RCCL all-reduce) and parallel-filesystem I/O (e.g., Lustre, VAST). This architectural convergence is efficient in terms of cabling and switch port utilization, but it introduces contention that can degrade training throughput. The authors study a real Graph Neural Network (GNN) training workload on a Slingshot-11 system to quantify the costs of this sharing. They identify two distinct penalties: (1) heavy-tailed DataLoader stalls caused by storage I/O traversing the shared fabric, and (2) traffic-class contention on collective communication when storage and collectives share the same traffic class. The paper's central claim is that both penalties share a common root cause—storage I/O traversing the shared fabric—and that node-local NVMe staging via DYAD (a Flux Framework project) eliminates both by keeping storage I/O off that path. The work is motivated by the observation that typical DataLoader wait times are small (15 ms at steady state) but spike to multiple seconds in a significant fraction of iter

Innovation

The experiments reveal two distinct storage penalties. First, heavy-tailed DataLoader stalls: the typical DataLoader wait is just 15 ms at steady state, yet spikes to multiple seconds in 28% of Lustre iterations and 12% of VAST iterations. This heavy-tailed behavior causes significant stragglers in the training loop. Second, traffic-class contention on collective communication: in an isolated benchmark, Lustre I/O stalls the all-reduce by up to 145x. This contention occurs only when storage and collective communication share the same traffic class. The two costs arise from different mechanisms: I/O stall latency affects any storage path that traverses the shared fabric, whereas all-reduce network contention occurs only when storage and collective communication share the same traffic class. DYAD eliminates both effects by keeping storage I/O off the shared fabric. Across a full training epoch, DYAD achieves a 7.4x speedup over direct Lustre reads and a 1.06x speedup over VAST. By the second epoch, once the local cache is fully warmed, DataLoader stalls are eliminated entirely, allowing DYAD to reach a 1.31x speedup over VAST. These results are summarized in the following table: |

Distributed deep learning (DL) training on high-performance computing (HPC) systems increasingly relies on a single network fabric to carry both collective communication (e.g., NCCL/RCCL all-reduce) and parallel-filesystem I/O (e.g., Lustre, VAST). This architectural convergence is efficient in terms of cabling and switch port utilization, but it introduces contention that can degrade training throughput. The authors study a real Graph Neural Network (GNN) training workload on a Slingshot-11 system to quantify the costs of this sharing. They identify two distinct penalties: (1) heavy-tailed DataLoader stalls caused by storage I/O traversing the shared fabric, and (2) traffic-class contention on collective communication when storage and collectives share the same traffic class. The paper's central claim is that both penalties share a common root cause—storage I/O traversing the shared fabric—and that node-local NVMe staging via DYAD (a Flux Framework project) eliminates both by keeping storage I/O off that path. The work is motivated by the observation that typical DataLoader wait times are small (15 ms at steady state) but spike to multiple seconds in a significant fraction of iterations (28% for Lustre, 12% for VAST), and that Lustre I/O can stall all-reduce by up to 145x in an isolated benchmark.

The authors conduct experiments on a Slingshot-11 system using a real GNN training workload. They compare three storage configurations: direct Lustre reads, direct VAST reads, and DYAD-based node-local NVMe staging. DYAD is a data-aware staging library that intercepts storage I/O and caches data on node-local NVMe, thereby avoiding the shared fabric for storage traffic. The methodology includes:

Why it matters

The findings highlight a fundamental tension in HPC system design: sharing a network fabric between collective communication and storage I/O can lead to severe performance penalties for DL training. The two penalties—heavy-tailed DataLoader stalls and traffic-class contention—have different mechanisms but a common root cause: storage I/O traversing the shared fabric. This suggests that isolating storage traffic from collective communication is key to achieving predictable and high-performance training. DYAD's approach of node-local NVMe staging effectively decouples storage I/O from the shared fabric. By caching data on local NVMe, DYAD not only reduces latency but also eliminates the variability that causes heavy-tailed stalls. The 7.4x speedup over Lustre and 1.31x over VAST (after cache warm-up) demonstrate the potential of this approach. However, the effectiveness of DYAD depends on the availability of node-local NVMe and sufficient cache capacity to hold the working set. For workloads with very large datasets that exceed local cache capacity, the benefits may be reduced. The paper's taxonomy candidates include Architecture, Cybersecurity, Network, and Cryptography. While the work is primarily about network architecture and storage systems, it also has implications for cybersecurity (e.g., traffic isolation) and cryptography (e.g., secure collective communication). Future work could explore adaptive traffic-class assignment, dynamic cache management, and integration with other staging solutions. In conclusion, the paper provides a clear demonstration of the costs of sharing a fabric and a practical solution via DYAD. The code is publicly available at https://github.com/flux-framework/dyad, enabling reproducibility and further research.

Who should read this

CS practitioners and researchers

Opening member content…