Two networks. Same switch BOM. Same rack space. Same power envelope. Same 256 H100s.
The first was wired as a two-tier fat-tree, every leaf connected to every spine, oversubscription 1:1, the textbook layout. The second was wired as a four- rail topology, each GPU’s primary NIC anchored to its own dedicated rail- leaf, no cross-rail wiring at the leaf layer.
NCCL all_reduce_perf on the same 70-billion-parameter workload. Same nccl-tests parameters. Same CUDA driver. Same kernel.