At 02:14 UTC on a Tuesday in March, an all-reduce step on a 64-GPU training job started taking 4.2 seconds instead of 0.6. The checkpoint was due in nine hours. The on-call engineer pulled up the fabric utilization dashboard. Average across the leaf-to-spine fabric: 23%. No alarms. No drops. PFC counters at zero. Every link green.
She switched to the per-link view. Two of the eight spine uplinks were sitting at 96.4% and 94.1%. The other six spines, on the same fabric, were under 8%.
This is what hash polarization looks like when the workload changes underneath a network that was designed for something else. The fabric was not broken. It was doing exactly what it had been told to do. Hash a 5-tuple. Pick a path. Forward. The problem was that across 64 GPUs running ring- based all-reduce, only sixteen distinct conversations existed at any one moment, and the ECMP hash on this particular ASIC family had a known weakness on RoCEv2 traffic where the UDP destination port is constant. Every conversation in the ring landed on one of two spines. The other six might as well have been unplugged.