Two clusters. Same 256 H100s. Same 70-billion-parameter model. Same NCCL build at the same git hash. One ran InfiniBand HDR end to end, the other ran RoCEv2 over 400G Ethernet, both with the same ConnectX-7 NICs.
On a clean day they finished an iteration within four percent of each other. RoCEv2 ran a little slower. Nobody cared.
Then someone yanked a transceiver during the second day of a long training run.