PFC, ECN, and DC-QCN. Three layers, three timescales, one fabric that doesn’t drop packets
03:47 UTC. A spine in row 2 reported elevated egress queue depth on a single port. Twelve seconds later, every leaf in the fabric was sending PFC PAUSE frames upstream. Sixteen seconds in, NCCL connections across two pods began timing out. The training job had been running cleanly for eleven days.
It lost forty-two iterations before someone correlated the spine counter with the failing flows and pulled the offending NIC out of rotation.