@thegibson foolishly I stayed in that HPC job for a while longer.

Somehow (despite general lack of networking background) was made responsible for overseeing a wholesale re-wire of the Infiniband Network for expansion of a cluster.

Rats nests of cables... my oversight failed to catch a label with '6' being read as '9' by a colleague. Didn't have a downtime impact. But took literally weeks for someone else to find and correct the performance issues that resulted. Unmanaged IB switches suck.