More info on the NVSwitch: https://www.nextplatform.com/2018/03/27/nvidia-memory-switch-welds-together-massive-virtual-gpu/
Unlike some of the switches that we see in the HPC arena coming out of China, which are derivatives of InfiniBand switching that are licensed from Mellanox Technologies, the NVSwitch ASIC is a homegrown device that has been under development for the past two years, Ian Buck, vice president and general manager of accelerated computing at Nvidia, tells The Next Platform. The NVSwitch ASIC has over 2 billion transistors, about a tenth of the Volta GPU, and it creates a fully connected, non-blocking crossbar switch using the NVLink protocol, which means every port on the GPUs linked to the switch can talk to all of the other ports (in a point to point manner) at full speed.
The switch chip has a total of 18 ports, which provide 50 GB/sec of bandwidth per port and which, if you do the math, use 25 Gb/sec signaling just like the NVLink ports on the Volta GPU accelerators and the IBM Power9 chips. The NVSwitch has an aggregate of 900 GB/sec of switching bandwidth, and a bunch of the switches can be interconnected and cascades to scale the Tesla network in any number of topologies. The ports on the switch aggregate eight lanes of 25 Gb/sec signaling