When thousands of GPUs train one model, they spend a lot of time swapping data. InfiniBand is the network most big clusters use for that. This page walks through how it works, one picture at a time. No networking background needed.
Ethernet, the network your laptop uses, was built for the internet. Nobody owns the whole internet, so every switch has to figure things out on its own, and losing a packet now and then is treated as normal.
InfiniBand was built for a different world. One owner, one building, one job, which is moving memory from one machine to another as fast as possible. Once you can assume a single owner, you can make very different design choices. The rest of this page is about those choices.
Short for Host Channel Adapter, InfiniBand's name for a network card. It is a small computer of its own. It reads memory, builds packets and handles retries in hardware, so the CPU can stay out of it. In GPU clusters each GPU usually gets its own HCA.
Forwards packets. On the fast path it does exactly one thing. It looks up the destination in a table and sends the packet out the matching port.
Every port on the fabric gets a small number called a LID, short for Local Identifier. Each packet carries the LID it is headed to, a bit like an apartment number on an envelope.
A piece of software running somewhere on the fabric. It hands out every address and fills in every switch's table. More on why it exists in Part 6.
Let's follow a chunk of gradients from a GPU on Server A to a GPU on Server B. Step through it with the buttons.
Two things to notice. The switches never think, they only look things up. And neither CPU ever touches the data. Software leaves a note for the HCA and hardware does the rest. That second idea is called RDMA, short for Remote Direct Memory Access.
Each switch holds a forwarding table. It is basically an array where the index is a destination address and the value is which port to send the packet out of. Here is what Leaf L1's table might look like.
| Destination LID | Send out port | Which leads to |
|---|---|---|
| 7 | 1 | Server A |
| 9 | 2 | Server A2, the neighbour |
| 42 | 3 | Up to Spine S |
| 55 | 4 | Up to the other spine |
There is no learning and no route calculation inside the switch, which keeps it simple and very fast. But it raises an obvious question. Who fills in the table? Hold that thought for Part 6.
On Ethernet, a switch with a full buffer simply drops new packets. TCP on the sending computer eventually notices and resends. That works, but the waiting and resending hurts a lot when thousands of GPUs are all waiting on each other.
InfiniBand uses credits instead. The receiver tells the sender how much buffer space it has free. Each packet costs one credit. No credits means no sending, so the sender pauses until the receiver frees up space and hands a credit back.
This is called lossless flow control. Congestion makes senders wait, and nothing gets thrown away.
Go back to the packet's trip in Part 3. It only worked because every port already had an address and every switch already had a full table. When you first power on a cluster, none of that exists.
So the Subnet Manager (SM for short) does the setup, roughly in this order.
Only someone who sees the whole map can spread traffic evenly over every spine. That even spread is what lets every GPU talk at full speed at the same time.
In a network that never drops packets, a bad set of routes can leave packets waiting on each other in a circle forever. Planning with the full map makes that easy to rule out.
All the thinking happens in one place, so the switches can stay dumb and fast.
What if the brain crashes? Traffic keeps flowing, because the switches keep their tables. The fabric just can't adapt to changes until a backup SM takes over, which is why real clusters run a standby.
UFM (Unified Fabric Manager) is NVIDIA's software for running a big InfiniBand fabric. Think of it as the Subnet Manager plus a control room.
On a big cluster that matters a lot. Instead of logging into hundreds of switches one by one, you deal with a single place.
In June 2024 the AI company Imbue published From bare metal to a 70B model, a detailed write-up of bringing up their own training cluster from scratch. Here is roughly what it looked like.
They describe InfiniBand as having "one brain for the entire network", which is the Subnet Manager from Part 6. A few of the problems they ran into make sense with the pieces above.
| Ethernet | InfiniBand | |
|---|---|---|
| Who decides routes | Each switch, on its own | One central Subnet Manager |
| When a buffer is full | Drops the packet | Sender waits for a credit |
| Who handles reliability | Software (TCP) on the CPU | The HCA, in hardware |
| Designed for | The open internet | One cluster, one owner |