AI & Compute
Networking Between Chips And Why It Limits Scale
Training a large model splits work across thousands of accelerators, so the links between them determine how much of the purchased compute is actually used.
A cluster's headline performance is the sum of its chips. What it delivers is set by how fast those chips can exchange results, because the work is not independent.
Parallel training requires constant agreement
Splitting a model across devices means each one holds part of the computation. After every step, partial results must be combined so that all devices continue from the same state.
That exchange happens thousands of times, and every device waits for the slowest participant. Communication is therefore not overhead at the edges; it is inside the inner loop.
As cluster size grows, the amount of coordination grows with it, which is why scaling is never linear.
There is a hierarchy of connections
Chips inside one server are linked by dedicated high-bandwidth interconnect. Servers within a rack share a switch, racks share another layer, and buildings connect over longer links.
Each step up that hierarchy is slower and more expensive per unit of bandwidth. Placement therefore matters: work that talks frequently should sit as low in the hierarchy as possible.
Schedulers and training frameworks spend considerable effort mapping the model onto that topology.
Bandwidth and latency are separate problems
Bandwidth is how much data can move per second; latency is how long the first byte takes. Large gradient exchanges are bandwidth-bound, while frequent small synchronizations are latency-bound.
A network tuned for one can be poor at the other, and the mix depends on how the model was partitioned rather than on the hardware alone.
This is why the same cluster can show excellent utilization on one workload and poor utilization on another.
Failures scale with the number of parts
A job spanning thousands of devices fails if any of them does. Optical links, cables and transceivers are numerous and individually unreliable.
Long training runs therefore checkpoint frequently and restart from the last saved state, and the time lost to restarts is a real cost of scale.
Operators track link errors closely because a degraded connection slows everyone rather than only itself.
Why interconnect drives cluster design
The cabling, switching and optics of a large cluster represent a substantial share of its cost and most of its physical complexity.
Buyers who compare accelerators alone often find that identical chip counts deliver very different throughput, and the difference is almost always in the fabric between them.
