AI & Compute
Memory Bandwidth And Why Chips Wait
Modern accelerators can multiply far faster than they can be fed with data, which is why memory bandwidth, not arithmetic, sets the speed of most AI workloads.
Processor performance is quoted in operations per second, yet many workloads run far below that figure. The gap is explained by where the numbers being operated on have to come from.
Arithmetic has outrun memory
Compute capability has grown faster than the ability to move data into the chip. The arithmetic units are therefore often idle, waiting for operands to arrive.
Engineers describe a workload as compute-bound or memory-bound depending on which side is the limit. Most inference work sits firmly on the memory side.
This is why a chip with impressive arithmetic specifications can disappoint on a real model.
The arithmetic intensity of the work decides which limit applies
Some operations reuse each loaded value many times, so the cost of fetching it is amortized. Large matrix multiplications behave this way and can approach the arithmetic ceiling of the chip.
Other operations touch each value once. Generating text one token at a time reads the entire set of model weights per step and does relatively little arithmetic with each.
The same hardware therefore looks fast during training and comparatively slow during single-stream generation.
Caches and stacked memory exist to shorten the trip
On-chip memory is fast and small; main memory is large and slow. Performance work is largely about keeping the data being reused in the fast tier.
Stacked memory placed adjacent to the processor narrows the gap by shortening the physical distance and widening the path, which is why it appears on high-end accelerators.
Its supply is constrained and its packaging is difficult, which makes it one of the real bottlenecks in accelerator production.
Batching converts a memory problem into a compute problem
If several requests are processed together, the weights loaded from memory are used for all of them at once. The fetch cost is shared and utilization rises.
This is why serving systems group requests and why a lightly loaded deployment is far less efficient per request than a busy one.
It also introduces a trade: waiting to assemble a batch adds delay for the user who arrived first.
What this means for buying and tuning
Comparing accelerators on arithmetic alone predicts the wrong winner for most deployments. Memory capacity and bandwidth usually decide which models fit and how fast they respond.
Similarly, the largest practical speedups often come from reducing data movement rather than from faster math, which is the reasoning behind quantization and smarter caching.
