What a GPU actually does, and why it is scarce

Machine learning runs on hardware originally designed to draw triangles. Understanding why that hardware suits the workload explains both the performance and the shortage.

Parallelism

A general-purpose processor is optimised for latency: executing a single instruction stream as fast as possible, with large caches and sophisticated branch prediction.

A graphics processor is optimised for throughput: executing the same operation across enormous numbers of data elements simultaneously, with thousands of simple cores.

Neural network training and inference are dominated by matrix multiplication, which is exactly that shape of problem — the same multiply-accumulate applied across a large array with no data-dependent branching.

That is the whole reason for the fit, and it was largely accidental. The hardware existed for games.

Specialised units

Modern accelerators go further, adding dedicated matrix multiplication units that perform small matrix operations as a single instruction.

These deliver most of the headline throughput figures, and they operate at reduced numerical precision.

Which brings in the precision question.

Precision

Neural networks tolerate low numerical precision remarkably well.

Training has moved from thirty-two-bit floating point to sixteen-bit formats, and increasingly to eight-bit for parts of the computation. Inference routinely runs at eight-bit or lower with quantisation techniques that preserve accuracy.

Halving precision roughly doubles throughput and halves memory footprint, which is why each hardware generation emphasises support for smaller formats.

There is a limit, and where it sits is an active research question rather than a settled one.

Memory bandwidth is the actual bottleneck

The point most often missed.

For large model inference, the constraint is usually not arithmetic throughput but the rate at which parameters can be moved from memory to the compute units.

Generating each token requires reading the model's weights. A model with many billions of parameters must stream a large volume of data per token, and the arithmetic units sit idle waiting.

Which is why high-bandwidth memory — stacked memory dies placed on the same package as the processor — is the critical component, and why memory capacity and bandwidth appear so prominently in accelerator specifications.

It is also why batching helps so much: processing many requests together amortises each weight read across multiple tokens, dramatically improving utilisation.

Interconnect

Large models do not fit on one accelerator. They are split across many, which must exchange data constantly.

Training in particular requires synchronising gradients across every device at every step.

Which makes the interconnect — the network between accelerators within a node and between nodes — a first-order design consideration. A cluster with fast accelerators and slow interconnect performs badly.

This is why full systems, rather than individual chips, are what large operators actually buy.

Why supply is constrained

Several bottlenecks, all real.

Leading-edge fabrication. The most advanced process nodes are produced by a very small number of foundries, effectively one at the leading edge, in a geographically concentrated location.

Advanced packaging. Attaching high-bandwidth memory stacks to the processor die requires specialised packaging capacity that has been the binding constraint for extended periods — tighter than wafer supply itself.

High-bandwidth memory. Produced by a handful of memory manufacturers, with capacity committed far in advance.

Software ecosystem. Not a physical constraint and a real one. The dominant vendor's software stack has years of accumulated libraries, kernels and framework integration. Competing hardware can match on paper and still lose on time-to-working-model.

The alternatives

Purpose-built accelerators from cloud providers and startups exist and are deployed at scale internally.

They generally trade flexibility for efficiency on specific workloads, and their adoption is limited less by silicon than by the software gap.

Inference-specific hardware is a particularly active area, because inference has different characteristics from training — lower precision tolerance, higher memory bandwidth sensitivity, and latency requirements.