AI & Compute
Why Training Runs Are Scheduled Like Factories
Large model training occupies thousands of chips continuously for weeks, so the work is planned around hardware failures, network topology and utilisation the way industrial production is.
A large training run is closer to a manufacturing campaign than to running a program. It occupies a fixed asset continuously, and the operational discipline required reflects that.
Idle hardware is the primary loss
A cluster represents an enormous fixed cost that accrues whether or not it is computing. Every hour of idleness is money spent without output.
Teams therefore measure utilisation obsessively and treat gaps between jobs as defects to be engineered away rather than as normal scheduling slack.
This produces the same pressures found in capital-intensive manufacturing, where the plant runs continuously because stopping it is the expensive option.
Failures are expected rather than exceptional
Across thousands of accelerators running for weeks, some hardware will fail during any given run. The probability of a completely clean campaign is low.
Systems are therefore built to checkpoint state regularly, so a failure costs the work since the last save rather than the entire run.
Checkpointing itself consumes time and storage bandwidth, so its frequency is tuned against the expected failure rate, exactly as maintenance intervals are set in a plant.
The network shapes how work is divided
Training splits a model across many chips, which must exchange large volumes of intermediate results at every step of the calculation.
How that split is arranged depends on the physical topology, because chips connected by fast local links can exchange more than those separated by slower paths.
A configuration that suits one cluster performs badly on another with different interconnect, which is why these choices are rarely portable between facilities.
Power draw varies enough to matter
A cluster's consumption swings as computation and communication phases alternate, and those swings appear at the scale of a substantial industrial load.
Utilities need warning of such behaviour, and facilities sometimes smooth it deliberately, accepting slightly lower performance to present a steadier demand.
This is one of several ways training has become a facilities question rather than purely a software one.
Preparation dominates the calendar
Before a large run begins, the data must be assembled and cleaned, the configuration validated at smaller scale, and the whole pipeline tested end to end.
Discovering a data problem partway through is extremely costly, since restarting means discarding weeks of compute that cannot be recovered.
The result is a rhythm of long preparation, a committed execution window, and analysis afterwards, which is unfamiliar to teams used to continuous deployment.
