AI & Compute
Why AI needs so much electricity
Training is a large one-off cost. Inference is a small cost repeated an enormous number of times, and it is the one that scales with success.
Data centre electricity demand has become a live infrastructure question in several countries, and machine learning workloads are a substantial part of the reason.
The arithmetic is worth understanding, because the public discussion frequently conflates two very different costs.
Training versus inference
Training is the process of adjusting a model's parameters over a large dataset. It runs for weeks on thousands of accelerators and consumes a large, well-defined quantity of energy.
It happens once per model, plus experiments and failed runs that never ship — and those are a substantial and rarely reported share of the total.
Inference is running the trained model to produce an output. Each individual inference is small.
The important asymmetry is that training cost is fixed and inference cost scales with usage. A model used by a hundred million people daily will, over its life, consume far more energy serving requests than it did being trained.
Which means the energy question is fundamentally about deployment scale, not about the headline training figure that gets reported.
Where the power goes inside a data centre
The accelerators themselves. High-end training chips draw substantial power each, and a rack of them draws more than an entire traditional server rack.
This is the reason data centre design has changed: racks that once drew a few kilowatts now draw tens, and in dense AI deployments considerably more.
Cooling. Historically a large overhead, now improved.
The industry metric is power usage effectiveness — total facility power divided by IT power. A value of 1.0 would mean no overhead at all. Modern large facilities operate close to that; older or smaller ones are worse.
Rack power densities have pushed the industry toward liquid cooling, because air cannot remove heat fast enough at these densities.
Networking and storage, a smaller but non-trivial share, and networking matters more in training than in inference because the accelerators must exchange gradients constantly.
Why efficiency gains do not reduce total demand
Per-operation efficiency has improved enormously — successive accelerator generations have delivered large gains in performance per watt, and model architectures and serving techniques have improved further.
Total consumption has risen anyway, because capability improvements drive usage growth faster than efficiency improvements reduce per-use cost.
This is a well-documented pattern in energy economics: efficiency gains that make a service cheaper tend to increase consumption of it. It is not a reason to stop pursuing efficiency, and it is a reason not to expect efficiency alone to flatten demand.
The rebound that is not measured
One further complication in the accounting.
Reported energy figures generally cover the data centre. They do not cover the devices at the other end, the networks carrying the traffic, or the activity the service induces.
A model that makes a task cheaper causes that task to be performed more often, and frequently causes adjacent activity that would not otherwise have happened.
None of that appears in a per-query energy figure, which is why such figures are informative about the marginal cost of one request and uninformative about the aggregate effect of the service existing.
The grid problem
The constraint is frequently not generation but interconnection.
Connecting a large new load to the transmission network requires studies, equipment and often new lines, and the queues in several regions are measured in years.
Which is why operators are siting facilities near existing generation, signing long-term power purchase agreements, and in some cases pursuing on-site generation.
It is also why nuclear power — both existing plants seeking new offtake and proposed small modular reactors — has re-entered the conversation. Data centres want firm, continuous, low-carbon power, which is an unusual combination of requirements.
Water
Less discussed and locally significant.
Evaporative cooling consumes water, and the quantities at large facilities are meaningful in water-stressed regions.
Closed-loop and air-cooled designs use much less water and more electricity, which is a genuine trade rather than a free improvement.
What would actually change the trajectory
Smaller models for most tasks. A great deal of production traffic is served by models far larger than the task requires. Routing and distillation address this directly.
Better serving efficiency, including batching, quantisation and caching. These have already delivered large gains and continue to.
Hardware specialisation for inference specifically, which has different characteristics from training.
Siting and scheduling against grid conditions, running flexible workloads when clean power is abundant.
None of these are speculative. All are being pursued, and the honest position is that whether they outpace demand growth is not yet known.
