AI & Compute
What a large language model is actually doing
Next-token prediction at scale, which explains both the capability and the failure modes better than any metaphor about thinking.
A large language model is a neural network trained to predict the next token in a sequence. Everything it does emerges from that objective, applied at very large scale.
That description sounds reductive and it is the most useful frame available, because it predicts the failure modes as well as the capabilities.
Tokens, not words
Text is broken into tokens — subword units produced by an algorithm that finds frequent character sequences.
Common words are single tokens. Rare words fragment. Numbers frequently split in ways that do not respect place value, which is part of why arithmetic is unreliable in models that have not been given tools.
This also explains character-level failures. A model asked to count letters in a word is working with a representation in which the word may be a single opaque unit.
The training objective
During pretraining, the model sees vast quantities of text and is repeatedly asked: given this sequence, what comes next?
Its parameters are adjusted to make the observed continuation more likely.
Nothing in that objective says anything about truth, helpfulness or reasoning. What it produces is a system that models the statistical structure of the text it was trained on — which, because that text was written by people describing a world, encodes a great deal about the world incidentally.
That incidental encoding is the surprise. Nobody predicted with confidence how much capability would emerge from scaling a prediction objective, and the honest position is that the theory is still catching up with the observation.
Attention and the transformer
The architecture that made scaling practical.
The key mechanism, attention, lets each position in the sequence weigh every other position when computing its representation. It replaced recurrence, which processed sequences step by step and was hard to parallelise.
The practical significance is that transformers train efficiently on modern accelerators, which turned a research architecture into an industrial one.
The cost is that attention scales quadratically with sequence length in the naive formulation, which is why long context windows are an engineering problem rather than a free parameter, and why a great deal of work has gone into approximations and optimised implementations.
What happens after pretraining
A pretrained model predicts text. It does not by default answer questions helpfully.
Additional stages shape behaviour.
Supervised fine-tuning on curated examples of the desired behaviour.
Preference-based training, where human or model-generated comparisons between outputs are used to train a reward signal, and the model is optimised against it.
These stages substantially change how a model responds and do not add knowledge. A model that did not learn a fact in pretraining will not acquire it through preference tuning; it will only become better at how it presents what it has.
Why it makes things up
The failure mode that follows directly from the objective.
The model produces a plausible continuation. Where it has seen the relevant information many times, the plausible continuation is generally the true one. Where it has not, the plausible continuation is a well-formed statement that happens to be false.
It has no internal flag distinguishing recall from construction, which is why fabrications are delivered with the same fluency as facts.
Citations are a particularly reliable failure case: the format of a reference is highly learnable, the specific content of any given reference is not.
Retrieval — supplying the model with source text at inference — reduces this substantially, because the answer is then a summarisation task rather than a recall task.
Context and memory
A model has no memory between conversations unless one is built around it.
Within a conversation, everything it can use must be in the context window. Information earlier in a long context is available and is not attended to uniformly — performance on retrieving specific facts varies with position, a phenomenon that has been measured repeatedly.
What this frame predicts well
That fluency and accuracy are separate axes.
That performance degrades on tasks poorly represented in training data, including recent events and niche technical domains.
That arithmetic and precise counting are weak without tools.
That the model is highly sensitive to how a question is phrased, because phrasing changes the distribution of plausible continuations.
And that scaling has produced large gains without any change in the underlying objective — which is the observation that has driven the entire industry's capital allocation for several years.
