Tokens, Context Windows And What They Really Cost

Every interaction with a language model is measured in tokens, and almost every practical limit and price is expressed in them. Understanding what a token is explains several otherwise puzzling behaviours.

A token is a fragment, not a word

Text is split into pieces drawn from a fixed vocabulary built during training. Common words often become a single token, while unusual ones are broken into several fragments.

This is why the same idea expressed in different languages can consume very different numbers of tokens, and why code and identifiers are often more expensive than ordinary prose of equal length.

It also explains certain failures with spelling and character counting. The model does not see individual letters unless the tokenisation happens to separate them.

The context window is a hard boundary

A model can only attend to text inside its context window, which holds the instructions, the supplied documents, the conversation so far and the response being generated.

Anything outside that window does not exist for the model. There is no partial awareness of earlier material, only its absence.

Applications that maintain long conversations must therefore decide what to drop or summarise, and those decisions shape behaviour more than most users realise.

Longer context costs more than proportionally

The attention mechanism compares every position with every other, so the computation grows faster than the length of the input.

Various techniques reduce this in practice, but the underlying pressure remains, which is why very long contexts became available only as hardware and algorithms improved together.

Memory is a parallel constraint, since intermediate values for the whole sequence must be held while generating, and that storage scales with length as well.

Position within the window affects attention

Material placed at the beginning or end of a long context is often used more reliably than material buried in the middle.

This is an empirical tendency rather than a rule, and it varies between models, but it has practical consequences for how documents should be ordered in a prompt.

It also means that simply increasing window size does not guarantee the model will use everything placed inside it equally well.

Caching changes the economics of repetition

When the same long prefix appears in many requests, the work of processing it can be computed once and reused, provided the prefix is identical.

This makes stable instructions and reference documents far cheaper to include repeatedly, and it rewards putting variable content at the end rather than the start.

Applications built without regard for this often pay repeatedly for identical computation, which becomes the dominant cost at scale.