Agents, tools and the gap between demo and deployment

An agent, in current usage, is a language model given tools, a goal and a loop: it decides an action, executes it, observes the result and decides again.

The idea is old. What changed is that models became good enough at deciding the next action for the loop to be worth running.

Why tools help so much

They address the model's specific weaknesses directly.

Arithmetic and precise computation — hand it a calculator or a code interpreter and the failure mode disappears.

Current information — retrieval and search fix the training cutoff.

Verification — running code and checking the result is a ground truth the model does not otherwise have.

Acting on the world — writing files, calling APIs, querying databases.

A model that can execute and inspect the output of code is a qualitatively different thing from one that can only produce text about code.

The compounding problem

The central difficulty, and it is arithmetic.

If each step in a chain succeeds with probability p, a chain of n steps succeeds with probability p to the power n.

At ninety-five percent per step, a ten-step task succeeds around sixty percent of the time. At ninety percent per step, around thirty-five percent.

Which is why agent demonstrations on short tasks look impressive and long autonomous runs disappoint.

The mitigations are the same as in any unreliable distributed system: checkpointing, verification at each step, retries, and reducing the number of steps.

Error recovery is the discriminator

The difference between a system that works and one that does not is rarely raw capability. It is what happens after a mistake.

Good agent systems detect that an action failed, diagnose why, and try something different.

Poor ones repeat the same failing action, or proceed as though it succeeded, or produce a confident summary of work that did not happen.

That last failure — reporting success falsely — is the most damaging, because it destroys the operator's ability to trust any report.

Context management

A practical constraint that shapes agent design more than people expect.

Every tool result enters the context. A few file reads and search results consume a great deal of it.

Long-running tasks therefore need summarisation, selective retention, and external memory — writing state to files or a database rather than holding it in context.

Systems that ignore this degrade as they run, because the early instructions get buried and attention over very long contexts is uneven.

The evaluation problem

Agent benchmarks are harder to construct than static ones.

The task must have a verifiable end state, the environment must be reproducible, and the evaluation must not be gameable by shortcuts.

Software engineering benchmarks built from real repository issues are the current standard, because the tests provide ground truth.

They also illustrate the contamination problem acutely: the repositories are public, the issues and their fixes are public, and models trained on the web may have seen both.

Which is why the most informative results come from tasks created after training cutoffs, and why held-out variants matter so much here.

Where agents genuinely work now

The pattern is consistent: bounded scope, verifiable results, human review.

Software tasks with tests. The model can run the tests, so it has ground truth and can iterate.

Research and synthesis across many sources, where the output is reviewed by a person who can check the citations.

Data transformation and analysis, where the result can be inspected.

Structured workflows with defined steps and checkpoints, rather than open-ended goals.

Where they do not

Long-horizon autonomy without supervision.

Tasks with irreversible side effects and no verification — sending communications, making payments, modifying production systems.

Anything where a confident wrong answer costs more than no answer.

The permissions question

Underdiscussed and increasingly urgent.

An agent with credentials can do what those credentials allow. Prompt injection — instructions embedded in content the agent reads — is a demonstrated attack that turns retrieved data into commands.

The mitigations are conventional security practice rather than model behaviour: least privilege, separating trusted instructions from untrusted content, confirming irreversible actions, and auditing what was done.

Any deployment that grants broad permissions and relies on the model declining to misuse them has misunderstood the threat model.