AI & Compute
Evaluating an AI product claim
A short checklist for reading announcements, which mostly consists of asking what was measured and against what.
Topic
evaluation
Every Muskeology article tagged evaluation — 3 pieces across 1 section.
AI & Compute
Agents, tools and the gap between demo and deployment
Giving a model tools and a loop makes it far more useful and introduces reliability problems that compound with every step.
AI & Compute
Benchmarks and what they fail to measure
A score on a public test set tells you about the test set. Contamination, saturation and construct validity all get in the way.