Findings
Field notes from the bench — what the runs actually taught us. Every claim links to the attempt transcripts and reports that back it.
The anatomy of a 48.6 — what partial credit really means
A calculator that can't calculate still scores 48.6, because it shipped the right file with the right function names. Read the axes, not the blend.
Knowing when to stop is a separate skill
Models solved the task on turn 1, then burned the other 23 turns rewriting the same file — completion and termination are different capabilities.
The JSON-escaping cliff
Small models can write the code but can't ship it — the JSON action protocol collapses the moment the payload contains quotes.