Knowing when to stop is a separate skill

Our agentic mode gives a model tools (write_file, run_tests, done) and a 24-turn budget. The intended loop: write code, run the tests, say done. The observed loop, for every graded agentic attempt in run 1, was different: solve it immediately, then never stop.

What happened

• On palindrome (agentic), qwen2.5-coder:3b wrote a correct 159-byte solution on turn 1 — then wrote the identical file 23 more times. It never called run_tests. It never said done. The score of 100 came from our external grading pass, not from the model knowing it had finished.
• Every one of the model's graded agentic attempts in run 1 used all 24 model calls, including the ones that scored 100. Compare the one-shot attempts on the same challenges: one call, same score.

Why it matters

"Can complete the task" and "knows the task is complete" are separate skills, and the second one is what agentic deployment actually pays for — termination is the difference between an agent and a loop. Here the missing termination made correct answers cost 24× the compute of the one-shot path. Timing on the run report: 6 seconds one-shot vs 41–90 seconds agentic for identical output.

What we'll do about it

Model calls, self-test runs, and done-gate outcomes are now first-class metrics on every attempt page, so termination behavior is visible per model. A future protocol nudge experiment (reminding the model that run_tests and done exist after repeated identical writes) can quantify how much of this is prompt-fixable.