The open loop — a feedback loop is useless if you never close it

Run 2 put four local models through all nine challenges in both modes. The headline result is backwards from what agentic tooling promises: every small model scored worse in the harness than one-shot.

• llama3.2:3b — one-shot 91.4, agentic 77.9 (−13.5)
• qwen2.5-coder:3b — one-shot 87.2, agentic 49.8 (−37.4)
• glm4:9b — one-shot 87.0, agentic 49.4 (−37.6)
• qwen3:30b-a3b — 100.0 in both (the only model that broke even; it has its own problem)

The mechanism, from the flight recorders

The harness gives every model a run_tests tool — real pytest, real failure output, free to call every turn. Across all nine agentic sessions each:

• glm4:9b called it 0 times. 59 blind write_file actions, 19 rejected done claims.
• qwen2.5-coder:3b called it 0 times. 97 blind writes. It never even claimed done.
• llama3.2:3b called it 41 times — and posted the smallest penalty on the board.
• qwen3:30b-a3b: 6 writes, 6 test runs, 5 dones. Write once, verify once, stop.

The penalty tracks loop usage almost perfectly. The models that never close the feedback loop don't just fail to benefit from it — they're actively hurt, because each blind rewrite can overwrite working code. glm4 on temperature conversion is the cleanest specimen: its one-shot answer scored a perfect 100, but in the loop it rewrote the file 20 times without ever testing, and the version standing when the budget ran out didn't even import — 0 of 4 tests, final score 40. Same model, same challenge, same day: the tools turned a right answer into a wrong one. Its json-pointer collapse (82 one-shot → 0 agentic) is the same film again.

Why it matters

"Agentic" is not a free upgrade — it's a distinct skill stack: choosing to verify, reading failure output, knowing when to stop. Below ~10B, none of the models we tested have it, and giving them tools is giving them rope. If you deploy a small model in an agent loop, the loop itself is a regression risk, not a safety net.

What we'll do about it

run_tests calls per attempt are already a first-class metric on every attempt page. Next challenge set, we'll test a prompt-level nudge (an explicit reminder after N consecutive writes without a test run) to measure how much of the open loop is prompt-fixable versus capability-limited.