The JSON-escaping cliff

Both of our run modes make the model deliver code inside a JSON envelope: one-shot replies are {"files": {...}}, agentic turns are one JSON action like {"action": "write_file", "path": ..., "content": ...}. For a 3B model this envelope turns out to be a capability cliff of its own — separate from, and in our first runs larger than, the difficulty of the code inside it.

What happened

• In run 1, qwen2.5-coder:3b solved fizzbuzz, temperature, and vowel-count cleanly in both modes. But on text-stats (agentic) it emitted the same malformed JSON action 24 turns in a row — at temperature 0 the identical mistake, every turn, until the budget ran out. Score: 0.
• The failure is mechanical: the solution it wants to write contains a regex with quote characters ([A-Za-z0-9']+), and the model fails to escape them inside the JSON string. The parser dies at the same column every time.
• The same cliff took out the one-shot attempts on palindrome and text-stats in run 1, and produced run 2's first degenerate result on lru-cache.

Why it matters

Any leaderboard that makes models emit code through JSON is partly measuring JSON string escaping, not programming. At the 3B scale that confound can dominate: this model demonstrably writes correct Python and then loses every point on delivery mechanics.

What we'll do about it

Add a fenced-file protocol variant (code inside markdown fences, no JSON envelope for file content) as an explicit comparison axis, so "can write the code" and "can ship it through the protocol" get measured separately. Degenerate deliveries are already excluded from scores rather than counted as zeros — see the methodology.