Perfect, when it answers — the 100.0 with an asterisk

qwen3:30b-a3b tops the run 2 leaderboard with a perfect 100.0 in both modes — the only model to break the agentic penalty, and its loop transcripts are a masterclass in economy: write once, run the tests once, say done. Five of its attempts ended in a gated done with green tests, something no small model achieved even once.

Now the asterisk: only 12 of its 18 attempts graded at all. The other six died the same way — timed out: our client's 300-second per-request ceiling expired while the model was still thinking. Four agentic attempts and the two hardest one-shots (calculator, text-stats) never produced a single byte of answer in five minutes.

What's actually happening

qwen3 is a thinking model: it spends tokens reasoning before it answers, and on hard prompts with long transcripts that reasoning runs past our wall-clock budget. This isn't the model writing wrong code — it's the model not finishing a thought inside the lab's time limit. Per our failure policy, those attempts are recorded as errors and excluded from score averages — which is exactly what produces the shiny 100.0.

Why it matters

Two lessons, one for reading leaderboards and one for running them:

• A blended score hides the reliability axis. "100.0, when it answers" and

llama3.2:3b's "77.9–91.4, sixteen times out of eighteen" are different products. The failure counters next to every score on this site are not decoration — for this model they're half the story.

• The harness parameter is part of the experiment. A 300s request ceiling is a

lab decision, stamped into every run's config snapshot. With a higher ceiling qwen3's failures would likely become slow successes — and the leaderboard would reorder without a single model changing. Benchmarks that don't publish their timeout are hiding a knob that picks winners.

What we'll do about it

Wall-clock per attempt is already on every attempt page. For the next run we'll raise the request ceiling for thinking models and record time-to-first-token, so "didn't finish" and "can't do it" stop sharing a bucket.