Experiments in AI‑assisted software engineering.
LLM Coder Lab puts large language models to work on real software — then publishes what happened. Benchmarks on genuine tasks, complete build logs, and field notes on what these tools can actually do, where they break, and how to get the most out of them.
Current standings
🥇
qwen3-30b-a3b
agentic
100.0
qwen3-30b-a3b
oneshot
100.0
🥈
llama3.2-3b
oneshot
91.4
full leaderboard →
From the bench log
Benchmarks
Head-to-head coding evaluations on real tasks — not toy puzzles. Frontier and open-weight models, scored on working software.
Build logs
Complete, unedited session transcripts of real projects built with LLM agents — every prompt, every wrong turn, every fix.
Field notes
Practical techniques from the bench: agent workflows, prompting patterns, and the failure modes nobody writes up.
One-shot
1 model call- One prompt, one reply — complete files, blind
- No tools, no test access, no feedback, no retries
- Measures raw code generation
The harness
≤24 turns- Isolated workspace, five tools: read, write, list, run_tests, done
- Sees real pytest failures, repairs, iterates
- "Done" only counts if the suite is green
- Measures the engineering loop — including knowing when to stop
Every model runs every challenge both ways — the delta is the finding. Run 1: the same model solved fizzbuzz in 6 seconds and one call one-shot, then took 90 seconds and 24 calls to the same score in the harness — because it never realized it was finished. Read the field notes or inspect the harness, prompts and source included.
First results are in. See the leaderboard — more models and harder challenges are coming.