How the tests work

From make run to a number on the leaderboard, with no human judgment anywhere in between. Four diagrams, rendered to static SVG at publish time — this page, like the rest of the site, ships zero JavaScript.

01 the assembly line

one attempt = one isolated workspace; every result commits to SQLite as it lands, so runs resume after a crash

Run modes

one-shot vs agent loop

Turn budget

per agentic attempt

Score blend

60·20·20

tests · files · content

Temperature

deterministic replies

Dependencies

stdlib-only engine

Human scoring

pytest decides

Green boxes are inputs and outputs, blue is the engine, amber is grading, purple is storage. The challenge set is content-hashed and every run is stamped with it — edit one spec or test and the universe forks: old scores are never compared against new challenges. The full engine source is published on the harness page.

02 inside one agentic turn

the done gate is why a model cannot talk its way to a score — only green tests end an attempt early

Two red paths tell real stories. The corrective nudge loop is where small models die of JSON escaping — one model emitted the identical malformed action 24 turns straight (the escaping cliff). And budget exhausted catches the models that solve the task on turn 1 and never realize it (knowing when to stop).

03 the fate of every reply

failures are data, not zeros — averaging them in would let delivery mechanics masquerade as coding ability

The grade is external: after the attempt ends — by gated done, by exhaustion, or by failure — the harness runs the pytest suite itself and computes the blend. Every attempt page shows the axes, the raw pytest output, and the full transcript, so a suspicious number is always one click from its receipts.

04 who tests the testers

the harness itself sits under three test layers plus a publish gate — its first browser run caught a real mobile bug

Every challenge ships a reference solution that must pass its own suite in CI, so a model failure is always a model finding. A guard test forbids any page from referencing the network. The browser matrix drives the real site in five desktop and mobile profiles. And make publish refuses to ship a board whose numbers look implausible. Details in the methodology.