Methodology

One-shot — the model gets the challenge spec once and must reply with complete files in a single response. No test access, no feedback, no retries.

Agentic — the model works in an isolated workspace with five tools (list_files, read_file, write_file, run_tests, done), one JSON action per turn, up to 24 turns. Saying done is gated: it only ends the attempt if the pytest suite is green; otherwise the failure output is fed back and the clock keeps running.

Blend: tests 60% (fraction of pytest cases passing) · deliverables 20% (required files present) · content checks 20% (required substrings in deliverables). A challenge that doesn't use an axis has it reweighted away. The blend is computed and stored at grade time — the site only displays stored numbers, and every attempt page expands the blend into its axes with the raw pytest output.

Every challenge ships a reference solution that must pass its own test suite (enforced in CI), so a model failure is always a model finding — never a broken test.

Empty output, truncation at the token limit, unparseable JSON, transport errors — these are recorded as degenerate or error results, excluded from score averages, and counted separately. Averaging them in as zeros would let delivery mechanics masquerade as coding ability (see the JSON-escaping cliff).

Every run is stamped with the git SHA of the harness, the engine version, the full config snapshot (models, temperatures, token budgets, turn limits), and a content hash of the challenge set. Scores are only compared within one challenge-set hash. Inference runs at temperature 0. Local models run on one machine, sequentially — one model at a time, one attempt at a time. Full session transcripts (every prompt, action, and test run) are kept for every attempt and published with it.

Local models are served by Ollama on an Apple-Silicon Mac; remote models use the same OpenAI-compatible client. The harness is a dependency-free Python package; grading runs real pytest in the attempt's workspace. This site is fully static, generated from the results database — no JavaScript, no external resources, raw JSON published per page.