The anatomy of a 48.6 — what partial credit really means

Run 2 introduced harder challenges, and with them the lab's first partial-credit scores: qwen2.5-coder:3b got 48.6 in both modes on the calculator challenge (build an arithmetic expression evaluator with precedence and parentheses — eval() forbidden and test-enforced). A 48.6 sounds like "half a calculator." It isn't.

What the 48.6 is made of

Our blend is tests 60% / deliverables 20% / content checks 20%. On both attempts:

• Tests: 1 of 7 passed — and the one pass was the source-inspection check that verifies eval() isn't used. Every functional test failed. The one-shot parser crashes with IndexError on every expression; the agentic one returns None instead of numbers.
• Deliverables: 100% — calc.py exists.
• Content checks: 100% — it contains def evaluate.

So: 0.14×60 + 1.0×20 + 1.0×20 ≈ 48.6. The model produced something calculator-shaped that does not calculate, and shape is what the non-test axes reward.

Why it matters

Blended scores are convenient for ranking and terrible for understanding. Two lessons we're baking into the site rather than hiding:

• Every score on the leaderboard expands into its axes (the meters on each card), and every attempt page shows the actual pytest output — the blend is never the only number.
• Deliverable/content axes measure compliance, tests measure competence. When they disagree this hard, the disagreement is itself the finding: the model followed instructions perfectly and failed the engineering. Recursive-descent parsing appears to be where 3B coding models crack.

What we'll do about it

Weights stay frozen for the current challenge-set hash (changing them mid-set would corrupt comparability). For the next set we'll consider gating the compliance axes on at least one functional test passing, so "shaped like the answer" stops earning 40 points on its own.