Leaderboard

Mean blended score (0–100) per model and mode over the latest run. Failed attempts (transport errors, degenerate output, protocol violations) are excluded from scores and counted separately — never as zeros. Click any run for the full engineer report on every attempt, or see all runs and how scoring works.

🥇 llama3.2-3b oneshot

91.4

score

tests86%

deliverables100%

content checks100%

7/9

graded

2

failures

53s

wall time

1.2k

tokens out

🥈 qwen2.5-coder-3b oneshot

87.2

score

tests79%

deliverables100%

content checks100%

4/9

graded

5

failures

71s

wall time

1.7k

tokens out

🥉 llama3.2-3b agentic

77.9

score

tests71%

deliverables89%

content checks89%

9/9

graded

0

failures

918s

wall time

16.4k

tokens out

qwen2.5-coder-3b agentic

49.8

score

tests46%

deliverables56%

content checks56%

9/9

graded

0

failures

1663s

wall time

39.6k

tokens out

glm4-9b oneshot

—

score

tests—

deliverables—

content checks—

0/1

graded

1

failures

30s

wall time

252

tokens out

Model	Mode	Score	Graded	Failures	Run
llama3.2-3b	oneshot	91.4	7/9	2	run 2
qwen2.5-coder-3b	oneshot	87.2	4/9	5	run 2
llama3.2-3b	agentic	77.9	9/9	0	run 2
qwen2.5-coder-3b	agentic	49.8	9/9	0	run 2
glm4-9b	oneshot	—	0/1	1	run 2