Run 2
Attempts
37
29 graded
Mean score
73.7
over graded attempts
Failures
8
errors + degenerate
Tokens out
59.2k
367 model calls
Wall time
2735s
sequential
// attempts
Expression evaluator (no eval)
oneshot⚠ degenerate
glm4-9b
Python—
score
tests—
deliverables—
content checks—
1
model calls
252
tokens out
0
SLOC
0
complexity
30s
wall
Expression evaluator (no eval)
agentic✓ ok
llama3.2-3b
Python91.4
score
tests6/7
deliverables100%
content checks100%
24
model calls
5.9k
tokens out
2
SLOC
1
complexity
295s
wall
FizzBuzz
agentic✓ ok
llama3.2-3b
Python100
score
tests3/3
deliverables100%
content checks100%
3
model calls
119
tokens out
8
SLOC
5
complexity
8s
wall
JSON Pointer (RFC 6901)
agentic✓ ok
llama3.2-3b
Python82
score
tests7/10
deliverables100%
content checks100%
24
model calls
5.3k
tokens out
16
SLOC
8
complexity
305s
wall
LRU cache
agentic✓ ok
llama3.2-3b
Python76
score
tests3/5
deliverables100%
content checks100%
24
model calls
2.0k
tokens out
19
SLOC
4
complexity
111s
wall
Palindrome check
agentic✓ ok
llama3.2-3b
Python100
score
tests5/5
deliverables100%
content checks100%
3
model calls
77
tokens out
3
SLOC
3
complexity
5s
wall
Temperature conversion
agentic✓ ok
llama3.2-3b
Python100
score
tests4/4
deliverables100%
content checks100%
3
model calls
91
tokens out
4
SLOC
1
complexity
5s
wall
Text statistics module
agentic✓ ok
llama3.2-3b
Python0
score
tests0/1
deliverables0%
content checks0%
24
model calls
1.2k
tokens out
0
SLOC
1
complexity
67s
wall
Vowel count
agentic✓ ok
llama3.2-3b
Python100
score
tests5/5
deliverables100%
content checks100%
3
model calls
75
tokens out
3
SLOC
3
complexity
4s
wall
Word ladder (shortest path)
agentic✓ ok
llama3.2-3b
Python52
score
tests1/5
deliverables100%
content checks100%
24
model calls
1.6k
tokens out
17
SLOC
9
complexity
119s
wall
Expression evaluator (no eval)
oneshot✓ ok
llama3.2-3b
Python40
score
tests0/1
deliverables100%
content checks100%
1
model calls
254
tokens out
18
SLOC
4
complexity
12s
wall
FizzBuzz
oneshot✓ ok
llama3.2-3b
Python100
score
tests3/3
deliverables100%
content checks100%
1
model calls
93
tokens out
8
SLOC
5
complexity
4s
wall
JSON Pointer (RFC 6901)
oneshot⚠ degenerate
llama3.2-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
171
tokens out
0
SLOC
0
complexity
7s
wall
LRU cache
oneshot✓ ok
llama3.2-3b
Python100
score
tests5/5
deliverables100%
content checks100%
1
model calls
157
tokens out
18
SLOC
4
complexity
6s
wall
Palindrome check
oneshot✓ ok
llama3.2-3b
Python100
score
tests5/5
deliverables100%
content checks100%
1
model calls
52
tokens out
3
SLOC
3
complexity
3s
wall
Temperature conversion
oneshot✓ ok
llama3.2-3b
Python100
score
tests4/4
deliverables100%
content checks100%
1
model calls
68
tokens out
5
SLOC
1
complexity
3s
wall
Text statistics module
oneshot⚠ degenerate
llama3.2-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
155
tokens out
0
SLOC
0
complexity
6s
wall
Vowel count
oneshot✓ ok
llama3.2-3b
Python100
score
tests5/5
deliverables100%
content checks100%
1
model calls
43
tokens out
2
SLOC
3
complexity
2s
wall
Word ladder (shortest path)
oneshot✓ ok
llama3.2-3b
Python100
score
tests5/5
deliverables100%
content checks100%
1
model calls
196
tokens out
19
SLOC
9
complexity
9s
wall
Expression evaluator (no eval)
agentic✓ ok
qwen2.5-coder-3b
Python48.6
score
tests1/7
deliverables100%
content checks100%
24
model calls
9.4k
tokens out
2
SLOC
1
complexity
341s
wall
FizzBuzz
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests3/3
deliverables100%
content checks100%
24
model calls
3.3k
tokens out
12
SLOC
6
complexity
129s
wall
JSON Pointer (RFC 6901)
agentic✓ ok
qwen2.5-coder-3b
Python0
score
tests0/1
deliverables0%
content checks0%
24
model calls
3.5k
tokens out
0
SLOC
0
complexity
139s
wall
LRU cache
agentic✓ ok
qwen2.5-coder-3b
Python0
score
tests0/1
deliverables0%
content checks0%
24
model calls
5.5k
tokens out
0
SLOC
0
complexity
239s
wall
Palindrome check
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests5/5
deliverables100%
content checks100%
24
model calls
1.5k
tokens out
3
SLOC
3
complexity
63s
wall
Temperature conversion
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests4/4
deliverables100%
content checks100%
24
model calls
1.9k
tokens out
4
SLOC
1
complexity
77s
wall
Text statistics module
agentic✓ ok
qwen2.5-coder-3b
Python0
score
tests0/1
deliverables0%
content checks0%
24
model calls
8.1k
tokens out
0
SLOC
0
complexity
384s
wall
Vowel count
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests5/5
deliverables100%
content checks100%
24
model calls
1.4k
tokens out
2
SLOC
3
complexity
63s
wall
Word ladder (shortest path)
agentic✓ ok
qwen2.5-coder-3b
Python0
score
tests0/1
deliverables0%
content checks0%
24
model calls
5.1k
tokens out
0
SLOC
0
complexity
228s
wall
Expression evaluator (no eval)
oneshot✓ ok
qwen2.5-coder-3b
Python48.6
score
tests1/7
deliverables100%
content checks100%
1
model calls
500
tokens out
47
SLOC
23
complexity
16s
wall
FizzBuzz
oneshot✓ ok
qwen2.5-coder-3b
Python100
score
tests3/3
deliverables100%
content checks100%
1
model calls
138
tokens out
12
SLOC
6
complexity
6s
wall
JSON Pointer (RFC 6901)
oneshot⚠ degenerate
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
171
tokens out
0
SLOC
0
complexity
6s
wall
LRU cache
oneshot⚠ degenerate
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
148
tokens out
0
SLOC
0
complexity
6s
wall
Palindrome check
oneshot⚠ degenerate
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
89
tokens out
0
SLOC
0
complexity
3s
wall
Temperature conversion
oneshot✓ ok
qwen2.5-coder-3b
Python100
score
tests4/4
deliverables100%
content checks100%
1
model calls
79
tokens out
4
SLOC
1
complexity
5s
wall
Text statistics module
oneshot⚠ degenerate
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
303
tokens out
0
SLOC
0
complexity
14s
wall
Vowel count
oneshot✓ ok
qwen2.5-coder-3b
Python100
score
tests5/5
deliverables100%
content checks100%
1
model calls
60
tokens out
3
SLOC
3
complexity
3s
wall
Word ladder (shortest path)
oneshot⚠ degenerate
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
250
tokens out
0
SLOC
0
complexity
12s
wall
// full table
| Model | Mode | Challenge | Sample | Score | Tests | Wall | |
|---|---|---|---|---|---|---|---|
| glm4-9b | oneshot | calculator | 0 | degenerate | 30s | report | |
| llama3.2-3b | agentic | calculator | 0 | 91.4 | 6/7 | 295s | report |
| llama3.2-3b | agentic | fizzbuzz | 0 | 100.0 | 3/3 | 8s | report |
| llama3.2-3b | agentic | json-pointer | 0 | 82.0 | 7/10 | 305s | report |
| llama3.2-3b | agentic | lru-cache | 0 | 76.0 | 3/5 | 111s | report |
| llama3.2-3b | agentic | palindrome | 0 | 100.0 | 5/5 | 5s | report |
| llama3.2-3b | agentic | temperature | 0 | 100.0 | 4/4 | 5s | report |
| llama3.2-3b | agentic | text-stats | 0 | 0.0 | 0/1 | 67s | report |
| llama3.2-3b | agentic | vowel-count | 0 | 100.0 | 5/5 | 4s | report |
| llama3.2-3b | agentic | word-ladder | 0 | 52.0 | 1/5 | 119s | report |
| llama3.2-3b | oneshot | calculator | 0 | 40.0 | 0/1 | 12s | report |
| llama3.2-3b | oneshot | fizzbuzz | 0 | 100.0 | 3/3 | 4s | report |
| llama3.2-3b | oneshot | json-pointer | 0 | degenerate | 7s | report | |
| llama3.2-3b | oneshot | lru-cache | 0 | 100.0 | 5/5 | 6s | report |
| llama3.2-3b | oneshot | palindrome | 0 | 100.0 | 5/5 | 3s | report |
| llama3.2-3b | oneshot | temperature | 0 | 100.0 | 4/4 | 3s | report |
| llama3.2-3b | oneshot | text-stats | 0 | degenerate | 6s | report | |
| llama3.2-3b | oneshot | vowel-count | 0 | 100.0 | 5/5 | 2s | report |
| llama3.2-3b | oneshot | word-ladder | 0 | 100.0 | 5/5 | 9s | report |
| qwen2.5-coder-3b | agentic | calculator | 0 | 48.6 | 1/7 | 341s | report |
| qwen2.5-coder-3b | agentic | fizzbuzz | 0 | 100.0 | 3/3 | 129s | report |
| qwen2.5-coder-3b | agentic | json-pointer | 0 | 0.0 | 0/1 | 139s | report |
| qwen2.5-coder-3b | agentic | lru-cache | 0 | 0.0 | 0/1 | 239s | report |
| qwen2.5-coder-3b | agentic | palindrome | 0 | 100.0 | 5/5 | 63s | report |
| qwen2.5-coder-3b | agentic | temperature | 0 | 100.0 | 4/4 | 77s | report |
| qwen2.5-coder-3b | agentic | text-stats | 0 | 0.0 | 0/1 | 384s | report |
| qwen2.5-coder-3b | agentic | vowel-count | 0 | 100.0 | 5/5 | 63s | report |
| qwen2.5-coder-3b | agentic | word-ladder | 0 | 0.0 | 0/1 | 228s | report |
| qwen2.5-coder-3b | oneshot | calculator | 0 | 48.6 | 1/7 | 16s | report |
| qwen2.5-coder-3b | oneshot | fizzbuzz | 0 | 100.0 | 3/3 | 6s | report |
| qwen2.5-coder-3b | oneshot | json-pointer | 0 | degenerate | 6s | report | |
| qwen2.5-coder-3b | oneshot | lru-cache | 0 | degenerate | 6s | report | |
| qwen2.5-coder-3b | oneshot | palindrome | 0 | degenerate | 3s | report | |
| qwen2.5-coder-3b | oneshot | temperature | 0 | 100.0 | 4/4 | 5s | report |
| qwen2.5-coder-3b | oneshot | text-stats | 0 | degenerate | 14s | report | |
| qwen2.5-coder-3b | oneshot | vowel-count | 0 | 100.0 | 5/5 | 3s | report |
| qwen2.5-coder-3b | oneshot | word-ladder | 0 | degenerate | 12s | report |