Run 1
Attempts
10
8 graded
Mean score
87.5
over graded attempts
Failures
2
errors + degenerate
Tokens out
669
125 model calls
Wall time
495s
sequential
// attempts
FizzBuzz
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests3/3
deliverables100%
content checks100%
24
model calls
0
tokens out
12
SLOC
6
complexity
90s
wall
Palindrome check
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests5/5
deliverables100%
content checks100%
24
model calls
0
tokens out
3
SLOC
3
complexity
41s
wall
Temperature conversion
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests4/4
deliverables100%
content checks100%
24
model calls
0
tokens out
4
SLOC
1
complexity
51s
wall
Text statistics module
agentic✓ ok
qwen2.5-coder-3b
Python0
score
tests0/1
deliverables0%
content checks0%
24
model calls
0
tokens out
0
SLOC
0
complexity
251s
wall
Vowel count
agentic✓ ok
qwen2.5-coder-3b
Python100
score
tests5/5
deliverables100%
content checks100%
24
model calls
0
tokens out
2
SLOC
3
complexity
41s
wall
FizzBuzz
oneshot✓ ok
qwen2.5-coder-3b
Python100
score
tests3/3
deliverables100%
content checks100%
1
model calls
138
tokens out
12
SLOC
6
complexity
6s
wall
Palindrome check
oneshot✗ error
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
89
tokens out
0
SLOC
0
complexity
3s
wall
Temperature conversion
oneshot✓ ok
qwen2.5-coder-3b
Python100
score
tests4/4
deliverables100%
content checks100%
1
model calls
79
tokens out
4
SLOC
1
complexity
2s
wall
Text statistics module
oneshot✗ error
qwen2.5-coder-3b
Python—
score
tests—
deliverables—
content checks—
1
model calls
303
tokens out
0
SLOC
0
complexity
8s
wall
Vowel count
oneshot✓ ok
qwen2.5-coder-3b
Python100
score
tests5/5
deliverables100%
content checks100%
1
model calls
60
tokens out
3
SLOC
3
complexity
2s
wall
// full table
| Model | Mode | Challenge | Sample | Score | Tests | Wall | |
|---|---|---|---|---|---|---|---|
| qwen2.5-coder-3b | agentic | fizzbuzz | 0 | 100.0 | 3/3 | 90s | report |
| qwen2.5-coder-3b | agentic | palindrome | 0 | 100.0 | 5/5 | 41s | report |
| qwen2.5-coder-3b | agentic | temperature | 0 | 100.0 | 4/4 | 51s | report |
| qwen2.5-coder-3b | agentic | text-stats | 0 | 0.0 | 0/1 | 251s | report |
| qwen2.5-coder-3b | agentic | vowel-count | 0 | 100.0 | 5/5 | 41s | report |
| qwen2.5-coder-3b | oneshot | fizzbuzz | 0 | 100.0 | 3/3 | 6s | report |
| qwen2.5-coder-3b | oneshot | palindrome | 0 | error | 3s | report | |
| qwen2.5-coder-3b | oneshot | temperature | 0 | 100.0 | 4/4 | 2s | report |
| qwen2.5-coder-3b | oneshot | text-stats | 0 | error | 8s | report | |
| qwen2.5-coder-3b | oneshot | vowel-count | 0 | 100.0 | 5/5 | 2s | report |