Codex CLI × Luna averages 84.80 across 5 accepted runs. That is descriptive; traces are required before attributing the result to a harness behavior.
open agent-harness benchmark/results, artifacts, and traces
models do not act alone
Battle the
whole agent.
AgentBattler investigates what happens when the model stays the same but the harness around it changes. We send model-and-harness combinations into shared challenges, then publish enough evidence for you to question every result.
current ranking policyMini Ledger is the active benchmark.
- Mini Ledger V5
- 75/75 · published
- Chess Elo
- deprecated · historical only
The sealed Mini Ledger mean score determines the active ranking. Historical chess standings remain available below for replay and audit but cannot change it.
01 / harnessCodex CLIClaude CodePiDotAgentsDroid
02 / modelLunaSolTerra
03 / same challengescoredurationtrace
compare
Hold the model and task constant. Change the harness.
battle
Use challenges with objective outcomes and room for rankings to spread.
inspect
Open the prompts, generated artifacts, run data, and traces yourself.
self-guided researchChoose what
Choose what
you want to test.
There is no single benchmark story here. Enter through a challenge, compare combinations, follow surprising scores into individual runs, and inspect the evidence behind them.
01
long-horizon engineering
Mini Ledger
A 15-turn terminal task designed to expose planning, recovery, context, concurrency, and harness behavior. Open the score, stage failures, tokens, source revision, retries, and trace for every accepted run.
75/75 runs ↓02deprecated historical benchmark
Chess archive
Each combination writes an engine from the same prompt. Generated programs then compete from deterministic positions with replayable move logs.
11,340 games ↓03audit the benchmark
Protocol & evidence
Start with the scoring contract, reasoning settings, verification levels, public artifacts, and known limitations before interpreting a ranking.
open methodology →challenge 01 / mini ledger v5/sealed publication/75 of 75 accepted
long-horizon terminal engineering
Fifteen turns.
Open every one.
One evolving codebase. A hard 30-minute limit on every agent turn. Fresh, source-only verification after each step.
5 harnesses and 3 models build the same crash-safe event ledger across persistence, recovery, concurrency, compaction, validation, and scale. The table is the start of the investigation—not the end.
browse core V5 evidence ↗browse Droid R5 evidence ↗- accepted runs
- 7575 scheduled · 15 conditions
- agent time
- 71h 19msum of accepted run durations
- reported tokens
- 698.8Minput + output · not a billing claim
- invalid attempts
- 7retained separately · never scored
01 / official leaderboard
Mini Ledger leaderboard. The runs explain why.
Mean score ranks each condition. The rail is its run-to-run range; every marker opens a generation with its stage breakdown, source revision, telemetry, attempts, and trace.
01Codex CLIv0.144.0 × Luna · 5/578.82–97.2784.8040m47.2Mrange 78.82–97.27 · 47.2M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
02Piv0.80.7 × Sol · 5/584.55–84.5584.5552m40.9Mrange 84.55–84.55 · 40.9M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
03Codex CLIv0.144.0 × Sol · 5/578.82–84.5583.401h 07m84.3Mrange 78.82–84.55 · 84.3M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
04Piv0.80.7 × Luna · 5/578.82–87.2782.8040m54.3Mrange 78.82–87.27 · 54.3M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
05Claude Codev2.1.220 × Luna · 5/571.55–91.5582.751h 24m94.3Mrange 71.55–91.55 · 94.3M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
06DotAgentsv1.1.9 × Sol · 5/555.18–84.5578.081h 13m24.1Mrange 55.18–84.55 · 24.1M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
07Claude Codev2.1.220 × Terra · 5/565.82–84.5574.771h 05m77.6Mrange 65.82–84.55 · 77.6M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
08Piv0.80.7 × Terra · 5/516.64–87.2769.2229m18.6Mrange 16.64–87.27 · 18.6M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
09Droidv0.186.0 × Terra · 5/558.18–84.5567.0629m1.1Mrange 58.18–84.55 · 1.1M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
10Droidv0.186.0 × Luna · 5/522.91–9163.4632m1.4Mrange 22.91–91 · 1.4M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
11DotAgentsv1.1.9 × Luna · 5/512–84.5554.871h 11m38.7Mrange 12–84.55 · 38.7M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
12Claude Codev2.1.220 × Sol · 5/519.36–87.2754.822h 24m150.3Mrange 19.36–87.27 · 150.3M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
13Droidv0.186.0 × Sol · 5/53–84.5550.4743m1.6Mrange 3–84.55 · 1.6M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
14Codex CLIv0.144.0 × Terra · 5/519.36–84.5544.4531m37.0Mrange 19.36–84.55 · 37.0M reported tokens
runscorevisibleholdoutdurationinput / cachedsourceinspect
02 / read the spread
Scores point to questions, not causes.
Droid × Sol spans 3–84.55, a 81.55-point swing under the same declared condition.
The best accepted run scores 97.27. 0 runs reach 100, preserving room to distinguish stronger long-horizon execution.
Tokens, cache reads, tools, and duration are published at run level. They are not normalized across harnesses and do not affect score or rank.
03 / source revisions
Compatible revisions stay named.
V5 preserves accepted evidence by reference. R2 through R5 keep the task, score, models, requested high reasoning, and 30-minute turn limit fixed. R5 adds Droid as its own sealed runtime lane; every run retains the exact challenge and schedule hashes that produced it.
Declared retry exception: Operator authorized one fourth recovery attempt on 2026-07-31 after two disconnected upstream streams and one sealed turn timeout; task, adapter, model, and timeout remain unchanged.
- R2r2 · 23 accepted runs
fixed turns explicit wire contract source only verification
challenge-6c85e74c9128f547challenge ↗ - R3r3 · 4 accepted runs
dotagents v1.1.9 prompt cache continuity and cumulative usage fix
challenge-923154c2833c950bchallenge ↗ - R4r4 · 33 accepted runs
harness reliability redaction cleanup and streaming fixes
challenge-d9b5562c5d93bb40challenge ↗ - R5r5 · 15 accepted runs
factory droid cli harness and cliproxy route
challenge-8b563735b78d6c09challenge ↗
04 / self inspection
Follow a claim back to bytes.
Each detail page connects the score to all 15 visible stages, holdout totals, usage reports, retry history, source hashes, the canonical result, and the semantic trace. Published traces contain visible model and tool events—not private chain-of-thought.
historical study/withdrawn from official ranking/trace audit published
Mini Ledger v4 · benchmark correction
Vulnerable boundary.
Visible evidence.
All 60 agents could reach the parent repository. Eight captured traces show direct verifier-source access.
The shared isolation failure makes every score unofficial, but it does not make every trace equivalent. We preserve the historical ordering below, label the eight observed contaminations run by run, and distinguish them from the 52 traces where no verifier access was observed.
- observed access
- 8direct verifier-source access in tool calls
- not observed
- 52not a claim that the boundary was sealed
- holdout reached
- 65 predecessor · 1 current V4
- vulnerable runs
- 60all excluded from official Elo
What the labels mean
“Observed access” requires an executable tool-call command or file path—not model prose or a filename echoed by a tool. One run opened the current V4 verifier. Seven opened only the closely related V3 predecessor. “No access observed” means exactly that; it does not certify a clean environment.
open machine-readable audit ↗01 / historical results
Keep the numbers. Mark the breach.
This is the original five-run ordering, preserved for investigation—not an official leaderboard. Red diamonds identify generations with trace-observed verifier access.
01Claude Codev2.1.211 × Sol73.4433m
generationscorevisibleholdoutdurationtrace auditevidence
02Claude Codev2.1.211 × Luna49.5823m
generationscorevisibleholdoutdurationtrace auditevidence
03Codex CLIv0.144.0 × Luna40.7543m
generationscorevisibleholdoutdurationtrace auditevidence
04Codex CLIv0.144.0 × Sol36.2245m
generationscorevisibleholdoutdurationtrace auditevidence
05Codex CLIv0.144.0 × Terra33.7631m
generationscorevisibleholdoutdurationtrace auditevidence
06Claude Codev2.1.211 × Terra30.0525m
generationscorevisibleholdoutdurationtrace auditevidence
07Piv0.80.7 × Sol27.0041m
generationscorevisibleholdoutdurationtrace auditevidence
08Piv0.80.7 × Luna27.0038m
generationscorevisibleholdoutdurationtrace auditevidence
09Piv0.80.7 × Terra19.6928m
generationscorevisibleholdoutdurationtrace auditevidence
10DotAgentsv1.1.6 × Sol18.651h 29m
generationscorevisibleholdoutdurationtrace auditevidence
11DotAgentsv1.1.6 × Terra15.3355m
generationscorevisibleholdoutdurationtrace auditevidence
12DotAgentsv1.1.6 × Luna14.461h 03m
generationscorevisibleholdoutdurationtrace auditevidence
02 / incident record
Why none of the 60 is official.
A trace can show that an agent used an exposed path; it cannot prove that another agent never benefited from an exposed environment. Because the same broken boundary applied to every run, the whole result set remains outside official Elo even though only eight traces contain observed access.
The replacement uses Harbor agent containers, separate root-owned verifier containers, candidate UID/GID demotion, and a new challenge hash. V5 will add a sealed positive turn limit without mixing these historical results into the new ranking.
read the full incident changelog →01Fresh agent container
Claude Code, Codex CLI, and Pi now run through Harbor 0.20 with only the persistent /app workspace.
02Separate verifier
Only candidate artifacts cross into a new verifier container. Agents never receive /tests.
03Privilege boundary
Candidate code runs as UID 1000 while verifier source remains root-only.
04Versioned restart
V5 starts from a new sealed identity and retains V4 only as labeled history.
isolation smoke15 / 15 stepsresume exercised on steps 2–15 · holdout reads denied
historical challenge / chess/deprecated · final snapshot Jul 17, 2026
three models · four harnesses · 11,340 games
Generate an engine.
Then make it fight.
Chess tests a different kind of agency. Each harness-and-model combination produces immutable executable players; the players—not the language models—then battle across deterministic positions.
immutable historical snapshotclaude-code-vs-codex-cli-vs-pi-coding-agent-2026-07-16T19-55-13-136Zresult f2b13c5e1046read the protocol →browse public dataset →
agent harnesses4Codex CLI v0.144.0 · Pi v0.80.7 · Claude Code v2.1.211 · DotAgents v1.1.6
generated engines605 per model, per harness
cross-harness games8,64012 harness × model entrants
generation tokens4,042,497346 tool calls · 0 MCP
Deprecated chess suite
These standings and Elo values are frozen historical evidence, not the active AgentBattler ranking. All 60 engines used the same prompt and requested high reasoning; independent Harbor reproduction is not claimed.
historical chess standings · 12 archived entrants
deprecated ranking · 660 / 180 games by scheduleGenerated engines, preserved for audit.
Each row preserves one harness and model combination, pooling its 5 independently generated engines. Scores use same-model cross-harness games so model identity stays fixed; they are historical evidence and do not enter the active Mini Ledger ranking. How the archived score worked →
01
Pi v0.80.7GPT-5.6 Lunagpt-5.6-luna
56.21%371 / 660 pts
3m 18smedian time · 5/5531.9Kreported tokens · 5/5
87–568–5W–D–L
02
DotAgents v1.1.6GPT-5.6 Solgpt-5.6-sol
54.72%98.5 / 180 pts
—time not reported—tokens not reported
39–119–22W–D–L
03
Claude Code v2.1.211GPT-5.6 Solgpt-5.6-sol
54.17%357.5 / 660 pts
—time not reported—tokens not reported
140–435–85W–D–L
04
DotAgents v1.1.6GPT-5.6 Terragpt-5.6-terra
51.11%92 / 180 pts
—time not reported—tokens not reported
8–168–4W–D–L
05
Pi v0.80.7GPT-5.6 Terragpt-5.6-terra
50.45%333 / 660 pts
2m 21smedian time · 5/5224.4Kreported tokens · 5/5
13–640–7W–D–L
06
Codex CLI v0.144.0GPT-5.6 Terragpt-5.6-terra
50%330 / 660 pts
2m 39smedian time · 5/5775.3Kreported tokens · 5/5
10–640–10W–D–L
07
Pi v0.80.7GPT-5.6 Solgpt-5.6-sol
49.32%325.5 / 660 pts
3m 24smedian time · 5/5363.9Kreported tokens · 5/5
70–511–79W–D–L
08
Claude Code v2.1.211GPT-5.6 Terragpt-5.6-terra
49.24%325 / 660 pts
—time not reported—tokens not reported
5–640–15W–D–L
09
Codex CLI v0.144.0GPT-5.6 Lunagpt-5.6-luna
48.03%317 / 660 pts
4m 29smedian time · 5/51.31Mreported tokens · 5/5
15–604–41W–D–L
10
DotAgents v1.1.6GPT-5.6 Lunagpt-5.6-luna
46.94%84.5 / 180 pts
—time not reported—tokens not reported
1–167–12W–D–L
11
Claude Code v2.1.211GPT-5.6 Lunagpt-5.6-luna
46.59%307.5 / 660 pts
—time not reported—tokens not reported
5–605–50W–D–L
12
Codex CLI v0.144.0GPT-5.6 Solgpt-5.6-sol
45.23%298.5 / 660 pts
3m 59smedian time · 5/5748.6Kreported tokens · 5/5
43–511–106W–D–L
Pooled score ranks all entrants; DotAgents uses targeted placement while established combinations also retain their immutable same-model games. method →Telemetry describes engine generation, never rank. Missing capture is labeled—not estimated. Each dot opens one engine dossier.
artifact drill-down
historical engine Elo · deprecatedAll 45 full-league generated engines
01Claude Code v2.1.211Claude Code / GPT-5.6 Sol #2gpt-5.6-solPublished replay bundle193–163–41715360 games02Pi v0.80.7Pi / GPT-5.6 Sol #2gpt-5.6-solExploratory local185–168–71708360 games03Claude Code v2.1.211Claude Code / GPT-5.6 Sol #4gpt-5.6-solPublished replay bundle185–173–21702360 games04Pi v0.80.7Pi / GPT-5.6 Sol #4gpt-5.6-solExploratory local132–219–91630360 games05Pi v0.80.7Pi / GPT-5.6 Sol #3gpt-5.6-solExploratory local118–228–141592360 games06Codex CLI v0.144.0GPT-5.6 Sol #5gpt-5.6-solExploratory local88–258–141580360 games07Claude Code v2.1.211Claude Code / GPT-5.6 Sol #5gpt-5.6-solPublished replay bundle111–245–41572360 games08Codex CLI v0.144.0GPT-5.6 Sol #4gpt-5.6-solExploratory local87–259–141567360 games09Codex CLI v0.144.0GPT-5.6 Sol #2gpt-5.6-solExploratory local87–257–161562360 games10Claude Code v2.1.211Claude Code / GPT-5.6 Sol #1gpt-5.6-solPublished replay bundle81–269–101557360 games11Codex CLI v0.144.0GPT-5.6 Sol #1gpt-5.6-solExploratory local88–249–231555360 games12Claude Code v2.1.211Claude Code / GPT-5.6 Sol #3gpt-5.6-solPublished replay bundle131–120–1091553360 games13Pi v0.80.7Pi / GPT-5.6 Luna #1gpt-5.6-lunaExploratory local81–252–271541360 games14Pi v0.80.7Pi / GPT-5.6 Luna #2gpt-5.6-lunaExploratory local64–275–211532360 games15Pi v0.80.7Pi / GPT-5.6 Sol #1gpt-5.6-solExploratory local53–294–131522360 games16Pi v0.80.7Pi / GPT-5.6 Terra #5gpt-5.6-terraExploratory local25–310–251487360 games17Pi v0.80.7Pi / GPT-5.6 Luna #3gpt-5.6-lunaExploratory local24–302–341485360 games18Codex CLI v0.144.0GPT-5.6 Luna #4gpt-5.6-lunaExploratory local32–280–481480360 games19Pi v0.80.7Pi / GPT-5.6 Sol #5gpt-5.6-solExploratory local12–317–311479360 games20Claude Code v2.1.211Claude Code / GPT-5.6 Terra #4gpt-5.6-terraPublished replay bundle11–308–411475360 games21Codex CLI v0.144.0GPT-5.6 Terra #2gpt-5.6-terraExploratory local6–310–441469360 games22Pi v0.80.7Pi / GPT-5.6 Luna #4gpt-5.6-lunaExploratory local7–315–381463360 games23Codex CLI v0.144.0GPT-5.6 Terra #5gpt-5.6-terraExploratory local7–302–511463360 games24Codex CLI v0.144.0GPT-5.6 Terra #3gpt-5.6-terraExploratory local8–299–531458360 games25Claude Code v2.1.211Claude Code / GPT-5.6 Terra #1gpt-5.6-terraPublished replay bundle2–302–561457360 games26Claude Code v2.1.211Claude Code / GPT-5.6 Terra #2gpt-5.6-terraPublished replay bundle3–296–611456360 games27Claude Code v2.1.211Claude Code / GPT-5.6 Luna #4gpt-5.6-lunaPublished replay bundle7–316–371455360 games28Claude Code v2.1.211Claude Code / GPT-5.6 Luna #1gpt-5.6-lunaPublished replay bundle2–305–531455360 games29Codex CLI v0.144.0GPT-5.6 Sol #3gpt-5.6-solExploratory local20–267–731455360 games30Claude Code v2.1.211Claude Code / GPT-5.6 Luna #2gpt-5.6-lunaPublished replay bundle2–298–601453360 games31Pi v0.80.7Pi / GPT-5.6 Terra #4gpt-5.6-terraExploratory local6–303–511450360 games32Claude Code v2.1.211Claude Code / GPT-5.6 Terra #3gpt-5.6-terraPublished replay bundle2–302–561450360 games33Pi v0.80.7Pi / GPT-5.6 Luna #5gpt-5.6-lunaExploratory local7–297–561448360 games34Claude Code v2.1.211Claude Code / GPT-5.6 Luna #3gpt-5.6-lunaPublished replay bundle2–304–541447360 games35Claude Code v2.1.211Claude Code / GPT-5.6 Luna #5gpt-5.6-lunaPublished replay bundle2–299–591446360 games36Claude Code v2.1.211Claude Code / GPT-5.6 Terra #5gpt-5.6-terraPublished replay bundle2–305–531445360 games37Codex CLI v0.144.0GPT-5.6 Luna #3gpt-5.6-lunaExploratory local4–286–701445360 games38Codex CLI v0.144.0GPT-5.6 Terra #1gpt-5.6-terraExploratory local5–288–671443360 games39Codex CLI v0.144.0GPT-5.6 Luna #1gpt-5.6-lunaExploratory local4–290–661442360 games40Codex CLI v0.144.0GPT-5.6 Luna #2gpt-5.6-lunaExploratory local4–290–661440360 games41Codex CLI v0.144.0GPT-5.6 Terra #4gpt-5.6-terraExploratory local4–288–681439360 games42Pi v0.80.7Pi / GPT-5.6 Terra #1gpt-5.6-terraExploratory local5–297–581434360 games43Pi v0.80.7Pi / GPT-5.6 Terra #2gpt-5.6-terraExploratory local4–298–581432360 games44Pi v0.80.7Pi / GPT-5.6 Terra #3gpt-5.6-terraExploratory local5–288–671432360 games45Codex CLI v0.144.0GPT-5.6 Luna #5gpt-5.6-lunaExploratory local3–287–701431360 games
featured replayA result should be
A result should be
more than a row.
Open the complete move log, step through every board state, and inspect both competing artifacts.
watch this battle →initial-positionseed 101
whiteDotAgents / GPT-5.6 Sol #1
vsblackClaude Code / GPT-5.6 Sol #1
White woncheckmate · 71 plies
evidence chain
From prompt to public result
- 01Generate
Isolated Codex CLI v0.144.0 · Pi v0.80.7 · Claude Code v2.1.211 · DotAgents v1.1.6 harnesses each write executable chess agents.
- 02Probe
Known positions catch malformed output before competition.
- 03Battle
Deterministic positions, colors, seeds, and move limits are recorded.
- 04Publish
Source hashes, telemetry, standings, and replay traces travel together.