Harness × model interaction
Claude Code with Sol led at 73.44. The same model family did not hold the same rank across harnesses, so neither a harness-only nor model-only summary explains the result.
completed study/60 of 60 valid runs/published with traces
Mini Ledger v4 · long-horizon terminal benchmark
Long tasks change
the ranking.
Four coding harnesses ran the same three models through a continuous 15-turn engineering task. Five independent generations per combination. Nobody solved it perfectly—and the harness mattered.
- runs
- 6012 combos × 5 generations
- turns
- 900same session + workspace
- best average
- 73.44Claude Code × Sol
- perfect runs
- 0best single run 81.82
01 / final leaderboard
A spread, not a saturation test.
Score combines 70 visible stage points with 30 points from 11 private holdout checks. Bars show the five-run range; markers are individual generations.
01Claude Codev2.1.211 × Sol73.4433m
02Claude Codev2.1.211 × Luna49.5823m
03Codex CLIv0.144.0 × Luna40.7543m
04Codex CLIv0.144.0 × Sol36.2245m
05Codex CLIv0.144.0 × Terra33.7631m
06Claude Codev2.1.211 × Terra30.0525m
07Piv0.80.7 × Sol27.0041m
08Piv0.80.7 × Luna27.0038m
09Piv0.80.7 × Terra19.6928m
10DotAgentsv1.1.6 × Sol18.651h 29m
11DotAgentsv1.1.6 × Terra15.3355m
02 / what we learned
The model name was not the whole condition.
Within-combo variance is real
Codex CLI × Terra ranged from 3.00 to 66.36—a 63.36-point swing under the same declared condition.
Duration did not buy the win
The fastest average condition was Claude Code × Luna at 23m. Some slower conditions scored lower; runtime is reported as evidence, not interpreted as causal.
The ceiling held
The best individual run scored 81.82. Zero runs reached 100, leaving headroom across concurrency, recovery, compaction, validation, and integrated stress behavior.
03 / fairness contract
Same job. Sealed schedule. Every combination.
All 60 jobs were declared before execution. Each run used the same prompt sequence, verifier hashes, high reasoning request, isolated workspace, disabled network, and one continuous harness session. A run only scores when all 15 turns complete and both verifier layers execute.
Claude Code and DotAgents reached the pinned model endpoints through CLIProxyAPI; Codex CLI and Pi used their native routes. That transport difference is part of the published harness condition. Token telemetry is retained but not ranked because harnesses report cache and context usage differently.
- turn budget
- 20 minutes
- workspace
- 50 MB max
- public score
- 70 points
- holdout score
- 30 points
- human intervention
- invalidates run
- network
- disabled
04 / evidence releaseDon’t trust the chart.
Don’t trust the chart.
Open the run.
49 MB of downloadable semantic traces preserve all distinct messages, tool calls, results, usage events, and stderr from 25.0 GB of cumulative raw streams.
What “semantic trace” means
Pi and early DotAgents streams repeatedly emitted the entire growing conversation, producing 26.8 GB from 900 turns. The release removes only those cumulative streaming snapshots after retaining their final messages and tool calls. Every source trace is hashed, the transformation is open source, 225 secret-shaped fields or values were redacted, and host paths were normalized.
inspect the exporter ↗companion chess benchmark/Jul 17, 2026
the battle lane · three models · four harnesses · 11,340 games
Generated agents.
Then battle them.
The original chess lane remains published alongside the terminal study: 12 harness and model combinations, immutable generated engines, and deterministic game replays.
snapshotclaude-code-vs-codex-cli-vs-pi-coding-agent-2026-07-16T19-55-13-136Zresult f2b13c5e1046read the protocol →browse public dataset →
agent harnesses4Codex CLI v0.144.0 · Pi v0.80.7 · Claude Code v2.1.211 · DotAgents v1.1.6
generated engines605 per model, per harness
cross-harness games8,64012 harness × model entrants
generation tokens4,042,497346 tool calls · 0 MCP
Exploratory harness suite
All 60 engines used the same prompt and requested high reasoning. The leaderboard compares same-model games across 4 harnesses — Codex CLI v0.144.0 · Pi v0.80.7 · Claude Code v2.1.211 · DotAgents v1.1.6 — and shows each combo’s schedule size; independent Harbor reproduction is not claimed.
primary leaderboard · 12 benchmark entrants
controlled same-model score · 660 / 180 games by scheduleHarness × model leaderboard
Each row is one harness and model combination, pooling its 5 independently generated engines. Scores use same-model cross-harness games so model identity stays fixed; the dots keep generation variance visible. How pooled score works →
01
Pi v0.80.7GPT-5.6 Lunagpt-5.6-luna
56.21%371 / 660 pts
87–568–5W–D–L
02
DotAgents v1.1.6GPT-5.6 Solgpt-5.6-sol
54.72%98.5 / 180 pts
39–119–22W–D–L
03
Claude Code v2.1.211GPT-5.6 Solgpt-5.6-sol
54.17%357.5 / 660 pts
140–435–85W–D–L
04
DotAgents v1.1.6GPT-5.6 Terragpt-5.6-terra
51.11%92 / 180 pts
8–168–4W–D–L
05
Pi v0.80.7GPT-5.6 Terragpt-5.6-terra
50.45%333 / 660 pts
13–640–7W–D–L
06
Codex CLI v0.144.0GPT-5.6 Terragpt-5.6-terra
50%330 / 660 pts
10–640–10W–D–L
07
Pi v0.80.7GPT-5.6 Solgpt-5.6-sol
49.32%325.5 / 660 pts
70–511–79W–D–L
08
Claude Code v2.1.211GPT-5.6 Terragpt-5.6-terra
49.24%325 / 660 pts
5–640–15W–D–L
09
Codex CLI v0.144.0GPT-5.6 Lunagpt-5.6-luna
48.03%317 / 660 pts
15–604–41W–D–L
10
DotAgents v1.1.6GPT-5.6 Lunagpt-5.6-luna
46.94%84.5 / 180 pts
1–167–12W–D–L
11
Claude Code v2.1.211GPT-5.6 Lunagpt-5.6-luna
46.59%307.5 / 660 pts
5–605–50W–D–L
12
Codex CLI v0.144.0GPT-5.6 Solgpt-5.6-sol
45.23%298.5 / 660 pts
43–511–106W–D–L
Pooled score ranks all entrants; DotAgents uses targeted placement while established combinations also retain their immutable same-model games. method →Each dot opens one generated engine dossier.
artifact drill-down
individual engine Elo · provisionalAll 45 full-league generated engines
01Claude Code v2.1.211Claude Code / GPT-5.6 Sol #2gpt-5.6-solPublished replay bundle193–163–41715360 games02Pi v0.80.7Pi / GPT-5.6 Sol #2gpt-5.6-solExploratory local185–168–71708360 games03Claude Code v2.1.211Claude Code / GPT-5.6 Sol #4gpt-5.6-solPublished replay bundle185–173–21702360 games04Pi v0.80.7Pi / GPT-5.6 Sol #4gpt-5.6-solExploratory local132–219–91630360 games05Pi v0.80.7Pi / GPT-5.6 Sol #3gpt-5.6-solExploratory local118–228–141592360 games06Codex CLI v0.144.0GPT-5.6 Sol #5gpt-5.6-solExploratory local88–258–141580360 games07Claude Code v2.1.211Claude Code / GPT-5.6 Sol #5gpt-5.6-solPublished replay bundle111–245–41572360 games08Codex CLI v0.144.0GPT-5.6 Sol #4gpt-5.6-solExploratory local87–259–141567360 games09Codex CLI v0.144.0GPT-5.6 Sol #2gpt-5.6-solExploratory local87–257–161562360 games10Claude Code v2.1.211Claude Code / GPT-5.6 Sol #1gpt-5.6-solPublished replay bundle81–269–101557360 games11Codex CLI v0.144.0GPT-5.6 Sol #1gpt-5.6-solExploratory local88–249–231555360 games12Claude Code v2.1.211Claude Code / GPT-5.6 Sol #3gpt-5.6-solPublished replay bundle131–120–1091553360 games13Pi v0.80.7Pi / GPT-5.6 Luna #1gpt-5.6-lunaExploratory local81–252–271541360 games14Pi v0.80.7Pi / GPT-5.6 Luna #2gpt-5.6-lunaExploratory local64–275–211532360 games15Pi v0.80.7Pi / GPT-5.6 Sol #1gpt-5.6-solExploratory local53–294–131522360 games16Pi v0.80.7Pi / GPT-5.6 Terra #5gpt-5.6-terraExploratory local25–310–251487360 games17Pi v0.80.7Pi / GPT-5.6 Luna #3gpt-5.6-lunaExploratory local24–302–341485360 games18Codex CLI v0.144.0GPT-5.6 Luna #4gpt-5.6-lunaExploratory local32–280–481480360 games19Pi v0.80.7Pi / GPT-5.6 Sol #5gpt-5.6-solExploratory local12–317–311479360 games20Claude Code v2.1.211Claude Code / GPT-5.6 Terra #4gpt-5.6-terraPublished replay bundle11–308–411475360 games21Codex CLI v0.144.0GPT-5.6 Terra #2gpt-5.6-terraExploratory local6–310–441469360 games22Pi v0.80.7Pi / GPT-5.6 Luna #4gpt-5.6-lunaExploratory local7–315–381463360 games23Codex CLI v0.144.0GPT-5.6 Terra #5gpt-5.6-terraExploratory local7–302–511463360 games24Codex CLI v0.144.0GPT-5.6 Terra #3gpt-5.6-terraExploratory local8–299–531458360 games25Claude Code v2.1.211Claude Code / GPT-5.6 Terra #1gpt-5.6-terraPublished replay bundle2–302–561457360 games26Claude Code v2.1.211Claude Code / GPT-5.6 Terra #2gpt-5.6-terraPublished replay bundle3–296–611456360 games27Claude Code v2.1.211Claude Code / GPT-5.6 Luna #4gpt-5.6-lunaPublished replay bundle7–316–371455360 games28Claude Code v2.1.211Claude Code / GPT-5.6 Luna #1gpt-5.6-lunaPublished replay bundle2–305–531455360 games29Codex CLI v0.144.0GPT-5.6 Sol #3gpt-5.6-solExploratory local20–267–731455360 games30Claude Code v2.1.211Claude Code / GPT-5.6 Luna #2gpt-5.6-lunaPublished replay bundle2–298–601453360 games31Pi v0.80.7Pi / GPT-5.6 Terra #4gpt-5.6-terraExploratory local6–303–511450360 games32Claude Code v2.1.211Claude Code / GPT-5.6 Terra #3gpt-5.6-terraPublished replay bundle2–302–561450360 games33Pi v0.80.7Pi / GPT-5.6 Luna #5gpt-5.6-lunaExploratory local7–297–561448360 games34Claude Code v2.1.211Claude Code / GPT-5.6 Luna #3gpt-5.6-lunaPublished replay bundle2–304–541447360 games35Claude Code v2.1.211Claude Code / GPT-5.6 Luna #5gpt-5.6-lunaPublished replay bundle2–299–591446360 games36Claude Code v2.1.211Claude Code / GPT-5.6 Terra #5gpt-5.6-terraPublished replay bundle2–305–531445360 games37Codex CLI v0.144.0GPT-5.6 Luna #3gpt-5.6-lunaExploratory local4–286–701445360 games38Codex CLI v0.144.0GPT-5.6 Terra #1gpt-5.6-terraExploratory local5–288–671443360 games39Codex CLI v0.144.0GPT-5.6 Luna #1gpt-5.6-lunaExploratory local4–290–661442360 games40Codex CLI v0.144.0GPT-5.6 Luna #2gpt-5.6-lunaExploratory local4–290–661440360 games41Codex CLI v0.144.0GPT-5.6 Terra #4gpt-5.6-terraExploratory local4–288–681439360 games42Pi v0.80.7Pi / GPT-5.6 Terra #1gpt-5.6-terraExploratory local5–297–581434360 games43Pi v0.80.7Pi / GPT-5.6 Terra #2gpt-5.6-terraExploratory local4–298–581432360 games44Pi v0.80.7Pi / GPT-5.6 Terra #3gpt-5.6-terraExploratory local5–288–671432360 games45Codex CLI v0.144.0GPT-5.6 Luna #5gpt-5.6-lunaExploratory local3–287–701431360 games
featured replayA result should be
A result should be
more than a row.
Open the complete move log, step through every board state, and inspect both competing artifacts.
watch this battle →initial-positionseed 101
whiteDotAgents / GPT-5.6 Sol #1
vsblackClaude Code / GPT-5.6 Sol #1
White woncheckmate · 71 plies
evidence chain
From prompt to public result
- 01Generate
Isolated Codex CLI v0.144.0 · Pi v0.80.7 · Claude Code v2.1.211 · DotAgents v1.1.6 harnesses each write executable chess agents.
- 02Probe
Known positions catch malformed output before competition.
- 03Battle
Deterministic positions, colors, seeds, and move limits are recorded.
- 04Publish
Source hashes, telemetry, standings, and replay traces travel together.