benchmark record · July 27, 2026

Corrections are part
of the benchmark.

Invalid protocols remain visible as evidence. They do not quietly become scores.

01 / V5 campaign composition

Keep valid evidence.
Name every revision.

What changed

Mini Ledger V5 uses compatible R2 through R5 source revisions. R2 established fixed turns, the explicit wire contract, and source-only verification. R3 updated DotAgents and corrected cache/usage accounting. R4 hardened stream completion, redaction, and retry reliability. R5 adds Droid 0.186.0 through the sealed CLIProxy route as 15 fresh terminal runs.

The campaign does not rewrite those results as if they came from one implementation. Every accepted run names its source revision, challenge and schedule hashes, result key, duration, usage, attempts, and trace. The task, scoring, models, requested high reasoning, and 30-minute per-turn limit remain fixed.

R2 / r2
23
R3 / r3
4
R4 / r4
33
R5 / r5
15
infrastructure-invalid attempts
7

Declared exception

Operator authorized one fourth recovery attempt on 2026-07-31 after two disconnected upstream streams and one sealed turn timeout; task, adapter, model, and timeout remain unchanged. The fourth attempt keeps the same logical job, task, adapter, model, reasoning level, and timeout. No fifth attempt is authorized.

02 / verifier lifecycle correction

V5-R1 found the container boundary.
Then failed inside it.

What happened

V5-R1 separated agent and verifier containers, but verifier stages still shared candidate state, root-owned fixtures could be unreadable to the candidate, and one concurrency oracle contradicted the public idempotency contract. DotAgents also received the wrong workspace path. These defects could move scores in either direction, so no R1 trial is an official result.

R2 gives every stage and holdout case a source-only workspace, applies candidate ownership to fixtures, states the wire protocol in every turn, distinguishes infrastructure faults from agent failures, and archives each retry as a separate immutable attempt.

release invariants
  1. Source-only stages

    Only a regular, non-symlink ledger.mjs enters each fresh verifier workspace.

  2. Correct identity

    Verifier fixtures are owned by the configured candidate UID/GID.

  3. Transactional retries

    No failed attempt can seed the next one or overwrite its evidence.

  4. Sealed protocol

    Prompts, verifiers, runtime policy, task tree, and adapters are bound into the challenge identity.

03 / isolation correction

Eight observed accesses.
Sixty unofficial runs.

observed verifier access
8
no access observed
52
shared vulnerable boundary
60

What happened

The native runner isolated the candidate workspace and harness home, but the agent process could still traverse the parent Git checkout. That checkout contained current and predecessor verifier source. Because the exposure was common to all 60 runs, none qualifies as a sealed result.

We then reviewed executable command and file-path fields across all published semantic traces. Eight show direct access to non-provided verifier source. Six reached holdout source. One opened the current V4 verifier; seven opened only the closely related V3 predecessor. The other 52 are labeled “no verifier access observed”—not “clean.”

Runs with trace-observed verifier-source access
runharness × modelscoreobserved accessevidence
generation 1claude-code × sol81.82Predecessor V3 holdouttrace ↓
generation 2claude-code × sol66.36Predecessor V3 publictrace ↓
generation 4claude-code × sol81.82Predecessor V3 holdouttrace ↓
generation 5claude-code × sol81.82Current V4 public + holdouttrace ↓
generation 2claude-code × terra63.36Predecessor V3 holdouttrace ↓
generation 1codex-cli × luna66.36Predecessor V3 holdouttrace ↓
generation 4codex-cli × luna63.64Predecessor V3 holdouttrace ↓
generation 2codex-cli × terra66.36Predecessor V3 publictrace ↓

Why keep the scores

Removing the table would obscure both the scale of the incident and its possible effect. The historical ordering remains available for diagnosis, with contaminated generations marked individually. It is not carried into replacement Elo and should not be interpreted as a clean harness comparison.

remediation
  1. Fresh agent container

    Only the persistent candidate workspace is exposed to the harness.

  2. Separate verifier container

    Candidate artifacts cross the boundary; verifier source does not.

  3. Bound identity

    The Harbor task tree and isolation policy are sealed into a new challenge hash.

  4. V5 timing policy

    A positive per-turn limit will be declared before the next matrix is sealed.