Harness and Verification

What the harness is

A containerised environment plus a grader. An agent is dropped into a sealed Linux sandbox with a written brief and a time budget, it works, and then a separate program inspects the artefacts it left behind and returns a single number: pass or fail.

• The agent never sees the grader.
• The grader never asks the agent how it went.
• The same scenario, run a year later, produces the same environment byte for byte.

What it is not

• Not a demo. A scenario that every model passes measures nothing and earns no place in the suite.
• Not a chat transcript review. Reasoning that sounds right and output that is right are different claims, and only the second one is graded.
• Not a one-off. A single pass can be luck. Scenarios are run repeatedly and scored across trials.
• Not retired at launch. The same suite runs on a schedule after deployment as the audit.

Reproducible

• The base image is pinned by @sha256 digest, not by tag. A tag like python:3.13-slim silently changes underneath you, and :latest is never allowed.
• Base images come from a fixed approved list, so the fleet shares a known surface.
• Every dependency, including the test dependencies, is installed at build time with a pinned version. Grading downloads nothing.
• Tests use fixed seeds and fixed inputs. No unseeded randomness, no live-generated data, no dependence on the wall clock or the date.

Honest

• No answer file, reference solution, or expected output anywhere in the image.
• The verifier is absent during the run and overlaid afterwards, so there is nothing to read ahead.
• Ground truth is either held in the verifier's own copy or recomputed from inputs the agent cannot write.
• Where the scenario allows network access, it is designed so the answer cannot simply be fetched.

Weak

def test_report():
    """A report was produced."""
    assert Path("/app/report.json").exists()

An agent that writes {} and stops passes this. So does an agent that writes an empty file and reports success. The number this produces is not a measurement.

Sound

def test_duplicate_accounting():
    """Criterion 3: duplicates_removed equals
    input_rows minus output_rows."""
    d = json.loads(Path("/app/report.json").read_text())
    assert d["input_rows"] == 14032
    assert d["output_rows"] == 13887
    assert d["duplicates_removed"] == 145

Asserts on produced values, names the criterion it covers, and cannot be satisfied without doing the work.

What a packet declares

interface VerificationSpec {
  checks: Array<{
    type: "test" | "typecheck" | "lint"
        | "build" | "runtime"
        | "security" | "performance";
    command: string;
    required: boolean;
  }>;
}

Each check is a real command run in the packet's working directory. The agent's own account of its work is not an input.

What comes back

interface VerificationResult {
  passed: boolean;      // every required check passed
  evidence: VerificationEvidence[];
  confidence: "high" | "medium" | "low";
  summary: string;
}

Evidence carries the command, the captured output, and the duration for every check, so a pass can be re-read later rather than taken on trust.

Oracle must pass

Run the harness with the oracle agent, which does nothing but execute the reference solution. Reward must be exactly 1.0.

harbor run -p . --agent oracle
cat /logs/verifier/reward.txt   # expect 1.0

If the oracle fails, the scenario is unsolvable as written or the verifier is checking something the brief never asked for.

Nop must fail

Run it again with the nop agent, which does absolutely nothing. Reward must be below 1.0.

harbor run -p . --agent nop
cat /logs/verifier/reward.txt   # expect < 1.0

If nop passes, the verifier is scoring a result that was never produced. Tighten the assertions until doing nothing fails.

Silent failure

The wrong answer looks normal. No crash, no error, no signal to keep going. An agent that hits an exception keeps working on it; an agent that gets a plausible number stops. Every wrong path is engineered to be a believable near-miss.

No self-check

The visible sample is friendly and unrepresentative. Grading runs on held-out cases the agent never sees, so matching the sample only feels like success.

All or nothing

No partial credit. One wrong component fails the whole result. Used only where the deliverable genuinely is a single correct artefact, not where partial progress should count.

Figure-out-able

The answer is recoverable from what the agent is given. Nothing decisive is secret.

Real skill

It punishes a mistake a competent practitioner would avoid, not a random trick.

Fair margins

Pass and fail do not hinge on a hair-thin tolerance. The idea decides the outcome, not the threshold.

No uncorrectable lie

Hiding the deciding case is fair. Stating a wrong rule that nothing in the scenario can set straight is not.

Pass 1, calibrating the scenario

A scenario earns a place in the suite only if it discriminates. Run it against a current frontier model at high reasoning effort:

• Stage one, two trials. At least one genuine failure, or the scenario is not measuring anything and is retired to the regression set.
• Stage two, eight fresh trials. Triggered only if stage one clears. At least four genuine failures out of eight.

Only genuine failures count. Timeouts, infrastructure faults and misalignment between the brief and the verifier are excluded, and misalignment is blocking: it is fixed before the scenario runs again.

Pass 2, grading the agent

The calibrated suite is then run against the candidate agent, multiple trials per scenario, and reported as a distribution rather than a single verdict:

• Pass rate per scenario across trials, because one pass can be luck and one failure can be a bad draw.
• Consistency, how often the same scenario produces the same outcome. An agent that alternates is not ready even if its average looks acceptable.
• Cost and wall clock per scenario, tracked as first-class results.
• Failure class, separating wrong answers from timeouts, tool errors and refusals.

Signals watched between audits

• Success rate per task type, with a threshold that raises an alert rather than waiting for the next scheduled run.
• Cost per task over time. A climbing curve on unchanged work usually means prompt or context drift.
• Tool error rates, which catch breaking changes on the other side of an integration.
• Human override and retry rates, which catch the failures the verifier was never written to see.

What the audit record holds

• Suite version and the image digest for every scenario, so the run can be reproduced exactly.
• Model and agent identifiers, and the resource budget each scenario ran under.
• Per-scenario reward, per-check evidence, confidence, and duration.
• The diff against the previous baseline, and the disposition of every changed line.

1. Grading on a rule nobody stated

The verifier checks a format, an ordering, or a tie-break the brief never mentioned. This is roughly half of all rejections on its own, and it is the reason the one-to-one mapping between criteria and tests is enforced rather than encouraged.

2. Files that contradict the brief

A data file or the reference solution follows a different rule from the instruction. The agent reads the environment correctly and is marked wrong for it.

3. A brief that reads two ways

Both readings are defensible and the grader silently accepts one. This measures the author's phrasing, not the agent.

4. The only hard part is a mistake

The difficulty comes entirely from an error in the scenario. Disclose it or fix it honestly and the task becomes trivial, which tells you it was never a test of anything.

5. A decoy with no way out

The scenario points at the wrong answer and gives the agent nothing it could use to correct course. That is not a probe, it is a trap with no skill in it.

The test to apply before shipping a scenario

Write the deciding rule down plainly and add it to the brief. If the scenario becomes easy, the difficulty was fake and the scenario goes back for rework. A sound scenario stays hard even when every rule is stated clearly.