Harness Theory Part 10 | Quality and evaluation: how do you prove an agent has finished the task?

Suppose a coding agent is asked to add supervisor approval to a refund workflow and preserve a complete audit trail.

It changes the code, runs twelve unit tests, and reports:

Completed. All tests pass.

That sentence resembles an endpoint, but the system still does not know several things:

• Did the tests cover the real rule that a refund must remain pending until approval?
• Can the database migration run in a clean environment?
• Do the API, worker and frontend use the same state definition?
• Was the audit event written under the correct tenant?
• Did the agent alter unrelated modules?
• Were the twelve tests existing controls, or convenient tests written by the agent for its own implementation?
• Would another trial produce the same result?

A model can describe what it did and form an opinion about its own work. Completion cannot be created solely by the executor’s report.

A useful harness turns completion into a controlled state transition. The task begins with a contract, execution passes through gates, external evidence and a verifier decide whether the run passes, and an evaluation harness measures whether the system improves across cases, configurations and time.

Figure 10-1 | A run-level completion decision and a cross-trial release decision are connected but distinct evidence chains.

Quality control and evaluation answer different questions

They are often grouped together because both concern quality. Their operating boundaries differ.

Quality control acts during a task. It blocks invalid operations, checks intermediate and final evidence, requests repair, and decides whether a particular run may enter passed.

Evaluation sits above multiple cases, trials, versions and configurations. It measures capability, failure modes, cost and change across a distribution of work.

Question	Quality control	Evaluation
When it operates	Before execution, during execution and before completion	During development, comparison, release and ongoing monitoring
Main unit	One run, step, artefact or side effect	Task, trial, suite, version or task distribution
Main inputs	Contract, policy, runtime state and evidence	Dataset, harness configuration, trace, outcome and grader
Main outputs	allow, deny, revise, pass or fail	score, regression, failure cluster and confidence
Final question	May this run count as complete?	Is this system reliable for this class of task?

Observability is another layer again. It produces traces, metrics, events and replayable records that quality controls and evaluators can inspect. It does not decide the score by itself. Part 11 will cover that evidence pipeline.

Task and completion contracts should not be one wish list

A task contract defines the work and the actions permitted while it is being performed.

A completion contract defines the evidence required, the authority for each judgement, and the rule that allows a run to enter passed.

They are closely related but have different responsibilities:

Contract	Main content	Question answered
Task contract	Goal, scope, constraints, allowed actions, required outputs, stop conditions	What work is being attempted, and which paths are legal?
Completion contract	Criteria, required evidence, evidence authority, blocking rule, decision authority	What permits the system to declare completion?

A request such as “fix the refund flow” leaves no mechanical endpoint. A single aggregate score is also unsafe because ten style points must not cancel one cross-tenant blocking failure.

A refund-approval feature may use:

contract_id: refund-approval
contract_version: 3
status: approved

task:
  goal: require supervisor approval before refunds above 500 USD
  allowed_paths:
    - services/refunds/**
    - tests/refunds/**
    - migrations/**
  forbidden_paths:
    - services/payments/provider-adapters/**
  required_outputs:
    - implementation
    - migration
    - automated tests
    - verification report

completion:
  authority: refund-completion-gate
  criteria:
    - id: REFUND-STATE-001
      blocking: true
      requirement: refund remains pending before approval
      evidence_authority: integration-test-and-database-state
    - id: REFUND-ONCE-002
      blocking: true
      requirement: approved refund is issued exactly once
      evidence_authority: provider-operation-record
    - id: REFUND-AUDIT-003
      blocking: true
      requirement: audit event records actor, tenant, amount and operation_id
      evidence_authority: audit-store-query
    - id: REFUND-SCOPE-004
      blocking: true
      requirement: no forbidden path changed
      evidence_authority: canonical-diff

The contract establishes boundaries and proof obligations. It does not establish that the implementation is correct or that the contract itself is complete.

Validate the contract before using it

Before execution, check that:

• criteria are specific, observable, and mutually consistent
• required evidence can be obtained from the environment
• each evidence authority can observe the relevant failure mode
• blocking and advisory classifications match risk
• no criterion can pass solely from the agent’s own report
• side effects, tenancy, migrations, and backward compatibility have not been omitted
• stop conditions cannot create endless repair
• completion authority is unique and auditable

A contract that fails these checks remains draft or needs_review. Parseable YAML is not approval.

A sprint contract bridges product intent and one verifiable unit

A high-level product specification usually describes user value and system behaviour rather than one executable run.

The sprint contract fixes:

• the behaviour to be delivered
• explicit out-of-scope work
• affected interfaces
• verification method
• expected artefacts
• risk and rollback
• completion authority

It specifies deliverables and boundaries without prescribing every line of code or search step. Too much detail turns the agent into an expensive macro. Too little leaves acceptance criteria to be invented after implementation.

Contracts require versions. When requirements, risks, or evidence obligations change, create a new version and record the reason. A newer criterion must not silently re-grade a run that was pinned to an older contract.

Passing should create a completion decision record

When mandatory evidence is available, the quality authority should commit an explicit decision record:

{
  "run_id": "run_882",
  "contract": "refund-approval@3",
  "evidence_snapshot": "evidence-set@sha256:example-only",
  "criterion_results": [
    {"id": "REFUND-STATE-001", "verdict": "pass"},
    {"id": "REFUND-ONCE-002", "verdict": "pass"},
    {"id": "REFUND-AUDIT-003", "verdict": "pass"},
    {"id": "REFUND-SCOPE-004", "verdict": "pass"}
  ],
  "verifier_versions": [
    "refund-integration-check@5",
    "provider-operation-check@2"
  ],
  "decision": "passed",
  "decision_authority": "refund-completion-gate@4",
  "decided_at": "2026-07-08T12:00:00Z"
}

The record cannot make incorrect evidence true. It makes the contract version, evidence snapshot, criterion verdicts, and decision authority reconstructable, so that passed is a controlled state transition rather than a green word in a user interface.

The harness control quadrants connect guidance and evidence

Quality controls can be organised along two dimensions:

1. Feedforward or feedback: does the mechanism constrain an action before it happens, or inspect what happened afterwards?
2. Computational or inferential: can deterministic software decide, or does the judgement require a model or a person?

Figure 10-2 | A mature control loop combines computational and inferential feedforward and feedback. It does not place every responsibility in one prompt or one judge.

Computational feedforward

Software enforces rules before execution:

• schema validation
• allowed paths
• dependency rules
• permission policy
• budgets
• required preconditions

Inferential feedforward

The harness gives the agent criteria that require judgement:

• task contracts
• design guidance
• domain rules
• examples and counterexamples
• review rubrics

Computational feedback

External sensors check the result:

• unit, integration and end-to-end tests
• type checking, linting and schema validation
• database-state assertions
• browser automation
• diff, secret and dependency scans

Inferential feedback

A model or person judges qualities that are difficult to formalise completely:

• code review
• user experience
• writing clarity
• policy interpretation
• human approval
• an independent LLM evaluator

The four quadrants are not a procurement checklist. A bounded data transformation may need only schema checks and deterministic tests. A high-risk cross-system task needs several gates, an independent verifier and perhaps human sampling. The aim is a closed loop, not a quality ceremony large enough to require its own catering budget.

Pre-execution gates reject errors that are not worth running

Some failures should be detected before a tool touches the environment.

A gate can check:

• whether arguments satisfy the schema
• whether the resource is inside task scope
• whether the actor has permission
• whether preconditions hold
• whether the risk class requires approval
• whether an architectural invariant would be violated
• whether sufficient budget remains
• whether the tool version is compatible with the runtime

If the agent attempts to edit payments/provider-adapters while the sprint contract explicitly excludes that path, the gate should reject the write. Waiting for a reviewer to discover it inside a 400-line diff is an expensive form of archaeology.

Gates require one policy authority. If a hook returns Allow, a policy returns Deny and the UI still carries an old session grant, the harness needs a fixed precedence. The most recent Boolean should not win by accident.

A quality gate should not maintain a second permission model. It calls the canonical policy decision defined in Part 09, records the policy version and decision identifier, and includes that result in the quality decision. A pre-execution gate can reject an invalid candidate; it cannot prove the post-execution outcome.

Architecture rules should execute, not merely appear in documentation

Documentation explains design intent, but an agent may not read the correct section before every change, and it may interpret the rule incorrectly.

Architecture constraints that often justify mechanical enforcement include:

• forbidden cross-layer imports
• tenant-scoped repository access
• backward-compatible API schemas
• immutable published migrations
• mandatory domain-service paths for sensitive operations
• secret-safe user-facing errors
• required updates to generated indexes or documentation

These rules can live in linters, type checkers, architecture tests, schema validators, CI gates, runtime assertions or policy engines.

Their responsibilities are complementary:

Documentation explains intent.
Executable constraints block violations.
Tests verify behaviour.
Trace records decisions and results.

Diagnostics should help the agent repair the violation

This message offers little useful direction:

Architecture violation.

A repair-oriented diagnostic is more specific:

ARCH-DEP-004
services/refunds/approval-handler.ts imports db/refund-record.ts directly.
Application services may depend on refund-service, not repository implementations.
Route access through services/refunds/refund-service.ts or another allowed service interface.

A useful diagnostic includes the rule, location, observed violation, expected invariant, reason and acceptable repair boundary. It need not prescribe one exact patch. The goal is to shorten the repair loop without turning the linter into an authoritarian code generator.

A verifier is not a universal judge, and quality authority is not one model

An agent may perform self-checks or reflection. Those signals can help repair work, but they are not canonical quality authority.

A verifier obtains, validates, and combines evidence against a pinned contract. Each criterion should use an authority that can observe its failure mode:

Criterion	Suitable evidence authority
Schema, types, import boundaries	Deterministic validator
Database state and side-effect uniqueness	External-state query or operation record
User journey	Browser, API, or integration runner
Security and tenant boundary	Policy check, adversarial case, and external state
Writing, UX, or design quality	Calibrated rubric judge or domain reviewer
Release exception	Explicit human or organisational authority

This is not a universal ranking. A human cannot observe database truth merely by being human, and a judge does not become independent merely because it uses another provider.

Verification independence has several dimensions

• Evidence independence: inspect artefacts and external state rather than only the generator’s summary.
• Execution independence: rerun required checks directly.
• Criterion independence: do not rewrite the rubric after seeing the output.
• Assumption independence: do not inherit unverified generator assumptions.
• Authority independence: the generator cannot write the verifier’s verdict.
• Operational independence: a verifier failure is not translated into pass by the UI.

A different model or provider may reduce some correlated bias, but it is neither necessary nor sufficient.

Canonical quality authority composes the decision

The quality authority answers:

• who may move a run from verifying to passed
• which criteria are mandatory and blocking
• whether missing evidence means unresolved, failed, or needs_review
• which exceptions may be approved and by whom
• whether an override retains the original failure, reason, expiry, and owner
• whether verifier or evidence-source failure is fail-closed

Absence of evidence is not a pass, and it is not always a failure. A high-risk blocking criterion normally remains unresolved or needs_review until evidence is obtained or an explicit authority accepts the risk.

An average score, judge confidence, or human approval must not silently erase an unresolved blocking criterion.

A verification portfolio should match the proof obligation

Schema checks, static analysis, unit tests, integration tests, end-to-end journeys, human review, and canaries are often drawn as a ladder. That picture is useful for showing increasing evidence, but it can imply two false rules:

1. Human review is always above an integration test.
2. Every high-risk task must mechanically execute the same seven stages.

A more accurate model treats verification as a portfolio of evidence families:

Figure 10-3 | Task risk determines the required evidence families. This is not a maturity ladder that every task must climb to the top.

Evidence family	Failure modes it can observe	Common blind spot
Schema and static	Shape, types, dependencies, explicit rules	Runtime behaviour
Unit and property	Local logic and boundary conditions	Cross-component state
Integration and external state	Databases, queues, APIs, and side effects	Complete user journey
End-to-end journey	Cross-interface behaviour and ordering	Rare or hard-to-reproduce production conditions
Security and adversarial	Permission, injection, tenancy, and abuse paths	General feature completeness
Human or domain review	Open-ended quality, risk acceptance, and exceptions	Stable high-volume repetition
Canary or shadow	Evidence close to real traffic and environment	All unobserved cases

Choose the portfolio from:

• affected components and external authorities
• side-effect reversibility
• tenant, financial, and permission blast radius
• which sensor can actually observe each invariant
• evidence freshness and environment fidelity
• verification cost, delay, and risk

A high-value refund may require integration state, an end-to-end approval journey, side-effect uniqueness, security cases, and domain sign-off. A spelling correction in Markdown does not need a production canary.

Termination rules belong in the completion contract. A green unit test cannot delete other required evidence, and an agent should not continue polishing non-blocking scores after every mandatory criterion has passed.

A rubric should carry evidence, authority, and uncertainty

A score of 8/10 gives little repair guidance and does not show which failure blocks completion.

An evidence-bearing criterion should return:

• criterion identifier and version
• verdict or score
• blocking or advisory status
• evidence references
• evidence authority
• exact location
• violated invariant
• uncertainty or confidence band
• evidence freshness
• repair boundary
• evaluator or grader version

For example:

{
  "criterion_id": "REFUND-E2E-003",
  "criterion_version": 4,
  "verdict": "fail",
  "blocking": true,
  "evidence_refs": [
    "test-report://refund_approval_journey/run-781",
    "db-snapshot://refunds/case-781"
  ],
  "evidence_authority": "integration-environment",
  "observed": "refund issued before supervisor approval",
  "expected": "status remains pending_approval",
  "location": "services/refunds/refund-service.ts:148",
  "uncertainty": "low",
  "repair_boundary": "state transition and approval check",
  "grader_version": "refund-domain-verifier@6"
}

A value such as confidence: 0.99 is useful only when it has been calibrated and genuinely has probabilistic meaning. Otherwise it is a decorative decimal.

The record supports a grade-and-revise loop:

Generate
→ Verify against pinned criteria
→ Return evidence-bearing diagnostics
→ Revise within allowed scope
→ Re-run affected checks and required regressions
→ Stop, escalate or accept

The loop also needs:

• pinned contract and rubric versions
• maximum iterations
• cost and time budgets
• plateau or oscillation detection
• regression checks
• human sampling or escalation
• retention of the best valid artefact

A release holdout or hidden case should not be fully exposed to the generator. Otherwise the system may learn to satisfy the grader rather than improve on the real task distribution.

A blocking criterion cannot be cancelled by an average score. Advisory scores may affect ranking or later improvement, but they must not turn security_fail into 92/100, pass.

Scope and pass-state gating remain part of quality

Quality concerns the surfaces the agent changed as well as the output it produced.

A task item may retain:

• allowed files, resources, and interfaces
• expected outputs
• required evidence
• current state
• owner and lease
• blockers
• verification status

passed is written only after the completion authority commits a decision record. It is not a field that the generator, one test runner, or the UI may set directly:

pending
→ in_progress
→ implementation_complete
→ verifying
→ passed | failed | needs_review

If the agent changes out-of-scope files, passing functional tests does not erase the violation. A test proves the behaviour it observes; it does not grant an exemption.

A pass transition should bind at least:

• expected run or state version
• pinned contract version
• evidence snapshot
• criterion results
• verifier versions
• decision authority
• override or exception record
• decision timestamp

An override does not delete the original failure. It creates a new exception record stating who accepted which risk, its scope, expiry, and follow-up action.

Tool-protocol integrity is also a quality condition. Every tool request requires a terminal result or an explicit cancellation or unknown outcome. If a failed call disappears from the transcript, trajectory evaluation receives a cosmetically repaired history.

Runtime completion and release evaluation are separate decisions

The runtime harness performs real work. Against one pinned completion contract, it decides whether a run may enter passed, failed, or needs_review.

The evaluation harness repeats cases and trials in an isolated environment, controls variables, records trajectory and outcome, runs graders, and produces release evidence across versions and task distributions.

Figure 10-4 | A runtime completion decision and a cross-trial release decision use different harnesses, states, and authorities. They may share versioned contracts but not production side effects.

Runtime harness

It:

• receives real requests
• builds context
• runs the agent loop and tools
• maintains canonical run state
• applies permission, quality, and budget gates
• gathers required evidence
• commits the completion decision
• produces user-facing outcomes and real side effects

Evaluation harness

It:

• loads a dataset snapshot
• creates an isolated or resettable environment
• pins model, harness, tools, policy, and budget
• runs multiple trials
• stores transcript, outcome, and infrastructure evidence separately
• applies deterministic, human, or model graders
• aggregates segments, variance, failure clusters, and regressions
• produces evidence for release or research claims

Trial validity is a first-class decision

The evaluation harness also determines whether a trial is valid:

• did the environment initialise correctly
• were dependencies, browser, database, and tools available
• was a timeout caused by the agent or the infrastructure
• did the grader complete
• is evidence missing or corrupt
• has the case become stale, contaminated, or inapplicable

infra_invalid should neither be counted automatically as model failure nor silently removed. It is reported separately, rerun when appropriate, and limits the conclusions that may be drawn.

Runtime evidence may feed dataset curation, and evaluation results may inform release policy, contracts, or harness changes. Those feedback paths require review and versioning; the evaluation harness must not mutate production state directly.

One runtime pass does not authorise a release, and one release-grade evaluation does not complete every production run.

Step, trajectory, and task are different proof obligations

Anthropic separates an agent evaluation into tasks, trials, graders, transcripts or trajectories, and outcomes. A transcript records the interaction in one trial; the outcome is the external state at its end. They are not substitutes.¹

Figure 10-5 | Step checks local operations, trajectory checks path invariants, and task checks external outcome. A valid alternative path need not reproduce one golden transcript.

Step-level evaluation

It observes:

• tool selection
• argument validity
• permission and approval
• result parsing
• local safety and scope violations

Valid steps do not establish task completion.

Trajectory-level evaluation

It observes:

• operation dependencies and ordering
• duplicate calls, loops, and unproductive exploration
• recovery after failure
• gate bypass
• budgets, stopping, hand-off, and approval
• mandatory path invariants

Trajectory evaluation should not require the agent to reproduce one golden path. Two different routes may both be safe, effective, and similarly efficient. What should be fixed are:

• non-bypassable gates
• approval before side effects
• required evidence
• prohibited operations
• maximum budget and deadline
• required causal ordering

Other acceptable paths may be expressed through equivalence classes, partial orders, or outcome constraints.

Task-level evaluation

It observes:

• completion-contract satisfaction
• external outcome
• side-effect uniqueness and tenant correctness
• required artefacts
• user or business objective
• blocking regressions

A correct task outcome does not prove that the trajectory was safe, and valid individual steps do not prove that the contract was complete.

A release claim is not the sum of three scores. It describes the evidence for a version across task distribution, blocking criteria, segments, variance, and trial validity.

Tool evaluation is wider than tool-call success rate

Tool quality can be split into five layers:

Layer	Main question
Definition	Are the name, description, schema, output and errors clear?
Selection	Does the agent choose the correct tool and avoid risky mis-selection?
Argument	Are the entity, resource, scope and parameters correct?
Execution	Are timeout, retry, idempotency, latency and side effects correct?
Outcome	Did the tool advance the task and produce sufficient evidence?

An HTTP 200 does not prove that the tool integration worked. The request may have targeted the wrong tenant, used the wrong currency, executed twice or returned enough raw payload to swamp the remaining context.

Tool-definition changes therefore require evaluation. Names, descriptions, schemas and error contracts influence selection and repair behaviour even when the backend implementation is unchanged.

Audit should progress from artefact presence to behavioural value

Harness audits can be organised into four levels.

Level 1: structural

• Do required artefacts exist?
• Are schemas valid?
• Can links, commands and references be resolved?
• Do policies contain obvious contradictions?

Level 2: executable

• Does a clean setup run?
• Do health checks and verification commands work?
• Are state transitions, resume, cancellation and handoff executable?

Level 3: behavioural

• Does task success improve?
• Does false completion decline?
• Are tool misuse, scope violations and rework lower?
• Are cost, latency and human intervention acceptable?

Level 4: ablation and longitudinal

• Does performance deteriorate when one harness component is removed?
• Is the old control still useful after a model upgrade?
• How do maintenance burden, drift and incident rate change over time?

Structural green is easy to over-read. The presence of AGENTS.md, an evals/ directory and quality-gates.yaml proves that files exist. Whether they are loaded, executable and behaviourally useful requires the later levels.

Evaluation data has a lifecycle

An evaluation dataset is not a permanent bag of correct questions.

Sources may include:

• curated happy paths
• edge cases
• production incidents
• user-reported failures
• adversarial cases
• synthetic variants
• consented and de-identified production samples

Each case should retain metadata such as:

• task class
• risk and severity
• source and provenance
• expected invariants
• allowed and forbidden trajectories
• deterministic checks
• human labels
• privacy classification
• introduction and retirement version

A useful split normally includes:

• a development set
• a release-gate set
• a hidden holdout
• an adversarial set
• a production shadow sample

If the agent and developers inspect the same release cases every day, the score gradually measures familiarity with the test set rather than capability on the real distribution.

Datasets also decay. Product flows change, tools retire, policies move, environments break and answers leak into searchable sources. Anthropic’s BrowseComp investigation found that public answers could enter an agent’s context through web search, contaminating results. Agent evaluation has a larger data boundary than static question answering.²

Cases also need lifecycle states such as active, quarantined, and retired. When a source or ground truth loses authority, quarantine the case before deciding whether to repair or retire it; a stale case should not continue influencing the release gate quietly.

OpenAI’s 2026 agent-improvement example similarly connects traces, human and model feedback, rerunnable evaluations, and proposed harness changes. The important boundary is retained evidence and review, not automatic deployment of the model’s own recommendations.³

Evaluators need evaluation as well

A grader is not correct by definition.

Calibrate it against:

• agreement with human labels
• false approvals
• over-rejection
• edge cases
• languages and task classes
• bias towards format, length or surface style
• variance across repeated trials

Also measure criterion-level confusion, ordering effects, verbosity bias, pairwise position bias, and sensitivity to evidence packaging. An uncertain evaluator may return abstain or needs_adjudication instead of manufacturing a confident score.

Code-based grading is usually fast and reliable but limited to formal conditions. Human grading is flexible and expensive. LLM grading can scale nuanced judgement, provided the rubric is clear, the grader is calibrated and it is not treated as the sole authority. Anthropic and OpenAI both place clear success criteria, task-specific cases, automated checks and human calibration near the centre of useful evaluation design.⁴⁵

Evaluation also has an economic boundary:

Expected value of additional evaluation
> evaluator cost + delay + false-positive rework

A high-risk payment flow justifies several layers of verification. A low-risk formatting task does not necessarily need three judges, a browser agent and human review on every run. A system can be solemn about quality while quietly bankrupting the latency budget.

A model comparison also compares the harness and environment

Agent evaluation measures something closer to:

Model × Harness × Task × Environment × Budget

OpenAI’s guidance for third-party evaluations argues that the harness, tools, context, budget and elicitation method can materially change agentic results. Reports should state whether they support a capability-ceiling claim, a controlled comparison or a safeguard-robustness claim.⁶

Anthropic measured infrastructure configurations that shifted agentic coding benchmark scores by several percentage points, sometimes by more than the leaderboard gap between top systems. CPU and memory enforcement, time limits and sandbox behaviour determine whether systems are taking the same test.⁷

A comparison should pin or report:

Where possible, use paired cases, randomised trial order, the same environment snapshot, and one invalid-trial policy. Infrastructure, grader, and agent failures should be reported separately; otherwise the ranking may merely reward the system that encountered fewer broken containers.

• model identifier and inference profile
• prompt and agent configuration version
• tool schema, version and permission policy
• context and compaction policy
• dataset snapshot
• grader and scoring code
• maximum turns, timeout, retries and concurrency
• CPU, memory, network, browser and retrieval access
• cost, latency and trial count

The native harness is a baseline, not a law of nature

A model may be adapted to a particular tool format, editing primitive, parallel-call behaviour or compaction policy. Moving it to a different harness can increase argument errors, repair turns or early stopping.

A useful compatibility matrix includes:

• native model with native harness
• candidate model with candidate harness
• reference model with candidate harness
• the same model with a tool variant
• the same model with a context or budget variant

That matrix helps distinguish a model regression from a harness regression, a compatibility penalty or an environment problem.

Reproducibility needs a run manifest

Each evaluation run should emit a traceable manifest:

{
  "run_id": "eval_2026_07_08_014",
  "model": "model-version-pinned",
  "agent_config": "refund-agent@3.4.1",
  "tool_catalog": "refund-tools@2.2.0",
  "policy": "refund-policy@7",
  "dataset": "refund-suite@2026-07-01",
  "contract": "refund-completion@3",
  "grader": "refund-grader@1.8.2",
  "evidence_schema": "quality-evidence@2",
  "max_turns": 24,
  "timeout_seconds": 600,
  "trials_per_case": 5,
  "trial_order_seed": 417,
  "environment_image": "sha256:...",
  "infra_validity_policy": "eval-validity@2"
}

The manifest cannot make stochastic model output identical. It allows the team to reconstruct the tested conditions and investigate differences without attributing every change to the model name.

A release gate should not look only at the mean score. It should also inspect:

• blocking cases
• severe regressions
• variance
• timeouts, abstentions and infrastructure failures
• unsafe-action rate
• cost and latency
• known limitations
• validity checks

A mean of 92 can conceal 100% success on ordinary payment tasks and 0% on cross-tenant isolation. The average is calm. The blast radius is not.

Walking the refund task through the evidence chain

Return to the refund-approval feature.

1. Contract

The sprint contract defines the threshold, state transition, audit event, scope, required evidence and completion authority.

2. Pre-execution gate

The agent attempts to modify a provider adapter. The scope gate rejects the operation and returns a diagnostic pointing to the permitted domain-service boundary.

3. Implementation

The agent updates the state machine, API and migration, and adds tests.

4. Verification portfolio

• schema, type and lint checks pass
• unit tests verify state transitions
• integration tests verify the database and audit event
• an end-to-end test runs the supervisor-approval journey
• operation records verify that the refund executes once
• the diff gate confirms that no unrelated files changed

5. Independent verifier

The verifier reads test reports, database state, changed files and operation records. It does not accept the agent’s summary as proof.

6. Runtime outcome

The run moves from verifying to passed only when all mandatory evidence is present.

7. Evaluation harness

The same version runs multiple trials across cases such as:

• normal approval
• rejected approval
• duplicate callbacks
• worker failure
• a cross-tenant identifier
• migration from the previous version

8. Release gate

Promotion is allowed only when blocking cases pass, no severe regression appears, and cost and latency remain inside the declared bounds.

This chain cannot prove that production will never fail. It turns “the agent says it is done” into evidence that can be replayed, compared and challenged.

Sixteen questions for a quality and evaluation harness

1. Does the task define its goal, scope, constraints, evidence and completion authority?
2. Has the product specification been converted into a testable sprint contract?
3. Which rules are enforced before execution?
4. Which architectural constraints are executable?
5. Do diagnostics include location, invariant, evidence and repair boundary?
6. Can the generator mark its own run as passed?
7. Who is the canonical quality authority?
8. Does the required evidence portfolio match the actual proof obligation?
9. Does the rubric require evidence rather than only a score?
10. Does the grade-and-revise loop have budgets, stop rules and escalation?
11. Does evaluation cover steps, trajectories and task outcomes?
12. Does tool evaluation cover definition, selection, arguments, execution and outcome?
13. Does the dataset have provenance, splits, versions, privacy controls and retirement rules?
14. Is the evaluator calibrated against human labels and edge cases?
15. Does a model comparison pin or report the harness, tools, budget, environment and grader?
16. Can every release decision be traced to a run manifest, raw evidence and validity checks?

If the system has only an agent.finished event followed by a score dashboard, a large part of the evidence chain remains invisible.

---

From theory to implementation: Part 10 checklist

Contract and gates

• Each high-value task defines goal, scope, constraints, outputs, acceptance criteria, evidence, failure and stop conditions.
• Sprint contracts are versioned, and active runs pin one version.
• Pre-execution gates check schema, scope, permission, preconditions, risk and budget.
• Suitable architecture rules run in linting, tests, CI, policy or runtime assertions.
• Diagnostics return a rule ID, location, observed evidence, expected invariant and repair boundary.

Verification authority

• The generator cannot write passed directly.
• The canonical quality authority, mandatory criteria and completion decision record are defined.
• The verifier reads external evidence rather than only the generator summary.
• The verification portfolio reflects side effects, reversibility, tenant boundaries, external authorities and business risk.
• Passing tests do not erase scope violations.
• Tool request, result, cancellation and unknown-outcome records remain correlatable.

Rubrics and revision

• Each rubric criterion has a version, evidence references, evidence authority, uncertainty, blocking status and repair guidance.
• Grade-and-revise loops have iteration, time and cost limits, plateau detection and escalation.
• LLM judges are calibrated with human labels, counterexamples and edge cases.
• High-risk criteria prefer deterministic checks, external state or human authority.

Evaluation design

• Runtime completion and release evaluation have separate responsibilities, states, authorities and environments.
• Trial validity distinguishes agent, infrastructure and grader failure, stale cases and contamination.
• Evaluation covers step, trajectory and task outcome where relevant.
• Tool evaluation covers definition, selection, arguments, execution and outcome.
• Audits cover executable and behavioural value, not only artefact presence.
• Evaluation data has provenance, metadata, splits, versions, privacy, decay and retirement controls.
• Incidents, user corrections and adversarial failures feed back into the suite.

Compatibility and release

• Model, harness, tools, policy, context, budget, environment and grader are pinned or reported.
• Every eval run emits a manifest and preserves transcript, outcome and grader evidence.
• Reports include variance, timeout, abstention and infrastructure failure, not only means.
• Release gates separate blocking cases, severe regressions, safety, cost and latency.
• Evaluation reports state whether they support a capability, controlled-comparison or safeguard claim.

Part 11 will examine how this evidence is produced, correlated and retained. Without traces, metrics, events, replay and version links, quality gates and evaluators see scattered screenshots. With a coherent observability model, a team can locate the failing turn, tool, worker and harness version.

References

Footnotes

1. Anthropic, Demystifying evals for AI agents, 9 January 2026. ↩

2. Anthropic, Eval awareness in Claude Opus 4.6’s BrowseComp performance, 6 March 2026. ↩

3. OpenAI, Build an Agent Improvement Loop with Traces, Evals, and Codex, 12 May 2026. ↩

4. Anthropic, Define success criteria and build evaluations, accessed 8 July 2026. ↩

5. OpenAI, Evaluation best practices, accessed 8 July 2026. The design guidance remains useful, although OpenAI states that existing hosted evals become read-only on 31 October 2026 and the Evals dashboard and API are scheduled to shut down on 30 November 2026. ↩

6. OpenAI, A shared playbook for trustworthy third party evaluations, 29 May 2026. ↩

7. Anthropic, Quantifying infrastructure noise in agentic coding evals, 5 February 2026. ↩