Harness Theory Part 12 | Patterns, anti-patterns and evolution: how to grow a harness and know when to remove it

Suppose an agent system has been running for a while.

It has a planner agent, a review agent, memory, more than a dozen tools, several hooks, two retry layers, a long rules file, task-specific prompt templates and a fallback path nobody is willing to dismantle.

Then the model changes. The tool provider adds native capabilities as well.

The team can no longer answer a few basic questions:

Which component still catches a real failure mode?
Which rule is merely a patch for the previous model?
Which review agent improves quality, and which one only adds tokens and latency?
Who still owns the memory store?
Which tools now duplicate runtime features?
Would quality actually fall if one of these layers disappeared?

A large component count can look like architectural maturity. If nobody can answer those questions, it may be a well-organised warehouse of technical debt instead.

The later stage of Harness Engineering is not about adding more parts. It is about building a method through which controls emerge from observed behaviour and risk, then become simpler, conditional or removable when their assumptions stop holding.

Behaviour-to-Harness derivation loop

Figure 12-1 | Start with a desired behaviour or observed failure, then derive the missing control, smallest mechanism, external evidence and deletion condition.

Do not start with a component list

The first questions in a harness design discussion are often:

Should we add memory?
Should we introduce MCP?
Do we need a multi-agent system?
Should there be another reviewer?
Should we buy an agent platform?

All of them arrive too early.

They jump to a solution before naming the behaviour that must change, the failure that must be prevented, or the evidence that would show the mechanism works.

A more defensible derivation order is:

Desired behaviour or observed failure
→ Protected invariant or missing capability
→ Authority and canonical state
→ Smallest viable mechanism
→ External evidence and operating cost
→ Owner, review trigger and retirement condition

This turns a harness component from something the architecture is expected to contain into something responsible for a named problem.

A simple example

Suppose an agent must continue a long task the next day without rescanning the repository.

Observed failures include:

a fresh session does not know what the previous one completed
earlier side effects cannot be confirmed
completed work is repeated
task state exists only in the transcript

The protected invariants are:

completed work is not treated as unstarted
external side effects have queryable receipts
a fresh session can reconstruct from durable evidence
only canonical authority advances task state

The smallest useful mechanism may be:

a structured progress ledger
a runtime-managed checkpoint
referenceable artefacts
session-bootstrap validation
side-effect operation identifiers

Only then is there a reason to add the long-running mechanisms discussed in Part 7. If one small ledger solves the problem, a multi-agent coordination platform is unnecessary.

Record why each control exists

A control record may contain:

control_id: durable-progress-ledger
failure_addressed: repeated work after session reset
protected_invariant: completed task state is not lost
authority: canonical-task-store
evidence:
  - fresh-session-reconstruction-test
  - duplicate-work-rate
cost:
  latency: low
  maintenance: medium
owner: agent-platform
review_triggers:
  - runtime upgrade
  - task model change
retirement_condition: native runtime state proves equivalent behaviour

This is not permanent residency for the component. It records why the control exists, who owns it, how it is verified, and when changed underlying capabilities should trigger review.

Mechanism economy

A failure mode may have several treatments. A useful default preference is:

use an existing deterministic control
improve the artefact or context
add an executable sensor
use a narrowly scoped semantic evaluator
add another agent or orchestration layer only when required

This is a default, not a doctrine. If a deterministic mechanism cannot express the semantic risk, or its maintenance cost exceeds that of a bounded evaluator, compare them with representative evidence.

A JSON Schema should reject malformed arguments without another model call. An architecture test should enforce dependency direction rather than leaving the rule in a long prompt for repeated interpretation.

Models are useful for semantics, uncertainty, and open exploration. They should not replace constraints that can already be encoded clearly.

Deterministic-first does not mean anti-agent

Deterministic-first fixes boundaries that must not drift, then gives the model discretion where judgement genuinely improves the result.

The following usually belong in deterministic control:

authentication and authorisation
tenant and resource scope
input validation
retry, timeout, and budget limits
state transitions
destructive or financial commits
required verification gates
audit and release authority

Model-driven decisions are more appropriate for:

decomposing ambiguous tasks
semantic routing
open-ended search strategy
generating candidate solutions
content judgement that cannot be enumerated fully

This does not trap an agent in a rigid workflow. It asks:

Which freedoms improve task outcomes, and which merely increase unpredictability?

Anthropic distinguishes workflows with predefined paths from agents whose paths are directed dynamically by the model, and recommends adding agentic complexity only when it demonstrably improves outcomes.[1]

Preserve the invariant, not necessarily the implementation

“Payment requires valid approval” is an invariant. It may be implemented by a policy engine, workflow gate, or controlled transaction service. A platform may later provide a better native implementation, but a product claim that it “supports approval” is not sufficient evidence to delete the invariant.

Similarly:

a custom retry wrapper may disappear, but attempt budgets and unknown-outcome control must remain
a secondary reviewer may disappear, but independent evidence for blocking criteria must remain
a memory store may be replaced by native session state, but the transcript must not become the sole state authority

Build-to-delete therefore does not mean fewer controls at any cost. It means:

Preserve the required invariant through the lowest-cost mechanism with the clearest responsibility.

Exceptions and escape paths need governance

A legitimate exception to a deterministic gate must not be a model bypass. It needs:

an explicit exception type
authority
scope
expiry
evidence
an audit record
a revalidation trigger

Otherwise flexibility becomes an unnamed second policy system.

Correct components in the wrong order still fail

A harness is shaped not only by whether it contains policy, hooks, memory, and verification, but by where each mechanism enters the loop and which decision may override another.

A stable order can be reduced to:

Receive event
→ Authenticate actor and resolve tenant
→ Load canonical state and control signals
→ Apply pre-discovery visibility and data policy
→ Build bounded context and capability snapshot
→ Call model
→ Inspect proposed operation
→ Canonicalise resource and arguments
→ Apply per-operation authorisation and approval
→ Execute or accept background work
→ Persist receipt, effect state and audit record
→ Append correlated observation
→ Verify terminal candidate
→ Commit completion, checkpoint or continuation

There are two policy boundaries:

Pre-discovery policy controls which data, tool metadata, and resources may be visible.
Per-operation policy authorises the concrete operation after resource and arguments have been canonicalised.

Collapsing both into one vague policy_check() allows visibility and final authorisation to substitute for each other.

Order carries security and reliability meaning:

authorisation before path canonicalisation can be invalidated by ../, symlinks, or aliases
a changed dynamic tool set with an old prompt catalogue creates impossible plans
compaction that removes side-effect receipts can cause duplicate payment or email
marking a task completed before verification leaks pass state
a scheduled trigger that bypasses identity and policy is another unauthorised entry point
a background completion returned through the original tool-result channel may be mistaken for a second synchronous result

A production harness may be distributed across event-driven services rather than one literal loop. Protocol order, authority, correlation, and state transitions must survive the decomposition.

Ten anti-patterns are ten forms of escaped responsibility

An anti-pattern matters because it hides a responsibility the system still needs to carry.

Anti-pattern	Surface implementation	Responsibility that disappeared
Prompt-only control	Permission, budget and retry rules live in the prompt	Runtime enforcement
Model self-certification	The model reports completion and the run passes	Independent verification
Blind retry	Every exception gets the same retry policy	Error classification, idempotency and reconciliation
Hidden control flow	Hooks and callbacks have no documented order	Traceability and authority
Tool equals function	A callable function is treated as a complete tool	Schema, permission, timeout, receipt and error taxonomy
Split-brain state	Planner, tool layer and verifier each store their own truth	Canonical state authority
Final-answer-only evaluation	Only the final text is graded	Trajectory, side effects, safety and cost
Separate fake evaluation pipeline	Evaluation uses a simplified path	Production parity
Context dumping	All history, documents and results enter the prompt	Selection, budget and freshness
Framework-first	A platform is chosen before the workload is defined	Failure mode and completion contract

The most dangerous case is not one component making a mistake. It is a team gradually becoming comfortable with the assumption that another layer will handle it.

The planner assumes the tool layer validates inputs. The tool layer assumes policy ran upstream. The verifier assumes the transcript is canonical state. Each layer owns a fragment of truth and no layer owns the responsibility.

A pattern catalogue is a vocabulary, not an architecture

This series has introduced patterns such as:

deterministic-first orchestration
fresh-context iteration
workspace isolation
task-portfolio delegation
review-feedback promotion
artefact pipelines
entropy management
build-to-delete

A catalogue gives repeated problems a vocabulary of composable solutions. It does not mean that every harness should install every pattern.

Each pattern should have a qualification record:

pattern: fresh-context-iteration
problem: quality degrades across long sessions
applicability:
  - task can checkpoint safely
  - startup reconstruction is reliable
counterconditions:
  - low-latency conversational task
required_authority:
  - canonical progress state
required_evidence:
  - fresh-session reconstruction test
known_costs:
  - startup latency
status: conditional
owner: agent-platform
review_by: 2026-10-01

Before adoption, answer:

Which failure mode or protected invariant does it address?
Which state and authority does it require?
What runtime, security, and operational costs does it add?
Which evidence would show that it works?
Which counterconditions or interaction effects matter?
When should it be retested, demoted, or removed?

Pattern status may be:

candidate
trial
adopted
conditional
deprecated
retired

A pattern that works only for one model, repository, or workload should remain conditional rather than becoming an engineering law. The catalogue is a governed living document, not an architecture shopping list.

Harnesses are not limited to code

Any task that exposes inputs, constraints, intermediate artefacts, evidence and acceptance conditions can be organised as a harnessed workflow.

A domain-agnostic artefact pipeline may be:

Analyse
→ Build task-specific instructions
→ Draft
→ Critique
→ Revise
→ Validate
→ Publish

It can support:

research reports
translation and localisation
financial documents
data cleaning
regulatory comparison
technical articles
diagrams and slide production

The important design choice is not the number of model passes. It is the separation of artefacts and authority between stages.

The critique stage may diagnose only, while revision applies changes. Validation should use independent rules, sources or reviewers rather than allowing the drafting agent to approve its own output.

Large artefacts can be split into parallel chunks, but every worker needs the same:

glossary
style contract
source snapshot
global outline
cross-chunk reference rules

Passing every local section does not prove the complete document has no duplication, contradiction or argument drift. Global validation remains necessary after assembly.

Capability-tiered coordination should follow risk and verifiability, not price alone

A workload may contain a small amount of high-judgement work and a large volume of mechanical reading, classification, transformation, and querying.

A stronger coordinator may handle:

task decomposition
ambiguity resolution
cross-worker conflict
coverage review
final synthesis

Less expensive or faster workers may handle:

bounded retrieval
format conversion
classification
structured extraction
local checks with an existing rubric

“Plan with the strong model, execute with the cheap model” is not a universal saving formula.

Routing should consider:

task difficulty
failure blast radius
evidence of worker capability
verifier coverage
escalation cost
data and tool permission
latency objectives
total cost rather than token price

Tiering is a poor fit when:

subtasks are tightly coupled
worker errors are hard for a coordinator or verifier to detect
every step requires high-level judgement
coordination and verification cost exceed the saving
failure blast radius is large
child output becomes trusted parent instruction without verification

Each routing rule should retain segment-level outcome, cost, and escalation evidence. Routing only by token price often sends the hardest-to-detect errors down the cheapest path and spends the saving during incident response.

Build, Buy, and Compose are boundary decisions rather than one system-wide choice

A harness rarely needs to be entirely built, entirely bought, or entirely assembled from open-source components. A more useful decision is made at each boundary:

canonical state
agent loop or runtime
sandbox
tool and protocol integration
approval and policy
evaluation
observability
operator interface
scheduling and queue
knowledge and artefact store

Build

Build is suitable when:

the core flow is highly differentiated
security, data, or deployment constraints are unusual
low-level runtime control is required
existing systems already provide many primitives
the team can own maintenance, on-call, and upgrades

The cost includes reliability, observability, protocol compatibility, migration, incidents, and staff turnover, not only initial development.

Buy or adopt

Adoption is suitable when:

standard flows are needed quickly
the workload fits the product’s assumptions
the vendor’s state model and control plane are acceptable
SLA, data governance, export, and incident processes meet requirements

Risks include:

lock-in
hidden state
policy or identity mismatch
data residency
pricing change
capability or protocol drift
difficult export or migration

Compose

A small set of components may form the stack:

Issue tracker
+ Task runner
+ Coding agent
+ Container sandbox
+ Existing CI

Replaceability may improve, but integration contracts, state hand-off, version compatibility, security patches, and incident ownership remain yours.

Write down shared responsibility

For each boundary, answer:

Question	Required answer
State authority	Who owns canonical state?
Security	Who owns identity, credentials, patches, and incidents?
Recovery	How are crashes, timeouts, and unknown outcomes handled?
Evidence	Which logs, receipts, evaluations, and audits can be exported?
Change	Who migrates API, schema, or pricing changes?
Exit	How is the component disabled, moved, or rolled back?

Tool selection needs an evidence profile, not one ladder

Marketing, READMEs, demos, tests, trials, and production use have different credibility, but evidence is not one simple staircase.

Evaluate at least five dimensions:

Source authority: official specification, repository, or secondary description
Environment fidelity: demo, sandbox, representative workload, or production
Failure coverage: happy path, crash, cancellation, recovery, and security
Recency: version, commit, and review date
Exitability: export, migration, open contracts, and rollback

A tool may have excellent official documentation but no failure evidence for your workload. Another may work in one production case while providing no export path. A single L5 should not flatten these differences.

Due diligence should ask:

Is it a coding agent, runtime, task runner, orchestrator, protocol, or full platform?
Does it create new canonical state?
How are state, permission, cancellation, retry, observability, and export handled?
What are the licence, security policy, release, migration, and bus-factor conditions?
Can it recover after a crash, and does cleanup work?
Can kill switch, export, and provider outage be exercised?
What are success, intervention, cost, and evidence completeness on your workload?

A material decision should include an exit drill: export a real run, restore its artefacts and audit, and show that the work remains intelligible after the provider is disabled.

A coding agent, orchestrator, MCP server, and full lifecycle platform occupy different layers. Combining them in one weighted ranking produces a precise score that cannot support procurement.

The OpenAI App Server exposes threads, turns, streaming, approval, and client integration as product interfaces, illustrating that productisation is about stable contracts and lifecycle rather than a polished interface alone.[2]

Adoption is not one route; each task class needs an operating mode

Harness adoption operating modes

Figure 12-2 | Different task classes may remain in different operating modes. Promotion and demotion require evidence gates; an organisation does not need one universal highest stage.

Adoption maturity is measured through verified outcomes, controlled delegation, and recovery capability rather than the number of agents running each day.

Seven stages can be used as operating modes.

Stage 0: Chat-only exploration

Suitable for questions, drafts, and local explanation. It lacks repository context, direct execution, and external verification.

Stage 1: Agentic reproduction

Choose work whose correct answer is already known and ask the agent to reproduce it without seeing the manual solution.

The purpose is calibration rather than saving time: task size, context, verification, and failure patterns. Mitchell Hashimoto’s adoption account likewise used reproduction of his own work to build judgement before increasing delegation.[3]

Stage 2: Bounded assistance

Work is divided into clear, verifiable units and begins to use repository tools, tests, scripts, and completion criteria.

Stage 3: Warm-start delegation

Research, triage, bounded experiments, or non-destructive review run during natural gaps and produce a starting artefact for the next work period.

Stage 4: Confident task outsourcing

Only task classes with strong success evidence run in the background. People inspect evidence at natural checkpoints rather than supervising every step.

Stage 5: Harness improvement loop

Repeated failures drive changes to guidance, tools, sensors, executable rules, and regression evaluations.

Stage 6: Portfolio-level orchestration

The organisation manages a task portfolio, priorities, concurrency, human attention, and integration windows.

A stage belongs to a task class, not to the company

The same team may:

run API documentation at Stage 4
keep a security migration at Stage 2
keep open-ended research at Stage 3
move low-risk triage to Stage 6

A promotion gate should require:

representative outcomes
verification coverage
failure containment
human-attention cost
incident and recovery readiness
an owner
a rollback or demotion path

When the model, tool, policy, task distribution, or risk changes, a task class may return to a more bounded mode. That is risk realignment, not organisational regression.

Anthropic’s Managed Agents engineering article makes the point directly: harnesses encode assumptions about model weaknesses, and those assumptions can go stale after a model upgrade; a previously useful context-reset mechanism became dead weight on a newer model.[4]

When there is no high-value task, verification is immature, or the review queue is saturated, choosing not to run an agent is a mature decision.

Productisation adds capabilities; it does not require the highest level

Harness productisation capability layers

Figure 12-3 | Files and Scripts, Structured Runtime, Operator Controls, Team Integration, and Managed Platform are composable capability layers. Each needs a user, responsibility, and exit condition.

Progress files, shell scripts, task JSON, and Git checkpoints may prove that a pattern works. They do not yet make it a dependable team product.

Layer 1: Files and scripts

progress file
initialisation script
task JSON
manual resume
Git checkpoint

Suitable for an individual or an early experiment.

Layer 2: Structured runtime

stable identifiers
typed state
event and protocol contracts
checkpoints
background work
versioned configuration

Layer 3: Operator controls

timeline
pause, resume, and cancel
approval queue
failure diagnosis
cost, quality, and attention signals
runbook links

Layer 4: Team integration

issue and project integration
roles and permissions
shared review queue
audit and retention
branch and workspace lifecycle
release gates

Layer 5: Managed platform

multi-tenant isolation
scheduling and queues
autoscaling
policy administration
upgrades and migration
SLO and incident ownership
billing and quotas

Layers are not a mandatory sequence or a value ranking

An internal tool may need durable runtime from Layer 2 and team integration from Layer 4 without a complete managed platform. A high-risk service may need operator controls before it expands its team surface.

For each layer, answer:

Who is the user?
Which lifecycle is being productised?
Which state or authority is added?
Who is on call?
How does export and migration work?
Which failure occurs without this layer?
When is further productisation uneconomic?

Productisation is not a script with a polished interface. The OpenAI Codex App Server turns long-lived threads, turns, streaming, approvals, and client integration into stable interfaces; Anthropic Managed Agents decouples session, harness, and sandbox so implementations may change behind stable boundaries. Both show that lifecycle contracts and replaceability are the core.[2][4]

A platform also creates governance debt: migration, backward compatibility, tenant isolation, operator training, and incident ownership. Add layers only when reuse, collaboration, or operational demand can pay those costs.

Templates reduce variety and can also freeze the wrong architecture

Organisations often build a small number of recurring topologies:

API business services
event processors
batch pipelines
data dashboards

Each topology can have a versioned harness template containing:

repository structure
architecture guides
executable dependency rules
approved toolchain
bootstrap, run and test workflows
observability contract
security baseline
deployment checks
evaluator rubric

The template narrows the legal solution space, allowing guides and sensors to cover more changes while leaving local implementation freedom. MartinFowler.com describes a harness as a regulating system that combines feedforward guides and feedback sensors; a topology template can package those controls into a reusable environment.[5]

Templates also drift.

They need at least:

a template version
a distinction between generated and owned files
an upgrade path
compatibility tests
a local override policy
a contribution path back upstream

Where domains differ substantially, or a template forces an unnatural architecture, variety reduction becomes accelerated replication of the wrong decision.

Promote repeated review, but treat it as a hypothesis first

Review feedback lifecycle

Figure 12-4 | Review feedback becomes a testable hypothesis before representative cases, false-positive checks, and failure injection justify a diagnostic, executable gate, or runtime invariant. Rules may also be demoted or retired.

A problem that appears repeatedly in code review, incidents, or agent runs should not depend forever on a person repeating the same comment.

Repetition does not mean the root cause is understood. A safer lifecycle is:

Observation
→ Cluster repeated evidence
→ Form a control hypothesis
→ Provisional guidance or checklist
→ Build diagnostic and representative cases
→ Evaluate false positive, false negative and repairability
→ Promote to executable gate when justified
→ Monitor, demote or retire

OpenAI’s harness-engineering account describes a similar movement: recurring feedback and inconsistency are promoted into repository knowledge, tooling, mechanical checks, and cleanup processes.[6]

Google Conductor’s Automated Reviews places specification and plan compliance, test execution, guidelines, and basic security checks into a post-implementation verification step, showing how review feedback can become rerunnable evidence rather than remain a comment.[7]

Promotion is most suitable when a problem is:

repeated and representative
bounded clearly
costly when violated
observable by a sensor
measurable for false positives and false negatives
able to produce an agent-actionable diagnostic
owned and reversible

Poor candidates include:

contextual design taste
one-off exceptions
incidents whose cause is not understood
judgement requiring cross-domain trade-offs
vague preferences without distinguishable good and bad cases

A promoted rule should retain:

source incident or review
protected invariant
applicability and exclusions
severity
diagnostic
test or evaluation
owner
version
expiry or review trigger
demotion and removal path

Promotion is not one-way. A gate with excessive false positives may return to a diagnostic or checklist, and an obsolete control may retire after underlying capabilities improve. Otherwise the improvement loop becomes a compost heap of rules.

Harness entropy grows out of success

As a harness produces more output, entropy accumulates gradually:

duplicate prompts and conflicting instructions
stale tool schemas
ownerless memory
dead configuration and feature flags
progress ledgers that disagree with the repository
documentation referring to removed code
duplicate hooks
evaluations testing obsolete behaviour
local wrappers duplicating native platform capability
three components maintaining the same invariant
deprecated components without a consumer inventory

OpenAI’s agent-first repository account notes that agents copy existing patterns, including inconsistent and undesirable ones, which creates a need for golden principles, mechanical checks, and ongoing cleanup.[6]

Entropy needs an inventory and dependency graph

The system should be able to query:

component, rule, or knowledge artefact
owner
protected invariant
upstream and downstream dependencies
active consumers
last use
last verification
cost
replacement candidate
deprecation status
retirement blocker

Without a consumer inventory, an apparently unused hook may fail only during a month-end job. Without an invariant map, merging two rules may erase one safety boundary.

Prefer deterministic cleanup sensors

Examples include:

broken-link and freshness checks
duplicate and dead-code analysis
schema-compatibility tests
architecture fitness functions
unused-tool telemetry
state-to-repository reconciliation
dependency and security scanning
feature-flag age
unowned artefact and rule reports

An LLM may propose suspected problems, clusters, and repair candidates. Deleting a tool, rule, memory item, policy, or artefact still requires evidence, dependency checks, and authority.

Give maintenance complexity a budget

Possible signals include:

active control count
controls without owners
rules beyond review date
duplicate authority
deprecated but active components
mean time to explain a run
upgrade and migration work
false-positive review cost
telemetry and storage overhead

These are triage signals rather than a maturity score. Reducing component count blindly can create the opposite failure because five clear components may be easier to maintain than one universal controller.

The target is complexity that cannot be explained, has overlapping responsibility, lacks evidence, or cannot retire.

Operational knowledge needs lifecycle and authority

When repository documents, specifications, skills, runbooks, and evaluation cases become operational knowledge, they need governance comparable to code.

Each knowledge artefact should retain:

owner
status
last verified date
canonical source and source authority
related code, schema, or policy
supersedes and superseded-by relationships
review cadence
generated or hand-authored status
consumers and retrieval surfaces
retention and archive rules

A documentation-gardening loop may be:

Scan
→ Classify stale, orphaned, duplicate or conflicting knowledge
→ Open a bounded repair
→ Verify against code or authority
→ Update indexes and links
→ Archive, supersede or delete safely

Google Conductor moved project context from ephemeral chat logs into version-controlled specifications and plans, then used Automated Reviews to verify implementation against those artefacts. Operational knowledge gains value from being versioned, referenceable, and verifiable rather than merely abundant.[7]

A knowledge score is a triage signal rather than a substitute for findings. A knowledge base scoring 92 may still contain an obsolete runbook that directs an agent to the wrong production credential.

Separate core, reference, watchlist, and expired material

Sending every new item into the core degrades retrieval and maintenance together.

Possible tiers are:

Core knowledge: general, high-signal, well sourced, and maintainable
Reference or source-only: useful to inspect but product- or version-dependent
Watchlist: emerging, disputed, or awaiting trials
Rejected or expired: obsolete, duplicate, incorrect, or more expensive to maintain than its value

Agents may fetch, summarise, deduplicate, and suggest a tier. Admission to core remains a governance decision through a human gate.

Archive and deletion are also different:

content referenced by historical runs, audits, or citations needs an immutable archive or tombstone
content with no consumers or legal and audit need, and with complete supersession, may be deleted
retrieval indexes and caches must invalidate when status changes

For each new item, ask:

Is the retrieval and decision value greater than the maintenance entropy it introduces?

An ecosystem radar is a decision queue, not a popularity ranking

The agent-harness ecosystem changes quickly. Placing every repository, product feature, and market number in the stable theory core turns it into archaeology.

A radar may use four rings:

Adopt: verified in the organisation’s environment and meeting safety and quality thresholds
Trial: controlled proof of concept with a hypothesis, owner, and exit condition
Assess: worth investigating but not yet tested
Hold: obsolete, overlapping, risky, unmaintained, or unsuitable

A ring is an organisational adoption decision rather than a global product score. The same product may be Adopt for one team and Hold for another because of data residency.

A candidate record may contain:

name: example-runner
category: task-runner
primary_use_case: issue-to-isolated-run
source_snapshot: 2026-07-08
licence: verify-from-primary-source
deployment_model: self-hosted
provider_dependency: none
state_authority: external-database
evidence_profile:
  source: official-repository
  environment: executable-example
  failure_coverage: incomplete
  recency: current
  exitability: untested
ring: assess
owner: platform-team
known_risks:
  - incomplete cancellation semantics
  - no verified export path
next_review: 2026-08-15

The record explains research status. It does not establish reliability or promote the product automatically.

A promotion path may be:

Discovery
→ Primary-source review
→ Architecture classification
→ Sandbox trial
→ Failure injection
→ Security and data review
→ Measured comparison
→ Exit drill
→ Adopt

GitHub stars, community attention, awesome lists, and polished demos can prioritise Assess work but are not production evidence. An awesome list should be retained with a snapshot commit and review date, while each tool returns to primary sources for verification.[8]

Landscape snapshots need dates and review-by dates; coding agents, runtimes, orchestrators, task runners, platforms, and protocols should be compared within their layers. When a product is renamed, acquired, abandoned, or absorbed by a model-native capability, it should be possible to demote, replace, or retire it.

The radar’s value is not keeping up with every new noun. It makes “why have we not adopted this?” a defensible answer.

Every component needs an assumption ledger

A harness component is not a law of nature. Each one contains a claim:

Without this component, the model, tool, or environment will fail in a particular way.

An assumption ledger may contain:

component: secondary-review-agent
version: reviewer-7
protected_invariant: cross-module regressions require independent evidence
assumption: primary agent and deterministic suite miss a material class of regressions
failure_addressed: unverified integration changes
dependencies:
  - integration-test-suite@12
  - completion-contract@5
baseline: primary-agent-plus-deterministic-suite
evidence:
  - eval-run-2026-07-08
segments:
  - high-risk-invoice
  - ordinary-invoice
cost:
  latency: high
  tokens: high
  operations: medium
risk_floor:
  security_non_inferiority: required
ablation:
  method: shadow-disable-reviewer
  primary_variable: reviewer
  success_margin: no material loss on blocking criteria
decision: conditional
owner: quality-platform
last_tested_model: model-profile-v7
next_review_trigger:
  - model upgrade
  - verifier coverage increase
  - task distribution change
retirement_condition: equivalent evidence with lower cost

A useful ledger answers:

Which invariant does the component protect?
Which weakness is assumed to remain?
Which other components does it depend on?
What are the baseline and representative segments?
Which evidence supports it?
What cost and failure surface does it add?
Which safety or quality floors may not decline?
Who may change the decision?
When should it be retested, demoted, or retired?

decision: retain is not permanent residency. It means that evidence supports retention under a particular model, task distribution, verifier, and cost environment.

Review triggers may include:

model upgrade
tool or runtime improvement
changed context window or strategy
improved codebase architecture
changed task distribution or risk mix
cost or latency pressure
evaluator recalibration
native capability replacement
incident or near miss

Anthropic’s 2026 Managed Agents engineering article provides a concrete example: context resets needed by an older model became dead weight for a newer model. The lesson is not that harnesses are useless, but that the assumption ledger must reopen.[4]

Build-to-delete removes a mechanism while preserving its invariant

Harness ablation and retirement lifecycle

Figure 12-5 | A component moves from assumption, baseline, experiment, and segmented evidence to Retain, Simplify, Condition, Replace, or Retire. Consumers migrate and the protected invariant is verified after removal.

When designing a component, prepare:

feature flag or shadow mode
stable or replacement interface
independent metrics and evidence
clear ownership
dependency and consumer inventory
rollback path
an executable ablation experiment
deprecation, migration, and removal plans

Ablation is not casually turning off a flag

A basic flow is:

Define protected invariant and baseline
→ choose representative tasks and segments
→ disable, bypass or replace one primary mechanism
→ run shadow, paired, canary or isolated trials
→ compare quality, safety, cost and failure modes
→ inspect uncertainty and interaction effects
→ retain, simplify, condition, replace or deprecate
→ migrate consumers
→ verify invariant after removal

Changing one primary variable is a useful default, but components may interact with prompts, verifiers, tools, and state schemas. Where interactions matter, use paired trials, factorial experiments, or staged migration. The aim is to avoid changing five things and receiving one unexplained green score.

Compare at least:

blocking criteria and task success
safety violations and hard invariants
segment-level quality
trajectory efficiency
latency and critical path
token and monetary cost
human intervention
maintenance burden
incident and unknown-outcome rate
trial validity and uncertainty

A non-inferiority floor comes before average benefit

A reviewer whose removal reduces average pass rate by 0.2% may still be essential if the loss is concentrated in high-risk invoices or a cross-tenant or financial invariant loses protection. The average does not have decision authority.

An ablation decision should separate:

mandatory safety and quality floors
allowed non-inferiority margin
segment
confidence and uncertainty
operational saving
migration cost

Decisions extend beyond Retain and Remove

Retain: still load-bearing.
Simplify: preserve the invariant with fewer layers or lower cost.
Condition: enable only for a risk or task segment.
Replace: move the invariant to a native capability or clearer mechanism.
Deprecate: stop new consumers and begin migration.
Retire: consumers are removed and evidence shows the invariant still holds.

High-risk, regulated, and complex systems may need more control. The candidate for deletion is a mechanism with no measured value, overlapping responsibility, or an invalid assumption, not the protected invariant represented by permission, tenant boundaries, audit, or independent verification.

Native capability is a replacement candidate, not a deletion command

When a model, runtime, or platform adds a native feature, run parity, failure, and migration tests:

Is the meaning actually equivalent?
Do authority and audit remain?
Are failure, cancellation, and recovery complete?
Can old data and state still be read?
What happens during provider outage?
How do cost and lock-in change?

After those checks, the local mechanism may be replaced. Before them, “the platform has this now” is only a new assumption.

Build-to-delete maturity is not the number of deleted components. It is the ability to prove safely that the protected behaviour survives when a mechanism leaves.

A complete evolution example

Suppose a team wants an agent to review internal invoices.

Version one: make it work

The team adds:

a long system prompt
an invoice lookup tool
an email tool
a reviewer agent

Problems appear quickly:

The agent sometimes treats a test invoice as a live invoice.
An email completes but the response times out, so a retry sends it twice.
The reviewer sees only the final answer, not the tool side effect.
The rules file keeps growing.

Derive controls from failures

Instead of adding more prompt text, the team introduces:

tenant and environment scope
an idempotency key and email receipt
canonical task state
a completion contract
tool trajectory verification
an approval gate for external email

Bounded adoption

Known invoices are reproduced first. The system then handles non-destructive review before high-confidence task classes are allowed to run in the background.

Productisation

Progress files and scripts become:

durable runs
an operator timeline
an approval queue
failure diagnosis
team roles and audit

Feedback promotion

A repeated tax-number formatting error becomes schema validation.

A special contractual clause that depends on legal interpretation remains a human review point rather than becoming a brittle rule.

Ablation

After the model and deterministic integration suite improve, the team disables the secondary reviewer in a controlled comparison.

If integration errors remain stable while cost and latency fall, the reviewer can be simplified or removed. If errors rise only for high-risk invoices, the reviewer remains conditional on that risk tier.

The maturity of this harness is not the number of agents left at the end. It is that every control has a failure, evidence, owner and exit condition.

The final maturity test

A mature harness is not necessarily the one with the most components or the greatest number of autonomous agent-hours.

It usually has:

clear boundaries and authority
one canonical state
classified and recoverable failures
external evidence for important behaviour
correlated and observable execution
replaceable components
owned and fresh knowledge
patterns with applicability and counterconditions
retestable assumptions
feedback that can be promoted and demoted
queryable consumers and dependencies
mechanisms that can retire safely
protected invariants that survive simplification

The first eleven parts covered context, capability, runtime, reliability, safety, quality, and observability.

The final judgement is:

A harness is a continuously calibrated control system, not a component collection with no exit door.

When a team can explain why each component exists, who owns it, how its value is demonstrated, what depends on it, and when it should be simplified, replaced, or retired, the harness has moved from scaffold to maintainable engineering capability.

The more precise objective is a minimum sufficient harness:

sufficient to protect required invariants
sufficient for representative work
sufficient for recovery and investigation
free of redundant, overlapping, or ownerless controls
recalibrated when models and platforms change

From theory to implementation: Part 12 checklist

Every major component has a failure or protected invariant, authority, owner, evidence, and retirement condition.
Deterministic controls fix safety, state, budgets, and pass authority; model freedom is reserved for semantic judgement.
Build, Buy, and Compose decisions are made per boundary, with shared responsibility, export, outage, and exit drills.
Each task class has its own adoption mode, promotion or demotion gate, and human-attention capacity.
Patterns, templates, rules, knowledge, and ecosystem candidates have versions, status, consumers, review dates, and demotion paths.
Repeated feedback becomes a testable hypothesis before promotion is decided through false-positive, false-negative, and repair evidence.
Expensive or overlapping components have assumption ledgers, segmented baselines, safety floors, ablation experiments, and migration plans.
After deletion or replacement, the protected invariant is reverified rather than inferred from lower cost alone.

This series now completes the theoretical map of harness engineering. Implementation should begin with one bounded workload, a named failure mode, canonical state, and an executable completion contract rather than an all-purpose agent city.

References

Anthropic, Building effective agents, 19 December 2024.
OpenAI, Unlocking the Codex harness: how we built the App Server, 4 February 2026.
Mitchell Hashimoto, My AI Adoption Journey, 5 February 2026. This is a first-person adoption account used as one practical path rather than a universal maturity model.
Anthropic, Scaling Managed Agents: Decoupling the brain from the hands, 8 April 2026.
Birgitta Böckeler, MartinFowler.com, Harness engineering for coding agent users, 2 April 2026.
OpenAI, Harness engineering: leveraging Codex in an agent-first world, 11 February 2026.
Google Developers Blog, Conductor Update: Introducing Automated Reviews, 13 February 2026.
AutoJunjie, Awesome Agent Harness, repository snapshot reviewed 7 July 2026. Used only as an ecosystem discovery index; individual tools require primary-source verification.