Harness
The execution layer around a model
In agentic AI, a harness is software around a model that connects it to context and tools and controls execution. Sources differ on whether this means the control loop or the wider non-model runtime. [4] [5] Editorial interpretation
On this page
Definition comparison
ASD advisory
Institutional; security and governance
ASD uses harness for the non-LLM components of an agentic system, including connections to tools, data, memory and planning workflows, privileges and the control loop. [4]
SWE-agent architecture
Peer-reviewed; agent-computer interface
SWE-agent supplies a task-oriented interface for a language-model agent to navigate repositories, edit files and execute tests. This informs the runtime boundary; the paper is not treated as a general definition of Harness. [15] Editorial interpretation
Managed Agents architecture
First-party; product boundary
Anthropic separates session, harness and sandbox. Its harness drives the loop and routes model-requested tool calls; the sandbox executes them. [5]
Conceptual proposal
Preprint; not established consensus
Macedo proposes an operational boundary against SDKs, frameworks, IDE plugins, evaluation scaffolds and orchestrators. This is one author’s definition proposal. [6]
Scope
Focus here is an agent runtime harness. An evaluation harness instead runs tasks and measures results; a framework or SDK supplies construction abstractions and may include a runtime. Shared naming does not make these identical objects. [6] Editorial interpretation
Includes
- Repository example: a running loop that assembles context, dispatches tools, records outcomes and applies stop or permission rules qualifies under our working definition. [14] Editorial interpretation
- A harness can realize fixed workflows as well as dynamic agent loops. Describe its actual control responsibilities rather than inferring them from a product name. [4] [5] Editorial interpretation
Excludes and counterexamples
- Model weights alone do not supply the surrounding runtime. [4] Editorial interpretation
- An SDK import or workflow diagram alone does not demonstrate a running harness. An evaluation runner is labeled evaluation harness unless it also has the claimed agent-runtime responsibilities. [6] Editorial interpretation
Different usages and open boundaries
- ASD’s broad organizational boundary and Anthropic’s separated runtime components answer different questions. Neither establishes that a harness is a kind of process. [4] [5] Editorial interpretation
- A runtime may be delivered inside an SDK or framework; packaging and execution responsibility are separate axes. [6] Editorial interpretation
Harness vs Workflow
Workflow describes how work proceeds; Harness describes how execution is governed. This is an editorial distinction between responsibilities, not an official two-term taxonomy. A workflow can run within a harness, and one harness can realize several workflows. [3] [20] [14] Editorial interpretation
The unqualified word harness is not exclusive to language models: NIST describes a healthcare test harness that executes and orchestrates testing services. Keep that usage separate from ASD's specifically agentic-AI scope. [20] [4] Editorial interpretation
| Dimension | Workflow | Harness |
|---|---|---|
| Main question | What activities run, with which dependencies and control flow? [3] | Which interfaces, permissions, execution policies and checks govern those activities? [4] |
| LLM required? | No. Steps can involve people, deterministic code, LLM calls or agents. [14] | No for the broader term. An agentic AI harness surrounds an LLM/agent; NIST's testing usage has another scope. [20] [4] |
| Order and loops | Sequence, branching, concurrency and iteration describe the process; a workflow need not be a simple linear chain. [3] | May implement a loop or realize workflows; governance does not prescribe one universal step order. [20] [5] |
| Checks and recovery | Validation and retry branches may be steps in the process. [13] [14] | May enforce validation and retry budgets across calls, hold state and record failures. Exact capabilities must be specified. [4] [14] |
| Typical design concern | Missing dependencies, unsuitable branches or an incorrect task decomposition. [3] | Excessive permissions, unbounded execution, lost state or inadequate failure records. [4] [5] |
Schema extraction: an illustrative experiment
For a paper and dataset, a proposed workflow is candidate discovery -> evidence retrieval -> attribute extraction -> deterministic schema assembly -> validation. Model calls and code can occupy different stages; this describes the task logic. [14] Editorial interpretation
The proposed harness fixes the input snapshot and adapters, checks outputs against the declared JSON Schema dialect, and tracks field-to-evidence support separately. A structurally valid JSON object does not by itself prove that a field exists in the source or that an inferred attribute is true. [19] [14] Editorial interpretation
Example policies include a three-attempt limit, explicit token/time budgets, checkpointing, timeout and truncation handling, and recording model/configuration versions, prompts, outputs and failure reasons. Allow unknown only where the output contract permits it. These are proposed experiment controls, not mandatory properties of every harness. [14] Editorial interpretation
For this experiment, an orchestrator schedules model/replicate/device assignments and task dependencies; the harness supplies the configured runtime and controls. They can overlap: NIST's test harness itself includes orchestration. This is a responsibility split, not three mutually exclusive software boxes. [20] [14] Editorial interpretation
Overlap, reliability and reproducibility
The same extract -> validate -> retry code can play both roles. Describing its branch and loop structure is a workflow view; enforcing a shared attempt budget, validation contract and failure record is a harness view. The distinction follows responsibility, not Python syntax. [14] Editorial interpretation
The harness metaphor suggests bounded, observable and recoverable execution. It does not establish that the model becomes deterministic or always correct, or that every harness actually enforces its advertised policies. An implementation must test those policies and state the limits of its guarantees. [4] [14] Editorial interpretation
Recording inputs, configuration, model versions and seeds supports an auditable experiment setup. It is not a universal promise of identical outputs: OpenAI's archived 2023 example explicitly allows differing responses even when seed, request parameters and system fingerprint match. That provider-specific example is not current API guidance for every model. [21] [14] Editorial interpretation
Operational criteria
Repository test: locate the running component responsible for model invocation, context/tool mediation, execution feedback and termination or privilege policy. State where session storage and tool execution live. If the software only constructs that component, label it a framework or SDK. [14] Editorial interpretation
Concept relationships
Only is-a means a strict subtype. Uses and supports/orchestrates describe composition and control.
Harness supports/orchestrates Workflow
A harness realizes or coordinates workflows; runtime-to-process is not strict subtype nesting. [4] [5] Editorial interpretation
Repository working definition
Repository · Proposed / not certified
Use agent harness for the model-surrounding execution layer that mediates context, tools, feedback and execution policy. For the proposed experiment, Workflow emphasizes flow and dependencies; Harness emphasizes controls and boundaries. Specify the policies and guarantees actually implemented. [14]
No consensus decision recorded.
Sources and research gaps
Research notes
Peer-reviewed definition gap: this review did not establish a directly applicable peer-reviewed consensus definition of agent harness. The conceptual paper is a June 2026 preprint, not promoted to tier 2.
The NIST testing description establishes an additional, non-LLM usage; it does not supply a universal Harness definition. The flow/governance table and experimental example are editorial synthesis and an unratified internal proposal. The OpenAI reproducibility recipe is explicitly archived.
- Agentic AI harnesses
Australian Signals Directorate. Tier 1 / Institutional security advisory
Published and reviewed 11 September 2026. What is a harness; Model vs harness. Accessed 2026-10-10.
Context: Organizational security and governance of agentic AI; not a universal taxonomy.
Verification: Original text and publication record checked 2026-10-10.
- High-Level Requirements: Health IT Test Infrastructure
NIST Information Technology Laboratory. Tier 1 / Institutional engineering description
Created July 14, 2016; updated April 8, 2026. High-Level Requirements; Test Infrastructure System Components, Test Harness Component. Accessed 2026-10-11.
Context: Healthcare interoperability and conformance testing, not a universal definition of every harness. Demonstrates a non-LLM execution and orchestration harness.
Verification: Read the original NIST page, including the test-harness/workflow-management descriptions and creation/update dates.
- Workflow Patterns
Wil M. P. van der Aalst, Arthur H. M. ter Hofstede, Bartek Kiepuszewski and Alistair P. Barros. Tier 2 / Peer-reviewed journal article
Distributed and Parallel Databases 14(1), 5-51; DOI 10.1023/A:1022883727209. Section 2, printed page 9; static-control-flow scope on page 8. Accessed 2026-10-10.
Context: Business workflow control-flow patterns; not an exhaustive theory of every process.
Verification: Final published PDF read; peer-review status, journal and DOI checked against the TU/e publication record.
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press. Tier 2 / Peer-reviewed conference paper
Advances in Neural Information Processing Systems 37, NeurIPS 2024; DOI 10.52202/079017-1601. Abstract and agent-computer interface architecture. Accessed 2026-10-10.
Context: Software-engineering agent architecture; relevant runtime evidence, not a universal harness definition.
Verification: Published NeurIPS proceedings PDF and conference record checked 2026-10-10.
- Scaling Managed Agents: Decoupling the brain from the hands
Anthropic. Tier 3 / First-party engineering blog
8 April 2026. Session, harness and sandbox architecture. Accessed 2026-10-10.
Context: Architecture of a specific hosted agent product.
Verification: Original text and publication record checked 2026-10-10.
- What makes a harness a harness: necessary and sufficient conditions for an agent harness
Sanderson Oliveira de Macedo. Tier 3 / Preprint; peer review not established
arXiv:2606.10106v1, 8 June 2026. Abstract; conceptual boundaries and inclusion/exclusion test. Accessed 2026-10-10.
Context: A proposed operational definition in coding-agent engineering; no consensus claim.
Verification: Original text and publication record checked 2026-10-10.
- Building effective agents
Anthropic. Tier 3 / First-party engineering blog
19 December 2024; live page includes subsequent tooling updates. What are agents?; Building blocks, workflows, and agents. Accessed 2026-10-10.
Context: Company-specific distinction between predefined LLM workflows and dynamic agents.
Verification: Original text and publication record checked 2026-10-10.
- JSON Schema: A Media Type for Describing JSON Documents
Austin Wright, Henry Andrews, Ben Hutton and Greg Dennis. Tier 3 / First-party specification; IETF Internet-Draft, not an RFC
Draft 2020-12 Core, draft-bhutton-json-schema-01, 16 June 2022. Sections 4.2, 4.3, 4.3.1 and 4.3.4: instance, schema, keywords and meta-schema. Accessed 2026-10-10.
Context: The JSON Schema dialect and instance data model; assertions and annotations have different roles.
Verification: Dated Core text and definitions checked. The document identifies itself as an Internet-Draft with intended Informational status; it is not relabeled a final IETF standard.
- How to make your completions outputs consistent with the new seed parameter
Shyamal Anadkat / OpenAI. Tier 3 / Archived first-party engineering recipe
November 6, 2023; archived recipe. Implementing consistent outputs; seed and system_fingerprint. Accessed 2026-10-11.
Context: Historical Chat Completions example for 2023 model versions; not a current feature-support promise for all APIs or a statement about every LLM.
Verification: Read the original recipe, its archived warning, date and explicit statement that matching seed, parameters and fingerprint does not guarantee identical outputs.
- Repository terminology guide
Repository internal proposal. Tier 4 / Repository internal proposal; not certified
Repository proposal 1.3 / not certified. Term-specific proposal sections and editorial policy. Accessed 2026-10-12.
Context: Internal proposals and editorial interpretations, not external consensus.
Verification: Versions 1.1 and 1.2 add the Schema proposal and the flow/governance experimental example while preserving prior proposal sections. All remain unratified; no adoption decision claimed. Version 1.3 changes branding and certification scope only; it does not certify any proposal.
Revision record
- 1.0 / 2026-10-10
Researched and rewritten with attributed definitions, typed relations and an unratified Repository proposal.
- 1.1 / 2026-10-11
Add the cited flow/governance comparison, overlap boundaries and illustrative experiment; Repository meaning remains proposed.
- 1.2 / 2026-10-12
Scope the working definition and certification labels to this repository under Open AI Glossary. Citations and proposed status remain unchanged.