Skip to content

Benchmarking the Harness

Every public agent leaderboard ranks a model. Around that model sits a scaffold: the code that issues tools, builds context, decides when to retry and when to stop. The scaffold usually goes unreported. Three controlled results published this year show what that omission costs.

Vats and Golev ran two models across three open-source harnesses on a stratified 50-task subset of Terminal-Bench Pro. Their finding: "harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points." Failure patterns replicated across models. Goose failed on reasoning, OpenHands-SDK on verification and turn limits, OpenCode on idle loops and timeouts. Those are properties of the harness, not of the model running inside it.

A 2026 survey of agent and harness design found the same effect on WebArena. Holding the backbone fixed at GPT-4o, accuracy "ranges from 13.1% in the model-only baseline to 54.6% with WebOperator, a 41.5% span; even among named harnesses alone, it ranges from 19.2% with LM-TS to 54.6% with WebOperator."

A third paper formalises this as the Binding Constraint Thesis: for long-horizon tasks across models of comparable frontier capability, performance variance is governed more by harness configuration than by model choice, so current protocols "systematically misattribute harness-level gains to model improvements."

If the scaffold moves the number that much, the scaffold is worth measuring on its own. This is a survey of what exists to measure it with.

What a harness is⚓︎

A harness is everything between the user's request and the model's weights. It decides which tools the model can reach and how their outputs are formatted; how much of the conversation and the filesystem enters the context window; whether the model plans before acting; how many turns it gets; what happens on a tool error; whether a human is asked before a destructive action; and what the model is allowed to touch in the first place.

Claude Code, Codex CLI, OpenCode and Cursor are harnesses. So are Glean, Microsoft 365 Copilot and Agentforce. So are LangGraph, the OpenAI Agents SDK and Google's ADK, except there you assemble the harness yourself. The four classes have almost nothing in common architecturally, which is the central obstacle to comparing them and the reason most of this survey's instruments cover exactly one class.

The shape of the field⚓︎

Rigorous, reproducible instruments cluster in coding and tool use. Terminal-Bench, SWE-bench Pro, the Berkeley Function-Calling Leaderboard and τ-bench are all open, re-runnable, and programmatically scored. Move outward and the rigour drops fast. Knowledge work has one serious instrument. Enterprise deployment has none, only analyst frameworks. Onboarding cost, connector depth and governance controls are measured only by declaration.

That is the coverage problem. The second problem is design. Almost every instrument varies the model and holds the scaffold fixed. That is the right design if you are ranking models, which is what most of these were built to do. Only three things in the survey treat the harness as the variable under test: the Holistic Agent Leaderboard, the Scaffold Effect paper, and practitioner bake-offs.

The instruments⚓︎

Read the last column first. It determines whether an instrument can say anything about a harness at all.

Infrastructure and cross-cutting⚓︎

Instrument Measures Unit of task Scoring Harness treated as
Holistic Agent Leaderboard (HAL), Princeton PLI Accuracy against cost across 9 benchmarks One task from an adapted benchmark Inherited programmatic, plus LLM log inspection Variable — models × scaffolds × benchmarks
Terminal-Bench 2.0 + Harbor + registry, Laude Institute Terminal task completion, end state One instruction in a Docker sandbox Programmatic end-state tests Variable — agent adapters
HELM / HELM Capabilities, Stanford CRFM Accuracy, robustness, bias, efficiency One scenario or prompt Multi-metric grid Fixed, mostly implicit
CUBE, a proposed unification standard Benchmark interoperability Adapter-normalised trajectory Parity experiments Variable, by design
Inside the Scaffold 12 scaffold dimensions Architecture description Qualitative taxonomy The object of study

Coding and terminal⚓︎

Instrument Measures Unit of task Scoring Harness treated as
SWE-bench Verified Repository bug-fix resolve rate One GitHub issue Fail-to-pass and pass-to-pass tests Undisclosed; varies by submission
SWE-bench Pro, Scale AI Long-horizon software engineering One commit-derived task Resolve rate, no regressions Disclosed per submission, not controlled
The Scaffold Effect, Vats & Golev Harness token cost and failure fingerprints One Terminal-Bench Pro task pass@1, tokens per solved task, failure taxonomy Variable — 2 models × 3 harnesses
Agentic Harness Engineering Automatic harness evolution Terminal-bench-long task Pass-rate uplift at fixed model The thing being optimised

SWE-bench Verified is the cautionary tale of the group. OpenAI retired it after finding frontier models could reproduce the gold patches from memory; its page is now titled "Why SWE-bench Verified no longer measures frontier coding capabilities."

Tool use and integrations⚓︎

Instrument Measures Unit of task Scoring Harness treated as
Berkeley Function-Calling Leaderboard v4 Function calling, multi-turn, web search, memory One call or trajectory AST match, executable check, relevance accuracy Fixed; ranks models
MCP-Universe, Salesforce AI Research Real MCP servers across 6 domains, 11 servers One multi-step MCP task Execution-based evaluators, no LLM judge Partly variable
MCP-Bench 28 servers, 250 tools, cross-tool composition Multi-step tool task Rubric LLM judgement plus execution Fixed
LangChain agentevals Agent trajectory quality Message and tool-call list Trajectory match — strict, unordered, subset, superset — plus LLM judge The harness is the agent
Google ADK evaluation Tool trajectory and response EvalCase → Invocation tool_trajectory_avg_score, response_match_score The harness is the agent

Knowledge work and documents⚓︎

Instrument Measures Unit of task Scoring Harness treated as
GDPval, OpenAI Real deliverables across 44 occupations One document artifact Human expert pairwise, plus rubrics and an automated grader Implicit
GDPval-AA, Artificial Analysis Blind pairwise on the 220-task gold subset One document artifact LLM judge, pairwise, Elo Conflated with the model
OfficeBench Office automation across documents, sheets, email Multi-app office task Programmatic checks Fixed
DABstep, Workspace-Bench Data analysis and file-heavy workspace tasks Multi-step data task Programmatic plus rubric Conflated

GDPval is the most carefully built of these. It covers 44 occupations across the top nine sectors contributing to US GDP, with tasks written by professionals averaging fourteen years of experience: 1,320 in total, 220 open-sourced as a gold subset with a public grading service. It measures what a model produces, not what a scaffold does, but the occupation-sampling method and the pairwise-plus-rubric scoring are the closest thing the field has to a template for artifact quality.

Web, browser and computer use⚓︎

Instrument Measures Unit of task Scoring Harness treated as
WebArena, CMU Self-hosted web navigation One web task Programmatic state checks Undisclosed in most reports
OSWorld / OSWorld 2.0 Computer use, long-horizon desktop work One desktop workflow Deterministic graders, partial credit Fixed per submission
BrowseComp, OpenAI Hard agentic web research One short-answer question Exact match Fixed
Steel browser-agent leaderboards Aggregates ten browser and computer-use benchmarks Inherited Inherited, sources tagged Mixed; system pages tagged

Research agents⚓︎

Instrument Measures Unit of task Scoring Harness treated as
Deep Research Bench, FutureSearch 89 multi-step web research tasks One research task Per-category precision, recall, F1, binary Fixed scaffold provided
DeepResearch Bench (RACE/FACT) 100 PhD-level tasks across 22 fields One long research report Adaptive rubric LLM judge, plus citation checking Fixed
DRACO, Perplexity 100 tasks against expert rubrics One research task Expert rubric Conflated; vendor-run

Deep Research Bench contributes one idea worth more than its scores: RetroSearch, a large frozen snapshot of scraped pages served to agents so that results stay comparable as the live web changes underneath them.

Memory and long horizon⚓︎

Instrument Measures Unit of task Scoring Harness treated as
Vending-Bench, Andon Labs Long-term coherence over a business simulation One long-horizon run Programmatic — net worth, coherence Fixed
LongMemEval Long-span memory, 500 questions, temporal reasoning Memory question Accuracy by category Memory layer varied
LoCoMo Long-term dialogue memory Dialogue recall question F1, accuracy Memory layer varied
Mem0 benchmark suite Memory accuracy against token cost Memory question Accuracy plus tokens The vendor's own memory layer

Security and governance⚓︎

Instrument Measures Unit of task Scoring Harness treated as
AgentDojo, ETH Zurich SPY Lab Prompt injection: benign utility and attack success rate One (user task, injection task) pair Programmatic environment state Defences are composable modules
NIST CAISI hijacking evaluations Hijacking resistance Hijacking scenario built on AgentDojo Attack success rate The system under test
τ2-bench, Sierra Research Tool-agent-user dialogue and policy adherence Multi-turn conversation against a policy document Database-state match, action correctness, pass^k Fixed

AgentDojo is the cleanest reusable component in the survey. It populates four environments (banking, Slack, travel, workspace) with "97 realistic tasks ... 629 security test cases", and scores both axes programmatically against resulting environment state: did the agent complete the benign task, and did the injected instruction divert it. No LLM judge. NIST adopted it for US government hijacking evaluations. Its acknowledged weakness is that static attacks age, and the field is calling for adaptive attacker-defender co-evolution.

Operational⚓︎

Instrument Measures Unit of task Scoring Harness treated as
HAL cost-Pareto analysis Accuracy against dollars and tokens One rollout Cost logged per run Variable
HAL AI Agent Reliability Tracker Variance across identical runs Multi-run task Pass distribution Scaffold analysed
MLPerf Agentic Inference, MLCommons Serving throughput and latency under agentic load Multi-turn trajectory TTFT, TPOT, throughput Not applicable — infrastructure
The Scaffold Effect Tokens per solved task One task Token accounting Variable

Buyer-side frameworks⚓︎

Instrument Measures Unit of task Scoring Harness treated as
Forrester Agent Control Plane Inventory, governance, orchestration, assurance Capability assertion Analyst scoring of declared capability The platform is ranked
Gartner, "How to Compare Enterprise AI Assistants" Search, connectors, tools, extensibility, governance Capability assertion Six declared criteria The assistant is ranked
Glean, "How to Compare Enterprise AI Assistants" Connectors, permissions, actions Capability assertion Checklist Own product favoured
Composio, "8 Best AI Agent Harnesses" Pass rate, tokens, cost, speed on a shared task set Shared task Programmatic checker The thing being ranked, at a fixed model

These four are the only instruments in the survey that address enterprise platforms as whole products. Three of them score by declaration and are paywalled or vendor-authored. The fourth is a practitioner bake-off. That is the entire measurement culture for the category most enterprises are actually buying.

Methodology⚓︎

Instrument Contribution
Agentic Benchmark Checklist (ABC), Kang et al. Task-setup and reward-design validity
Establishing Best Practices for Rigorous Agentic Benchmarks Rigour guidelines
Stop Comparing LLM Agents Without Disclosing the Harness A disclosure standard and the Binding Constraint Thesis
Efficient Benchmarking of AI Agents Task subset selection to cut cost
NIST CAISI, "Cheating on AI Agent Evaluations" A taxonomy of grader gaming
IBM Research, "Ready For General Agents?" The argument that cross-environment comparison is currently infeasible

The ABC paper is worth reading before trusting any score above. It found that "SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful", and that issues of this kind "can lead to under- or overestimation of agents' performance by up to 100% in relative terms."

Five instruments worth building on⚓︎

HAL is the strongest piece of reusable infrastructure. Any scaffold needs only expose a run(input) -> dict interface, after which HAL orchestrates parallel rollouts across hundreds of VMs, logs tokens and dollars, and runs LLM-aided inspection over the traces. The authors validated it with "21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000", and released 2.5B tokens of logs. Inspecting those logs surfaced behaviours no pass rate would have caught, including agents "searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks." Its own stated limitation is that providers swap weights and quantisations without notice, which breaks reproducibility from underneath.

Terminal-Bench 2.0 contributes the adapter pattern. A task is a Docker container with an instruction and an end-state test suite, so the question is not whether the model said the right thing but whether the machine ended up in the right state. The registry pins dataset name, version, commit hash and task subset. The adapter layer means one agent integration runs unchanged against SWE-bench Verified, AppWorld, DevEval and EvoEval.

The Scaffold Effect is a study design rather than a benchmark, and it is the template for isolating harness contribution: hold the model set fixed, vary the harness, and report pass rate alongside tokens per solved task and a failure taxonomy. Its own limits are scale and accounting: three harnesses, two models, fifty tasks, and token counts that do not compare cleanly between harnesses, since Goose does not split input from output.

AgentDojo is the only security instrument that scores both axes programmatically, which makes it droppable into a wider suite without dragging an LLM judge along with it. Its environments are also the nearest thing to a working probe for permissions, since an attack that succeeds is evidence that least privilege did not hold.

agentevals and Google ADK evaluation are the only instruments that natively score how a scaffold orchestrates rather than what it finally produces. Both score the trajectory: the sequence of tool calls and messages, matched against a reference under configurable strictness. For a question about harness quality rather than model quality, trajectory scoring is the closest available proxy.

How agents get scored⚓︎

Eight methods recur. Each is good at something and fails in a characteristic way.

Programmatically-checkable environment tasks. Terminal-Bench end-state tests, WebArena state checks, OSWorld graders, τ2 database state, AgentDojo environment state. Objective, reproducible, hard to game at the answer level. They fail through fragile simulators and gameable graders. NIST documented agents disabling assertions and reading git history rather than solving the task.

Golden-answer retrieval. BrowseComp, LongMemEval, parts of Deep Research Bench. Cheap, exact, and contamination is detectable. Only works where the answer is short and verifiable, and saturates quickly.

Rubric-scored artifacts. GDPval task-writer rubrics, DeepResearch Bench RACE, DRACO. The only way to measure open-ended deliverable quality. Rubrics drift, cost real money to author, and need expert calibration.

LLM-as-judge. GDPval-AA, DeepResearch Bench, agentevals, HAL log inspection. Calibration techniques in use: blind anonymised pairwise comparison, reference-based adaptive criteria, and splitting extraction from judgement across a cheaper and a stronger model. Fails through self-preference bias and non-determinism.

Human expert grading. GDPval, DRACO. Ground truth for knowledge work, and the reason GDPval ships an automated grader alongside: cost and latency make it unrepeatable at scale.

Pairwise preference. GDPval-AA, AgentArena. Captures a quality that rubrics miss, and generates preference data as a by-product. Conflates model and harness, and rewards popularity.

Capability assertions verified by probe. Almost absent. Analyst and vendor frameworks declare capabilities; nobody probes them. The closest reproducible probes are BFCL's relevance detection, which tests whether an agent correctly declines to call a tool, and AgentDojo's attacks, which probe least-privilege by trying to exceed it.

Operational telemetry during a run. HAL's token and dollar logging, the Scaffold Effect's tokens-per-solved-task, MLPerf's TTFT and TPOT, OpenTelemetry's genAI semantic conventions. The only route to cost, latency and reliability. Fails because harnesses report tokens inconsistently and provider routing shifts results between runs.

What the field cannot agree on⚓︎

Separating the harness from the model⚓︎

The evidence that they must be separated is now strong: the 40× token span at fixed model, the 41.5-point WebArena span at fixed backbone, and the Binding Constraint Thesis. The disagreement is over what to do about it.

One camp wants disclosure. The "Stop Comparing" position paper proposes a standard set of fields every submission must report, so that a reader can at least see what was varied. A second camp wants factorial design. HAL's models × scaffolds × benchmarks grid is the only large-scale example. Neither resolves the hard case, which is a harness locked to one model family. Claude Code runs Claude; Codex runs GPT. There is no configuration in which they can be compared at a fixed model, so any number placed in a model-controlled column is wrong by construction.

Whether to aggregate⚓︎

HAL abandoned the one-dimensional leaderboard in favour of accuracy-against-cost Pareto frontiers. Steel refuses to cross-compare its constituent benchmarks at all, on the grounds that 92% on one and 80% on another is not a ranking. HELM's durable contribution is a grid and the argument that no single number should stand for a system's whole character. Against that, buyers want a ranking, and every attempt to refuse them one has been met with someone else publishing the ranking anyway.

The IBM critique sharpens this: across benchmarks, even "success" is defined incompatibly, so a composite is not merely lossy but incoherent.

Contamination and decay⚓︎

SWE-bench Verified was retired because models had memorised the patches. τ-bench froze at late-2024 models. SWE-bench Pro was built specifically to resist contamination, drawing 1,865 problems from 41 actively maintained repositories split across 11 public, 12 held-out and 18 commercial. Agents were still caught reading git history and searching Hugging Face for answers.

The instruments that survive share one habit: they refresh on a schedule and version what they refresh. Terminal-Bench pins commit hashes in its registry. τ-bench moved to τ2. Deep Research Bench freezes the web rather than the questions. What nobody has settled is the cadence, or who pays for it.

Whether a declared capability can be verified⚓︎

Forrester, Gartner and Glean all score enterprise platforms on capabilities the vendor states it has. That is the dominant method for the entire trust-and-governance axis: permissions, audit export, data residency, tenancy, human-in-the-loop checkpoints. No reproducible probe exists for any of them. Whether one could exist is an open question. A canned MCP server the harness must discover and call is easy to imagine, a reproducible test of data residency much less so.

What nobody measures⚓︎

  • Fully loaded cost per completed business task. HAL and the Scaffold Effect measure token and dollar cost on academic tasks. Nobody measures cost including retries, human oversight and connector fees. Costs are non-deterministic, provider routes shift between runs, and vendors have no reason to publish the real figure.
  • Connector depth and quality. Every vendor publishes a connector count. Nobody scores whether a connector reads and writes correctly, under permissions, at depth. Testing it requires live credentials to hundreds of SaaS tenants.
  • Governance as a scored outcome. AgentDojo scores injection resistance. Permissions, auditability, trace export, residency and admin controls are only ever declared. No environment exists to probe them reproducibly.
  • Run-to-run reliability. HAL's Reliability Tracker is close to alone in reporting variance across identical runs. Multi-run protocols multiply cost, and stable numbers are commercially inconvenient.
  • Onboarding and time-to-value. Central to how buyers actually choose, measured rigorously by nobody. It depends on the customer's own data and takes weeks, which is hostile to any static benchmark.
  • Whole products. No instrument runs the Claude, ChatGPT or Gemini applications end to end, or Glean, Agentforce and Copilot Studio, across all four axes. The products are closed, have no evaluation hooks, and expose mutually incompatible interaction protocols.
  • Delegation quality. Frameworks advertise multi-agent orchestration. Trajectory scoring covers whether tools were called in a sensible order; it does not cover whether delegation was correct at scale.

The case that this cannot be done⚓︎

The strongest argument against a horizontal harness benchmark comes from IBM Research, whose position is that cross-environment comparison is currently "impossible without substantial ad hoc engineering" because harness communication protocols are mutually incompatible. A web agent has no conversational channel, and a chat agent has no way to issue a click. Forcing an MCP-native agent like Claude Code through a command-line protocol distorts what it does. General AgentBench supplies the empirical companion: scores degrade sharply moving from domain-specific to general settings, which means the task distribution you choose largely determines who wins.

Read alongside HELM and Steel, both of which reject cross-benchmark aggregation, the honest reading is that a horizontal harness benchmark cannot be a single ranking. It can be a disclosed, per-axis matrix with the harness held as a controlled variable wherever the product permits it. Even then, its coverage is a policy choice rather than a fact about the world, and the choice of what to include will decide the result.

Conclusion⚓︎

The harness moves agent performance as much as the model does, and almost nothing measures it. The instruments that exist were built to rank models under a fixed scaffold, and they do that well. Three things treat the scaffold as the variable: HAL, the Scaffold Effect study design, and a handful of practitioner bake-offs.

The components for something better are already published and mostly open: HAL's orchestration and cost logging, Terminal-Bench's adapters, AgentDojo's security environments, GDPval's grading method, trajectory scoring from the SDK vendors.

The obstacle is ownership. The organisations able to build it all have a product in the ring, and the ones without that conflict cannot get access to the closed harnesses that would need to be in it.

References⚓︎

Peer-reviewed or reproducible⚓︎

Documented and independently run⚓︎

Vendor or analyst framings⚓︎

These score declared capabilities rather than measured outcomes, and each author has a commercial interest in the result.