Skip to content

Archive⚓︎

  • 16 min read

Benchmarking the Harness

Every public agent leaderboard ranks a model. Around that model sits a scaffold: the code that issues tools, builds context, decides when to retry and when to stop. The scaffold usually goes unreported. Three controlled results published this year show what that omission costs.

Vats and Golev ran two models across three open-source harnesses on a stratified 50-task subset of Terminal-Bench Pro. Their finding: "harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points." Failure patterns replicated across models. Goose failed on reasoning, OpenHands-SDK on verification and turn limits, OpenCode on idle loops and timeouts. Those are properties of the harness, not of the model running inside it.

  • 16 min read

The Cost Control Checklist

Your AI bill is not a pile of line items. It is one multiplication:

total spend  =  users  ×  sessions per user
               ×  turns per session  ×  requests per turn  ×  tokens per request
               ×  price per token

Put real numbers in it and the shape becomes obvious:

100 users × 20 sessions × 50 turns × 3 requests × 20,000 tokens × $3/M  =  $18,000/month

Because it is a multiplication, halving any single term halves the whole bill. Cache properly so each request carries 10,000 tokens instead of 20,000, and you are at $9,000. Leave the tokens alone and route the work to a model at half the price — also $9,000. Ground the agent so it finishes in 25 turns instead of 50 — also $9,000. Do all three and the same delivered work costs $2,250.

  • 8 min read

Double Entry AI

(Q, A) → (Q, A, U, W)

Q is the question a harness forms. A is the answer it produces. U is whether the person working with it still understands the output. W is whether producing it is sustainable for them. Harnesses today are measured on Q and A. This post is about U and W.

The two ledgers Left: the Machine Ledger, what the harness produces. Right: the Human Ledger, what producing it does to the person.

The name is from bookkeeping. Single-entry records what came in and went out. Double-entry, in use since the 1300s, records every gain against its source, so a gain cannot appear in one account without its cost appearing in another. The analogy has a limit: in bookkeeping both entries are money, and here they are different quantities with no exchange rate. What carries over is the rule. You may not keep only one ledger.

  • 2 min read

The Economics of Difficulty

A machine arrives on the floor and everyone gathers around it.

The workers want to know where they stand, and what they are supposed to do while the machine runs. The consultant wants to know how the money moves now — which step everyone has to route through, and where a toll can be charged.

Whether or not the machine is clever, it can cost jobs and rebundle skills. Systemic change does not wait for evidence at the level of the individual worker. Belief moves first, then action, and the value chain is re-cut on the strength of a forecast — well before anyone has established what the machine can actually do, and sometimes instead of ever establishing it.

  • 3 min read

Launching Orqestra

The main stage at Ready Summit '26
Ready Summit '26.

I launched Orqestra earlier this week at Ready Summit '26, in front of several hundred people in Melbourne and again in Sydney. We built it to speed up the compliance-heavy, domain-specific work our clients do in Education, Workforce, Work Pathways, Local Government and Justice. But it's more than accelerating the work. We also built it to help the people behind that work understand and steer AI better, manage their careers and develop their workforces.

  • 12 min read

The Secret and the Standard

Evals you hoard, benchmarks you publish.

There are two ways to own a piece of knowledge. You keep it secret, or you publish it and own the standard.

Coca-Cola never patented its formula. A patent expires; a secret doesn't. So the recipe sits in a vault, and the property is worth exactly what the secrecy holds. Bell Labs did the reverse with the transistor: it patented the invention, licensed it to all comers, and an industry grew on the disclosure. One asset earns from what nobody can see. The other earns from what everybody has to reference.

AI evals are about to split along the same line. An eval is a graded set of tasks that defines what "good" means in your domain. Hoard it and it's one kind of property; publish it and it's the opposite kind. Most companies that own one don't know which game they're playing. Most don't know they own property at all.

  • 9 min read

The Plastic Substrate

In materials science, plastic deformation is the part of a material's response to stress that does not spring back. Push a steel beam past its elastic limit and the bend stays. Push a fluid and it flows away. Plastic deformation sits between the two.

A plastic substrate, in software, is an authored layer that behaves the same way under use. The user edits something. The behavior of the system on the next invocation is different in a way the user intended, in a form the user can read back. The deformation persists and the record is legible.

What follows is the architecture of that layer. While examples are from Claude Code for familiarity, the pattern should translate into any harness.

  • 6 min read

The Eval Checklist

Nine pieces of LLM evaluation, in the order you build them. Each item is a testable claim about your eval implementation. The unchecked items are your gaps.

Essential items apply to every eval system. Items marked (when applicable) depend on your system type — RAG, agents, multi-turn, etc.

  • 18 min read

Harness Engineering: A Composable Architecture

Three engineers at OpenAI produced a million lines of code last year. None of it was written by hand. What made that work was the structure around the model, not the model itself: a pipeline where quality gates check every transition and failures loop back as structural fixes rather than prompt patches.

Tatsunori Hashimoto named the discipline: when an agent makes a mistake, engineer a structural fix so it can never make that mistake again. Birgitta Böckeler, writing on martinfowler.com, distinguished what steers agents before they act from what corrects them after, mapping a taxonomy of guides and sensors. Anthropic's multi-agent research showed the shape at a different scale: separate generation from evaluation, make the evaluator skeptical, loop until everything passes. Four groups, different problems, the same skeleton.

That skeleton is a pipe. A signal enters, gets transformed through stages, and produces an artifact, with quality checked at every seam. The scientific method follows this shape; so does OODA. The structure is older than software, the basic form of structured inquiry.

But if harness engineering names only the pipe, then what has been named is not new. The field needed explicit terminology for the age of agents, and the terminology is valuable. The question is whether an architecture exists underneath: something an engineer can compose and configure, something that explains how the system that produces runs gets better over time. The pipe comes first, because you need the skeleton before you can see what has grown around it.

  • 16 min read

Harness Engineering

In late August 2025, a team at OpenAI made the first commit to an empty git repository. Five months later, that repository contained roughly a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over 1,500 pull requests had been opened and merged. The product had internal daily users and external alpha testers.

None of the code was written by a human.

Three engineers drove the work. They wrote no code themselves. Every line was generated by Codex, OpenAI's coding agent. They averaged 3.5 merged pull requests per engineer per day, and the throughput increased as the team grew to seven. The product shipped, deployed, broke, and got fixed. What the engineers did, from the first commit onward, was something other than programming.

Ryan Lopopolo, writing about the experiment in February 2026, put it plainly: "The primary job of our engineering team became enabling the agents to do useful work." When the agent got stuck, the fix was almost never "try harder." The engineers would step into the task and ask: what capability is missing, and how do we make it both legible and enforceable for the agent?

That question turns out to be the entire discipline.