Skip to content

Archive⚓︎

  • 12 min read

The Secret and the Standard

Evals you hoard, benchmarks you publish.

There are two ways to own a piece of knowledge. You keep it secret, or you publish it and own the standard.

Coca-Cola never patented its formula. A patent expires; a secret doesn't. So the recipe sits in a vault, and the property is worth exactly what the secrecy holds. Bell Labs did the reverse with the transistor: it patented the invention, licensed it to all comers, and an industry grew on the disclosure. One asset earns from what nobody can see. The other earns from what everybody has to reference.

AI evals are about to split along the same line. An eval is a graded set of tasks that defines what "good" means in your domain. Hoard it and it's one kind of property; publish it and it's the opposite kind. Most companies that own one don't know which game they're playing. Most don't know they own property at all.

  • 9 min read

The Plastic Substrate

In materials science, plastic deformation is the part of a material's response to stress that does not spring back. Push a steel beam past its elastic limit and the bend stays. Push a fluid and it flows away. Plastic deformation sits between the two.

A plastic substrate, in software, is an authored layer that behaves the same way under use. The user edits something. The behavior of the system on the next invocation is different in a way the user intended, in a form the user can read back. The deformation persists and the record is legible.

What follows is the architecture of that layer. While examples are from Claude Code for familiarity, the pattern should translate into any harness.

  • 6 min read

The Eval Checklist

Nine pieces of LLM evaluation, in the order you build them. Each item is a testable claim about your eval implementation. The unchecked items are your gaps.

Essential items apply to every eval system. Items marked (when applicable) depend on your system type — RAG, agents, multi-turn, etc.

  • 18 min read

Harness Engineering: A Composable Architecture

Three engineers at OpenAI produced a million lines of code last year. None of it was written by hand. What made that work was the structure around the model, not the model itself: a pipeline where quality gates check every transition and failures loop back as structural fixes rather than prompt patches.

Tatsunori Hashimoto named the discipline: when an agent makes a mistake, engineer a structural fix so it can never make that mistake again. Birgitta Böckeler, writing on martinfowler.com, distinguished what steers agents before they act from what corrects them after, mapping a taxonomy of guides and sensors. Anthropic's multi-agent research showed the shape at a different scale: separate generation from evaluation, make the evaluator skeptical, loop until everything passes. Four groups, different problems, the same skeleton.

That skeleton is a pipe. A signal enters, gets transformed through stages, and produces an artifact, with quality checked at every seam. The scientific method follows this shape; so does OODA. The structure is older than software, the basic form of structured inquiry.

But if harness engineering names only the pipe, then what has been named is not new. The field needed explicit terminology for the age of agents, and the terminology is valuable. The question is whether an architecture exists underneath: something an engineer can compose and configure, something that explains how the system that produces runs gets better over time. The pipe comes first, because you need the skeleton before you can see what has grown around it.

  • 16 min read

Harness Engineering

In late August 2025, a team at OpenAI made the first commit to an empty git repository. Five months later, that repository contained roughly a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over 1,500 pull requests had been opened and merged. The product had internal daily users and external alpha testers.

None of the code was written by a human.

Three engineers drove the work. They wrote no code themselves. Every line was generated by Codex, OpenAI's coding agent. They averaged 3.5 merged pull requests per engineer per day, and the throughput increased as the team grew to seven. The product shipped, deployed, broke, and got fixed. What the engineers did, from the first commit onward, was something other than programming.

Ryan Lopopolo, writing about the experiment in February 2026, put it plainly: "The primary job of our engineering team became enabling the agents to do useful work." When the agent got stuck, the fix was almost never "try harder." The engineers would step into the task and ask: what capability is missing, and how do we make it both legible and enforceable for the agent?

That question turns out to be the entire discipline.

  • 30 min read

The Phoenix Test

"Voting With Fire" made the case for the phoenix: a codebase designed for regeneration rather than permanence, where specifications and acceptance tests constitute the durable layer and application code is disposable. But it left a question open that the archaeological evidence answers with uncomfortable precision. Which specifications? Which tests? What, exactly, needs to survive the fire — and what looks durable but will burn with the structure it serves?

Luke Kemp's evidence across two hundred cases of civilizational collapse reveals a variable that determines everything, a variable that has nothing to do with the knowledge itself and everything to do with the social arrangement that carries it. The same knowledge, held differently, has radically different survival properties. The critical question is never what the knowledge is. It is who holds it, how it travels, and whether it serves the community or the palace.

  • 19 min read

Voting with Fire

In Goliath's Curse, the historian Luke Kemp renames civilization. He calls it Goliath: a collection of dominance hierarchies in which some individuals dominate others to control energy and labor. Named for the Bronze Age warrior — imposing in stature, reliant on violence, surprisingly fragile. Across two hundred case studies spanning five millennia, Kemp documents the same structural dynamic. As hierarchical societies age, inequality concentrates, decision-making deteriorates, and the system grows brittle. Complexity scientists call it critical slowing down. A healthy system absorbs shocks and recovers quickly. An extractive one recovers more slowly from each successive disturbance, like an aging body that takes longer to heal from each injury, until eventually a shock that the system would have once absorbed tips it into collapse.

The curse is internal. Goliaths don't die from external assault. They hollow themselves out. The wealth pump transfers resources upward; the exchange between rulers and ruled grows more unequal; elites compete for shrinking returns; and the population that once sustained the structure loses both the incentive and the capacity to defend it. Then drought comes, or invasion, or rebellion, and the system that looked permanent proves to have been perched on a knife's edge for decades.

Software engineers will recognize this dynamic because they live inside it.

  • 11 min read

DISCOVER

Software development has two scoreboards and a process between them. The first measures deployment: how fast code ships, how often it breaks, how quickly you recover. The second measures the organization: how fast signals reach the right person, how autonomous the response is. Comprehension — understanding the system well enough to decide and act correctly — sits between them.

The progression follows a logic that Simon Wardley mapped in his work on technology evolution: every capability that becomes commodity accelerates everything that depends on it, exposing the next constraint beneath.

  • 7 min read

The Decision Loop

DORA measures how fast code flows from commit to production. MOVE measures how fast the organization senses, decides, and acts. Both track real performance. Both share an assumption: that someone, somewhere, understood the system well enough to make the right call.

That assumption has a cost, and it's larger than most organizations realize.

  • 21 min read

The Replacement Rate

Two guys in the jungle. A tiger charges. One kneels to tighten his shoelaces. The other yells: "You can't outrun a tiger!" First guy: "I don't have to outrun the tiger. I only have to outrun you."

Thorsten Ball used this joke recently to make a point about AI and the average software engineer. The joke is more precise than he may have intended. It contains, in five sentences, both a correct economic model and a game-theoretic trap. The model: your value isn't absolute; it's relative to the next-best alternative. The trap: when everyone tightens their shoes, the tiger catches someone anyway, and the race never ends.

Sports analytics formalised this intuition decades ago. The framework is called VORP: Value Over Replacement Player.