The Secret and the Standard

Evals you hoard, benchmarks you publish.

There are two ways to own a piece of knowledge. You keep it secret, or you publish it and own the standard.

Coca-Cola never patented its formula. A patent expires; a secret doesn't. So the recipe sits in a vault, and the property is worth exactly what the secrecy holds. Bell Labs did the reverse with the transistor: it patented the invention, licensed it to all comers, and an industry grew on the disclosure. One asset earns from what nobody can see. The other earns from what everybody has to reference.

AI evals are about to split along the same line. An eval is a graded set of tasks that defines what "good" means in your domain. Hoard it and it's one kind of property; publish it and it's the opposite kind. Most companies that own one don't know which game they're playing. Most don't know they own property at all.

The consensus

Satya Nadella has spent months telling anyone who will listen that a company's private evals "may be the biggest IP." He wasn't first. Anthropic's Alexander Bricken said it a year earlier, and Y Combinator's Garry Tan called evals the real moat for AI startups before that. The vendor blogs have caught up since. In under two years the claim went from contrarian to consensus.

What the consensus skips is the only question that changes what you do about it: which kind of property? A secret and a standard are both worth owning, but you build them by opposite rules — one you guard, the other you give away. Until you've answered which one you're holding, "evals are IP" isn't a strategy. It's a slogan.

Start with what the thing actually is. A private eval is more than a vibe check or a dashboard. It's three artifacts. A failure taxonomy: the named ways your system fails, pulled from real production traces. A golden dataset: graded examples of good and bad output, held with train/dev/test discipline so you can't fool yourself. And validated judges: LLM-as-judge instruments whose agreement with your human experts is measured, not assumed.

Write it all down and it looks like a spreadsheet. Why would a spreadsheet be the most valuable thing a company owns?

The uncopyable part

Because you can't buy one that works.

The taxonomy and the golden set come out of your own traces. Hamel Husain, who wrote the standard playbook for this work, is blunt about where the failure categories come from: you read your own transcripts and name what you find, nothing borrowed from another project or brainstormed in advance. An expert reads fifty to a hundred real traces of your system failing in your domain and names what went wrong. Before your system has failed in front of your own experts, the asset doesn't exist — so there's nothing to copy.

The technique, though, is a commodity. Husain open-sourced it: how to run error analysis, how to build a judge, how to validate one. I laid out the full nine-step build in The Eval Checklist. Six of the steps generalise to any project, and the open tooling ships them. The three that don't — instrumenting your traces, fixing your bugs, wiring your own CI — are where the property lives. The method is free. The content it produces is yours.

There's a fast way to see that content. Take the eval and subtract everything a frontier model already knows. Generic correctness goes; fluent prose goes; any model has those. What's left is domain judgment: the threshold where a refund over $500 needs a human, the procedure a regulator expects to see followed, the difference between the failure that costs you a client and the one that costs you an apology. The eval is that judgment written down in a form a machine can check ten thousand times a day.

It's also why eval moats might hold where "data moats" disappointed. Companies sat on years of raw logs that never became an advantage, because raw data doesn't turn into product. An eval is the refined form: judgment already distilled, already wired to a deploy gate.

The control test

So the eval is real property. The next question is whether it's yours — and there's a test you can run this quarter. You're on model A. Switch to model B. Can you climb back to your quality bar within days?

If yes, you own something. The scorecard that defines "good" is yours, and any engine can be tuned against it. If no — if switching means months of vibes and regressions nobody can name — you never owned the quality. The model owned you.

Nadella puts the same test to enterprises, and the stakes only grow. Models are commoditising; the frontier changes hands every few months. Each handover is either an upgrade or a hostage negotiation, and your eval decides which. The scorecard defines the hill. The model is just whatever climbs it.

And it's never a bare model that climbs. A benchmark scores a system — the model wired into your harness, your tools, your context — which is why the same model posts different numbers under different harnesses. The harness is yours too. What you own measures the whole system, not the engine inside it.

The loop

Say the test comes back clean and the scorecard is yours. It still has a shelf life. Your system eventually passes every example in its own golden set, and the domain drifts underneath it — new products, new regulations. A static eval is a depreciating asset.

The objection is correct, and it tells you what the asset actually is. The eval isn't the scorecard. It's the measurement half of a loop — and measurement is the half that closes it. Strip the eval out and nothing feeds back: what's left isn't a loop but an open system, running until something external stops it. Traces come in; new failures get named and added to the taxonomy; the golden set regrows; judges get revalidated against the experts; deploy gates move. Then the system fails in new ways, and the loop turns again. What compounds is the loop. The eval set is just its latest reading.

Investors who push back on the moat story are right about the half they can see. Ed Sim's version: evals alone aren't defensibility — you also need the context, the workflow integration, the execution data. Concede it. His list is the rest of the loop.

The market is already pricing that loop, piece by piece. Anthropic's leaders have reportedly discussed spending more than $1 billion in a year on RL environments — the simulated workplaces where agents practise. Fleet, which sells them, reached a reported valuation around $750 million. Prime Intellect runs a hub of more than 2,500 of them. Nobody is buying the model. They're buying the gym.

And the loop repairs the one crack in the trade-secret frame. Coca-Cola's secret survives because nothing forces it to change; your domain grants no such mercy. The only trade secret that outlasts a drifting domain is one that regenerates faster than it leaks — which is exactly what the loop does.

The landlord's offer

If the moat is the loop, the next question is who runs it — and there's a pitch going around with a seductive answer. The same people who tell you evals are your biggest IP also sell the machine that builds them for you. In the version Nadella sketched at Stanford, the platform watches a workflow it already hosts — HR onboarding running through M365, say — and from what it observes, generates the evals and the training environment around them. A multi-tenant hill-climbing service. Bring any model. Press the button.

If the landlord builds your hill-climbing machine, whose hill is it?

Here's the part that's easy to miss, because on the surface it looks like everything above. It isn't. Until now the question was whether an eval is worth owning. This one is subtler: you can have a perfectly good eval — accurate, useful, climbing every day — and not own it. The taxonomy is real, the golden set is graded, the gate works. But if all of it lives in the platform's format, runs on the platform's telemetry, and gates deploys only inside the platform's walls, you are not the owner. You are a tenant with a very nice apartment.

Run the control test again, one level up. Last time you swapped the model; this time swap the platform. Can you take the taxonomy, the golden set, the judges, and the traces, and go climb somewhere else? If the answer is no, the eval is real and the ownership is rented — and the rent is paid in switching costs.

None of this is a reason to refuse the button. Generated evals are genuinely useful. They spare you the slowest part of the build, and a platform that has watched onboarding run ten thousand times across its tenants has seen failure patterns you haven't. But there's a part neither the platform nor the frontier lab behind it can absorb: which failure costs you a client, what "good" means to your domain experts, who is liable when the answer is wrong. That knowledge isn't in the telemetry.

So hold the line: generated evals are scaffolding; validated evals are IP. Keep the taxonomy in plain files, the golden set exportable, and judge agreement measured against your own people. Take the scaffolding. Own the validation.

Now read the same picture from the other chair. Every tenancy has a landlord, and the landlord's is the best seat at the table. If you run the platform other people build their evals on — if you own the format, hold the telemetry, and are the gate their deploys have to pass — then their switching costs are your revenue. You're not renting the hill. You're charging admission to it. That is a real strategy, not a hypothetical; it's what the easy button looks like from the other side. It carries one hard condition: you can only be the landlord if you own rails other people genuinely need to build on. Most don't — and mistaking your own apartment for the building is how you end up paying rent while you think you're collecting it.

The other move

Tenant or landlord, that whole contest is about the eval you keep — held close, guarded, the value in who can't get at it. Now turn the artifact over.

The same scorecard, published instead of hidden, becomes the opposite kind of property — the Bell Labs move, not the Coca-Cola one. It sounds like a contradiction: why give away the thing you just spent an argument learning to guard? Because disclosure can be worth more than secrecy, and one industry has already run the whole experiment.

When John Bogle launched the first retail index fund in 1976, it tracked the S&P 500 — and the index was worth so little as property that, as the Acquired podcast tells it, people inside S&P wondered who should be paying whom. It was public and effectively free.

Then the ecosystem grew, and the property appeared. S&P began writing licenses, and the teeth came through the courts: at one point it blocked Vanguard from launching an S&P 500 ETF for years. By 2025 a single fund whose entire job is tracking the index — SPY — paid S&P $186 million in license fees for the year, and S&P Global's index business booked $1.85 billion of revenue at a 69% operating margin.

Look at what kind of property that is: a trademark and a contract, no patent anywhere in the story. The methodology was public the whole time. Publish the reference and you own the name — the thing everyone has to license to say they track it. The fund is the commodity wrapper; the index is the asset. Disclosure didn't give the property away. Disclosure created it.

AI benchmarks are in their 1976 moment: published free, stewarded informally, nobody paying rent yet. And the standard-setter move is already visible. Sierra, an agent startup, published τ-bench for customer-service agents, and Anthropic's Claude 4 launch page reports scores on it. A frontier lab citing a startup's scorecard in its own launch material — that's the whole play in one act. Whoever owns the benchmark owns the axis everyone else is measured on.

But a benchmark is a stranger asset than an index, and the difference is the whole game. The S&P 500 costs almost nothing to maintain: a committee applies a rulebook — the large, liquid, profitable American companies — and the index goes on measuring the same thing forever. It doesn't wear out. A benchmark does. The day you publish it, every lab starts training against it; scores climb, the ceiling arrives, and the number stops telling good from great. MMLU ran from 43.9% to a frontier parked at 88–93% and the labs quietly stopped reporting it; Humanity's Last Exam, built because everything else had saturated, was half-solved within eighteen months.

So the value was never in publishing the benchmark — that part is done once, and cheaply. It's in the work of keeping it worth citing: re-cutting the hard cases as the old ones top out, controlling for contamination between releases, throwing out the tasks that turn out to be broken. That work is the loop from the first half of this essay, run in public instead of in private. OpenAI took over SWE-bench, an academic benchmark that was already public, cut a human-verified five-hundred-instance subset, and turned SWE-bench Verified into the coding number every lab now reports. The raw benchmark existed; the stewardship created the value. Abandoned, a benchmark decays from instrument into training target. Tended, it stays an instrument — and the $186 million is rent on the reference.

So who does the tending? Not naturally a platform. The companies that own reference standards in other industries — the ones that rate the credit, measure the audience, certify the wiring won't burn your house down — are trusted because they don't compete in the market they measure. The neutrality is the asset. Which puts a platform in an awkward spot: own the benchmark everyone measures against and you hold the rails and the ruler at once, the strongest seat on the board — but a referee who also plays stops being believed. You can build the standard. Staying credible with it may mean giving it away.

That is the move a strategist should be watching. As agents and the packaged workflows built from them commoditise, buyers face a shelf where everything claims to work and nothing proves it, and the scarce thing becomes certification — a trusted answer to which of these is any good.

Today that answer is thin on the ground. Coding has SWE-bench — agentic, continuously tended, the number every lab reports. Other kinds of work have benchmarks too, but mostly knowledge tests a model can pass without doing the job — the bar exam, the medical boards — and no single agentic standard has emerged the way SWE-bench has for code. The frontier labs are building the general version: OpenAI's GDPval already scores models on real tasks across forty-odd occupations. But that is the horizontal sweep. The deep, trusted, continuously-tended standard for one domain — the one a compliance officer would stake her name on — is still up for grabs in most of them, and the people positioned to claim it are the ones who already own that domain's judgment.

Which tells you what the steward actually sells. Not the dataset — publish that and it saturates. The product is the verdict. So the benchmarks built to last don't release their tasks at all; they publish the scores and lock the questions away, precisely so no one can train against them. Scale keeps its evaluation sets private, the ARC Prize holds its test out, FrontierMath has never released its problems. They hype the scoreboard and hoard the ruler. And look at what that move is: it keeps the method private — the loop from the first half of this essay — and publishes only the verdict — the standard from the second. It refuses the choice the essay has been drawing. It is what Sangeet Paul Choudary calls rebundling: once a thing splits into parts, the winner owns the one part everyone else has to route through, and folds the rest back around it. Here that part is the trusted number. A credit-rating agency is the pure form — Moody's sells you the letters, never the model behind them, and the letters are the whole business.

That arrangement carries the rating agencies' old cost. A benchmark you can't see is one you can't game, and also one you can't check. In 2008 the agencies stamped their top grade on mortgage-backed junk, because the firms being rated were the firms paying for the rating and no outsider could audit the model that produced the letters. The un-gameable benchmark and the unauditable one are the same benchmark. Whoever comes to own the trusted number for agentic work inherits the rent and the liability together.

Two ways to own it

One artifact, two ways to own it — the two the essay opened with. Keep it, and it's a trade secret: model-independent, compounding for as long as the private loop that regenerates it stays yours, whether you climb on it alone or rent the hill to everyone else. Give it away, and it's a bid to own a standard: you trade the secrecy for a reference others must cite, and take on the public version of that same loop to keep it worth citing. Or, if you can be trusted to hold the ruler, do both at once — publish the verdict and keep the method, the letters and not the model.

Both moves assume the scorecard exists. For almost every company, it doesn't. The judgment is still where it has always been — in a few experts' heads, unwritten and ungraded, walking out the door each time one of them leaves. And that is the strange part: the most valuable IP a company will ever hold is also the cheapest to mint. You make it by writing down what your best people already know. Almost no one does.