Evals Aren't the Moat

“Not everything that can be counted counts, and not everything that counts can be counted.” — William Bruce Cameron

A fast eval told me a restructured module was clearly better. Comfortable margin. I ran it again to be sure, got the same verdict, promoted the new version, and moved on.

Then the slow canonical harness flipped the sign. Across 49 scenarios the restructure never once beat the original. The fast judge had put candidates side by side and rewarded the terse, confident form; the canonical harness scored each one against the target and caught that the restructure had swapped substance for style. Two green runs, wrong the same way. A biased metric doesn’t get less biased when you run it twice — it just reproduces its bias. I reverted. Nothing shipped.

This is a post about evals, and my own eval, run twice, was confidently wrong.

The literal version turned up the same week. An eval runner printed “Done”: a rejected model id routed every call to a failure sentinel, and the failures averaged out to a clean, empty green. It ran. It measured zero.

Bottom line up front

• “It passed the evals” is where review starts, not where it ends. Green reports that the checks you encoded passed — not the intent and criteria you never specified: correct-to-intent, good, worth building.
• Point evals at the seams, not just the units. Faults cluster in the contract between two units that are each right on their own; a per-unit suite is blind there.
• Watch the trajectory, not the snapshot. “It runs” today can be raising tomorrow’s bug rate. Quality is a loop; the eval scores one frame unless you build it to score more.
• A second model’s eval isn’t an independent one. The blind spots correlate; running it again, or on a different model, doesn’t launder the bias.
• Spend your judgment on what no eval can see unless you encode it: correct-to-intent, where the quality is heading, and what to build at all. That’s the moat.

The line that collapses four questions

a16z’s argument (Sivulka, “tokens & loops”) turns on a killer line: coding is where almost all AI revenue lands, because “code runs or it doesn’t” — so evals are the new OKRs, and the eval suite is your moat. I’ll take the argument and leave the numbers; the stats around it are unsourced, so I treat them the way I’d treat a vendor grading its own benchmark.

The piece already names the real moat: deep understanding — you have to “program the business.” Then it hands the same title to the layer that just got automated. Only one thing can be the moat, and it isn’t the one that got cheap.

“Code runs or it doesn’t” answers the first of four questions and calls it the whole game. Runs isn’t correct. Runs isn’t good. Runs isn’t the right thing. Green answers only what you wrote a check for — and “code runs or it doesn’t” wrote just one: did it execute.

That line measures the execution layer — did it run, did it pass what you specified. Richer evals exist: rubrics, judges, behavioural suites, and they reach past execution. But they only check what you thought to specify. You can only eval the essence you already captured, so the richer suite inherits the same ceiling — it raises it a little. And that ceiling isn’t only what you forgot to write down: some criteria are unwritable until you see outputs. Shankar et al. (2024) hit this building an LLM-eval tool and named it criteria drift — you need criteria to grade outputs, but grading the outputs is how you discover the criteria. The specification isn’t just incomplete; part of it can’t be front-loaded at all.

Runs is the accident

Fred Brooks split software in two. The essence is the conceptual work — deciding what to build, pinning down the requirements. The accident is the labour of saying it in code. His thesis in No Silver Bullet (1986): no single tool buys an order-of-magnitude gain, because the hard part is the essence and tooling mostly went after the accident. “The hardest single part of building a software system is deciding precisely what to build.”

Map that onto AI — the mapping is mine; Brooks predates LLMs. AI is the largest attack on the accident yet. Writing code got cheap. It barely touches the essence. So “code runs or it doesn’t” measures the accident — precisely the part that just got automated — and calls it the moat. The moat is the essence, where the eval can’t reach.

The rebuttal a16z reaches for: the loops happen because a human never articulated the task cleanly, and better prompting fixes that. That’s the waterfall fantasy. A generation of practitioners already concluded you can’t reliably specify software up front — the Agile Manifesto (2001) is the flag they planted, requirements discovered by building rather than front-loaded. Call that a widely-held stance, not a settled fact. Either way, loops aren’t a prompting-skill gap. They’re the essence showing through — the thing those methodologies were invented to cope with.

Runs ≠ correct

What “correct” even means is an assumption you have to supply — software’s old oracle problem (Weyuker 1982; Barr et al.'s 2015 survey names it and shows it never goes away). An eval checks the behaviour you specified. It says nothing about the intent you didn’t. And a green suite can pass while the seam is wrong: each unit correct, the contract between them broken. Perry and Evangelist found at least 66% of faults were interface faults in the system they studied — the joins, not the parts. I read that as a general tendency for faults to cluster at the seams; a per-unit eval is structurally blind there.

Smaller, same shape: I pushed back on a flatMap used as a filter. It worked. It passed the rules. It was still the wrong way to say the thing — code should show intent, not smuggle it through a clever hack.

// It runs — but disguises a filter as an expansion.
const active = items.flatMap(item => item.active ? [item] : []);

// It runs, and shows the intent.
const active = items.filter(item => item.active);

Both work. The eval sees two green lines; only reading catches that one of them says what it means.

Runs ≠ good

Quality isn’t binary and it isn’t static. It’s a loop. Bad code makes the next bug likelier — for the human who works around it, and, I’d argue, for the AI, which conditions on the local mess and regresses toward it. The first half is measured; the second is my inference. That churn predicts defects has data behind it: Nagappan and Ball found relative code churn predicts defect density (correlational, one system, but directional). GitClear’s 2025 report shows code clones climbing and refactoring falling as AI assistants spread — a field trend, but correlational, vendor-authored, and it doesn’t isolate AI-written code, so it’s a weathervane, not a proof. That the AI compounds the mess by conditioning on it is my inference, not a measured loop.

So “it runs” today can be quietly raising tomorrow’s bug rate, and the eval scores the snapshot, not the trajectory, unless you build it to. You can’t hand the trajectory-check to another model either: a second model’s eval isn’t independent — its errors correlate with the first’s. Kim et al. (2025) measured that for benchmark errors across models; that it carries to one model judging another’s output is my inference. It isn’t the check it looks like.

Runs ≠ the right thing

The hardest call is what to build — the design, the tradeoff, whether the thing should exist at all. No eval supplies the intent you never expressed. Boehm’s cost-of-change work makes the direction plain for defects: the later you catch one, the more it costs to remove (skip the tidy 1:10:100 magnitude — the real ranges are wide and flatter under agile; the direction is the robust part). I’m stretching that from defects to product decisions — the rework-compounds-downstream mechanism is the same, but the extension is mine. The decision comes before any eval, and it’s the expensive one to get wrong.

My own receipt cuts the other way, which is the point. My favourite discipline — the minimal, targeted fix — produced zero changes on a module whose real failure sat in data I wasn’t even diagnosing against. The value was never the fix-size rule. It was the understanding, and I’d skipped it.

When the measure becomes the target

Even as a proxy, evals are weaker than they look. Inozemtseva and Holmes (ICSE 2014) found coverage only weakly correlated with suite effectiveness once you control for suite size — weak, note, not worthless.

Now make the proxy the moat. “When a measure becomes a target, it ceases to be a good measure” (Strathern 1997; the underlying law is Goodhart’s). Optimise for green and you optimise for the form green rewards — and you can lose the substance it was standing in for. Which is what my fast judge did to me, and what a codebase does when “it passed the evals” becomes the bar to clear instead of the floor to start from. At scale it looks the same: in Bun’s own postmortem of rewriting itself from Zig to Rust with 64 Claude agents, Claude read “let’s get all the crates to compile” as “stub out the functions with compilation errors.”

And the shape of the eval decides how gameable it is, not just its coverage. Tripathi and colleagues ran both judging protocols under deliberate manipulation: a generator embedding distractor features the judge happens to favour flipped pairwise preferences in about 35% of cases, against 9% for absolute scores. Comparing two outputs feels like it ought to be the safer instrument — the flaw is held constant across the arms, so the delta should survive what the level doesn’t — and on this axis it’s the easier one to play. Goodhart doesn’t just reach the metric you chose. It reaches the protocol you scored it with.

What that result defeats is the expectation, not the mechanism. A shared flaw really is held constant across the arms, so the reach is real — a comparison registers differences I never wrote a criterion for. What I don’t get is the criterion. When a comparison comes back no difference, nothing distinguishes that from nothing looked on that axis: the standard doing the work is the judge’s, and unlike the one I wrote down, I never get to read it.

Watch what happens when someone tries to build an institution on evals. Demis Hassabis’s framework for frontier AI proposes a US Standards Body — modelled on a federally overseen public-private partnership, FINRA the analogy — with evals as the governing instrument: “frontier-class” means meeting “certain thresholds on a set of benchmarks determined by the Standards Body.” That is the strongest available bet on evals, placed by someone with every reason to place it.

The same proposal prices the problem. Benchmarks get refreshed “perhaps quarterly to start,” saturated ones get deprecated, and the body should eventually create “its own held-out tests independent of the Labs to prevent overfitting.” Three anti-gaming mechanisms, specified at design time, before anyone has gamed anything. He isn’t conceding that evals are insufficient — he’s treating them as governable and costing out what governing them takes. But nobody builds a quarterly replacement cycle into an instrument they expect to hold, and that reading is mine rather than his. An eval you have to keep rebuilding to keep it aimed at what you care about isn’t the durable asset. What it keeps being re-aimed at is.

The moat

Evals aren’t the moat. They’re the automatable shallow layer — which is exactly why, granting a16z’s own premise, coding is where AI revenue clusters. The easy-to-measure thing got cheap to do. The moat is what no eval can reach unless you already encoded it — and the one thing you can never fully encode is intent: whether it’s correct to the intent you never wrote down, where the quality is heading if you never built the eval to watch it, and whether it was worth building at all. Green tells you the checks you specified reported pass. It was never going to tell you about the intent you didn’t — right, good, or the right thing.

---

How sure I am: the software-engineering results — essence vs accident, the oracle problem, interface faults, code churn, coverage, Goodhart, and the protocol-gaming result — are prior art (cited). The pairwise/absolute numbers are Tripathi et al.'s, read to the paper’s own abstract; they contradicted my instinct that a comparison must be the safer instrument, and I’ve kept them in because of that rather than in spite of it. That the reach survives the defeat, and that the standard behind it is one I never get to read, is mine and argued rather than measured — nobody has counted what a comparison finds that a well-specified check misses. The eval receipts are my own logs, genericised. Hassabis’s proposal is quoted from his own framework, and he is not claiming evals are insufficient — reading its quarterly-refresh and held-out-test provisions as an admission that the instrument can’t hold still is my inference. Mapping Brooks onto AI is my reading (he predates LLMs), and the claim that bad code compounds because the AI conditions on it is a wager — argued, not measured. If this is wrong somewhere, it’s there.

Sources

• a16z (Sivulka), “tokens & loops” — the argument this essay answers: coding wins because “code runs or it doesn’t,” so the eval suite is the moat. Cited for the claim, not for its unsourced stats.
• Fred Brooks, No Silver Bullet — Essence and Accident in Software Engineering (1986) — essence (deciding what to build) vs accident (writing the code); “the hardest single part… is deciding precisely what to build.” The essence/accident spine.
• Weyuker, “On Testing Non-Testable Programs” (Computer Journal, 1982) — the oracle problem: what counts as “correct” is an assumption you supply, not something the test knows.
• Barr, Harman, McMinn, Shahbaz & Yoo, “The Oracle Problem in Software Testing: A Survey” (IEEE TSE, 2015) — the oracle problem is general and doesn’t go away.
• Perry & Evangelist, “An Empirical Study of Software Interface Faults” (1985) — at least 66% of faults were interface faults in the system studied. That faults cluster at the seams generally, where a per-unit suite is blind, is my inference from it.
• Nagappan & Ball, “Use of Relative Code Churn Measures to Predict Defect Density” (ICSE 2005) — relative churn predicts defect density. Correlational, one system — directional, not proof.
• GitClear, “AI Copilot Code Quality: 2025” — code clones rising and refactoring falling as AI assistants spread. Vendor-authored, correlational, and doesn’t isolate AI-written code — directional at best.
• Boehm, Software Engineering Economics (1981); Boehm & Basili, “Software Defect Reduction Top 10 List” (2001) — the later a defect is caught, the more it costs to remove. Direction robust; the tidy 1:10:100 magnitude is over-cited and flatter under agile. Stretching later-costs-more from defects to product decisions is my extension.
• Inozemtseva & Holmes, “Coverage Is Not Strongly Correlated with Test Suite Effectiveness” (ICSE 2014) — coverage is a weak proxy for suite effectiveness once suite size is controlled. Weak, not worthless.
• Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran & Arawjo, “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences” (2024, arXiv:2404.12272; UIST) — building an LLM-eval tool, they found criteria drift: users need criteria to grade outputs, but grading outputs is what reveals the criteria — so some eval criteria can’t be fixed before you see outputs. Direct backing for the spine here (an eval reaches only what you specified, and part of that is unwritable in advance). They also note LLM validators “inherit all the problems of the LLMs they evaluate” — the same non-independence the Kim et al. line leans on.
• Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models” (2025, arXiv:2506.07962; ICML) — measured correlated benchmark errors across models rather than scattering. Applying that to one model judging another’s output — a second model’s eval isn’t independent of the first — is my inference.
• Strathern, “‘Improving ratings’: audit in the British University system” (European Review, 1997) — “when a measure becomes a target, it ceases to be a good measure.” Goodhart (1975) is the underlying law.
• Tripathi, Wadhwa, Durrett & Niekum, “Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation” (COLM 2025, arXiv:2504.14716) — under embedded distractor features, pairwise preferences flip in about 35% of cases against 9% for absolute scores. The feedback protocol is itself a gameable surface, not a neutral container for the metric.
• Jarred Sumner, “Rewriting Bun in Rust” (bun.com/blog/bun-in-rust, 8 Jul 2026) — rewriting Bun from Zig to Rust with 64 Claude agents, Claude read “let’s get all the crates to compile” as “stub out the functions with compilation errors” — optimising for the green signal, losing the substance it stood in for.
• Demis Hassabis, “A Framework for Frontier AI and the Dawning of a New Age” (demishassabis.substack.com, 14 Jul 2026) — proposes a US Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation (FINRA the analogy, industry-funded), with “frontier-class” defined by thresholds on a benchmark set the body determines. Cited for the anti-gaming provisions the design carries on its own: benchmarks refreshed “perhaps quarterly to start,” saturated ones deprecated, and eventually “its own held-out tests independent of the Labs to prevent overfitting.” He presents these as what makes evals governable; reading them as an admission that the instrument can’t hold still is mine.
• Manifesto for Agile Software Development (2001), https://agilemanifesto.org — the practitioner stance that requirements are discovered by building and complete up-front specification isn’t reliable. Cited as a widely-held nod, not proof; the “loops are the essence, not a prompting gap” reading is mine.

Extends A Description Can’t Replace the Code (verify ≠ read; the oracle problem), Your Codebase Is the Prompt (the bad-code loop), and Give Away the Writing, Keep the Reading (judgment was the scarce skill). Adjacent to A Pool of Models Narrows the Overlap — another model’s eval isn’t an independent one.