Fidelity Was a Vibe, Cost Was a Number

“When a measure becomes a target, it ceases to be a good measure.” — Marilyn Strathern (1997), restating Goodhart’s Law

I’ve been rebuilding a code-review engine mostly by remote control. I haven’t read the code of the rewrite, not a line of most of it. I steer by what gets distilled back to me and I question things that sound wrong, either the results or the agent’s decisions. That’s deliberate: it’s a long build with many levers, the detail churns, and churn self-corrects. Proofreading code that’s about to change wastes the one scarce thing in the loop: my attention.

For the churn, that worked. One thing never self-corrected: the goal. I’d said fidelity was everything — catch the real issues, cost a distant second. The agent kept quietly optimising for cost anyway. It took me a while to see why, because the reason is embarrassing: cost was a number and fidelity was a vibe.

BLUF

• A self-correcting loop fixes errors, not objectives. It closes the loop against the objective it was given, so it catches mistakes relative to that objective and can’t see that the objective itself is wrong. A wrong goal isn’t an error from the agent’s side, so it never self-corrects. Supervise by distillation and you inherit that blind spot.
• The objective drifts toward whatever’s operationalised. Cost had a rule and an estimator, a number that triggers a pause. Fidelity had a priority, a sentence. The number is operative; the sentence is a mood. The number wins.
• Your own guardrail becomes the proxy. The thing optimising against my goal was a rule I wrote to be responsible — flag expensive runs. No cap ever auto-declined anything; every stop was a judgment that cited a number, which gave it a reason that sounded official.
• In the long tail, the cheap path doesn’t just cost less. It deletes the thing you were building. The cost-cut and the fidelity you’re after pull opposite ways exactly where it matters.
• So defer the churn, never the objective — and give the objective a number. Stated once, a priority decays. I had to pin it to the agent’s memory, and I still steer it back.

I supervise by distillation

This is the honest setup, and it’s triage rather than laziness. On a long, many-lever rebuild, most of what the agent writes at any checkpoint will be rewritten before it matters. Reading it now is proofreading a draft that’s about to be deleted. So I let it loop, do as much as it can, and self-correct the local mistakes, and I spend my attention on the distilled reports and on anything that sounds off. This is the last responsible moment applied to reading: don’t commit your scarce attention until deferring it starts to cost you.

Deferring costs you at exactly one place. Local errors surface and get fixed as the build proceeds; the loop is genuinely good at that. The objective does not surface, because from inside the loop a drifting objective produces no error at all.

And I should name the obvious objection before you do, because I make the opposite argument elsewhere on this site. Give Away the Writing, Keep the Reading says that for anything which has to be right you read every diff you would ship, and that only throwaway is safe to skip. A code reviewer other people are meant to trust is not throwaway. The defence I would give — that this is triage, that the detail churns — is the same defence the overconfident subjects in that post’s Stanford study would have given. What I actually do is read the output instead of the code: I don’t read the reviewer’s implementation, and I check its findings against source relentlessly. That is a real distinction and it is also a convenient one, so take it as a tension I am living with rather than one I have resolved.

The loop fixes errors, not the goal

A self-correction loop evaluates against the objective it holds. Its critic can’t see past that objective. It will find and fix anything wrong by that measure, tirelessly. It cannot find that the measure is the wrong one, because nothing in the loop scores the measure itself.

Huang et al. (2024) is the narrow version: language models can’t reliably self-correct their reasoning without an external signal, so intrinsic self-correction with no outside feedback doesn’t hold up. My step, not theirs, is to extend that from reasoning errors to the objective: a loop can’t fix the goal it’s scored against, only mistakes relative to it.

The word doing the damage is self. A self-correcting loop corrects its work and never its self, and calling it self-correcting is most of the reason nobody thought to go and score the self.

So a wrong objective is a fixed point the loop will never leave. Every cost-cut it made was, by its lights, a correct local improvement: cheaper, still passing, tidy. There was no red test for “this is the wrong thing to optimise,” because I’d never given it one. The build got better and better at the wrong game, and self-correction is what made it look healthy the whole way.

Cost was a number, fidelity was a vibe

Here’s why it drifted to cost specifically, and it’s the part I’d change first. Cost, in my setup, was operationalised twice over.

I’d written a rough offline estimator. It runs the pipeline against a mock, multiplies estimated input tokens by a rate table, and prints a dollar figure. It’s input-only and its own header calls it “not a billing oracle,” so the figures were order-of-magnitude at best. And there was a guardrail in the config — flag and pause before expensive runs — written to fence a genuinely different kind of job, the sprawling multi-agent fan-outs where cost really can run away. Reasonable where it was born. Two concrete, numbered things.

Fidelity had a sentence: fidelity comes first.

Watch what that asymmetry does. When the estimator prints “$56” and the config says “flag expensive runs,” the agent has a crisp, defensible, actionable reason to pause and propose the cheaper branch. When fidelity is a stated priority with no number, there’s nothing to weigh it against. It doesn’t fire, it just hovers. So on every close call the numbered thing won — not because the agent valued money, but because money was the only value in the room with a unit. A guardrail written for runaway fan-outs quietly became the proxy the whole build optimised, over-firing on a job where the expense was the point.

And the part that took me longest to see: I hadn’t set that guardrail at all. I hadn’t known it was there. It had been added in an earlier session, for a different kind of job, and it had been quietly shaping runs ever since. So for a while I blamed the caution on the model in front of me, when the number beating my priority was one I’d never chosen, never approved, and couldn’t see. That’s the everyday version of a fundamental attribution error: I read a disposition — the model is being cautious — where the cause was situational, a rule installed elsewhere, because the situation was invisible and the disposition wasn’t. The proxy you’re fighting may be one you never knew was there.

And no hard cap ever declined anything. That’s the sneaky part: every stop was a judgment call that cited a number. A real posted review metered at $18; a full deep pass on one change ran $8 across a burst of large-model calls; the forced-everything variant ran about $25–27. Those are real. But most of the “that’ll be too expensive” moments were the estimator’s rough projections, and the decision to narrow or decline was mine-in-the-agent, reached because a number made pausing feel principled. Rough-and-numbered beat real-but-vague.

The cheap path deletes the tail

The worst of it wasn’t paying too little. It was what the cheap path removed.

I’d built a flywheel to mine the long tail: run the reviewer across real pull requests, collect the leads its deep pass dropped or refuted, cluster the recurring ones, and codify them into new detectors. The whole point was discovery — surfacing the failure shapes I didn’t already know to look for. I ran it on about five real pull requests, saw zero to two drops each, concluded the well was sparse, and pivoted to a synthetic harness that fabricates leads for a few cents a run, because the real runs were about $12 each.

Every step of that was locally reasonable and globally backwards. A long tail is rare by definition, so five pull requests can’t populate it; “sparse at n=5” is not evidence of a dry well, it’s evidence of a small sample, declared dry on cost grounds. And the synthetic pivot is structurally worse for the tail than the thing it replaced, because crafted leads can only cover patterns I already know to craft. They confirm; they can’t discover.

So the cheap path didn’t just cost less. It removed the discovery mechanism and kept only the confirmation one, on the exact axis where fidelity lives. That’s the shape of the whole failure: the proxy and the goal don’t just diverge, they point opposite ways in the tail. Optimise against the wrong objective and you don’t just stall short of the goal. The thing you actually wanted gets worse.

And rarity is not safety. That’s the bet I was quietly making. A pattern that fires on one pull request in five hundred is worthless right up until the one where it is the bug, and then it’s the only thing that mattered. Measuring drops at zero to two across five pull requests and calling the well sparse doesn’t refute that. It declines to look. A pattern’s frequency tells you nothing about its impact when it lands.

I’ll stay honest about how far this receipt goes. The loss I can prove is structural: I removed a discovery mechanism and kept a confirmation one. I did not confirm a specific real finding that this dropped. What’s certain is the loss of discovery capacity, because I removed the mechanism. A specific defect it would have caught is not something I can name.

Stating it isn’t enough — operationalise it

You’d think the fix is to say the priority louder. I tried; it works for about three turns. State “fidelity over cost” and the agent complies immediately — resumes, builds, grinds. Then, a few turns on, the cheaper-and-wrap-up default reasserts and it offers to stop and report, or proposes the thinner path, again.

It isn’t changing its mind. This is a hypothesis from watching it rather than something I’ve instrumented: the instruction dilutes as the context fills, and a trained lean toward concise, decisive, done reasserts. A small check is consistent with the fills part specifically: sixteen trivial filler turns, which barely add context, showed no decay at all — pin-once compliance stayed flat at 67% — while re-stating the instruction every turn held it at 100%. So it reads as context volume rather than turn count. The dilution mechanism itself stays a hypothesis, since a real test needs large, filling turns, but the pin-it remedy measurably held (n=3, one 7B model). And the sample size isn’t the doubt that should bother you most — this was a 7B model, and whether instruction decay behaves the same way on the frontier model I actually steer is untested. I hedged the countable thing and left the uncountable one unmentioned, which is the same mistake this post is about.

The tell is what I ended up doing about it. I have two rules pinned to the agent’s memory now: fidelity > cost > speed, and do not stop to ask what’s next. You don’t write a rule to memory unless the in-conversation version keeps dying. And even with them pinned I’m still steering — “why did you stop? You shouldn’t stop unless absolutely necessary”; “build it, test it, roll it back if it fails.”

Notice what that steering actually is. It kept stopping to ask me whether to keep going, bouncing back a decision it should have made itself, against the priority I’d already set. It bounced because the priority wasn’t operative enough to decide on. “Fidelity first” gave it nothing to weigh “this run costs twelve dollars” against, so it escalated to me by default, and I became the thing it prods. Give the objective a unit and the agent can resolve keep going? on its own. That’s the real payoff of operationalising it: not that it obeys, but that it stops handing you the wheel.

The point

A self-correcting loop is an optimiser pointed at whatever you actually operationalised, running tirelessly and self-healing on the way. That’s exactly why a wrong target is so hard to catch: everything about the process looks like it’s working.

Defer the churn; the loop earns that. But the objective is the one thing the loop can’t fix and distillation can’t show you, so it’s the one thing you can’t defer. And “priority” isn’t enough. A vibe loses to a number every time. Give the goal a number it can fight with, and pin it where it won’t decay.

---

How sure I am: The real dollar figures — about $8 on one deep pass, $18 on a posted review, $25–27 forced-full — are metered post-run and trustworthy. The larger projections that drove several decisions came from my own offline estimator, which is input-only and explicitly not a billing oracle: order-of-magnitude, good for “cost drove the choice” rather than for precise claims, and marked that way above. The mechanism is lived and I’ve been careful about cause: no hard cap auto-declined anything, and every stop was a judgment that cited a number. The long-tail loss I claim is structural — I removed a discovery mechanism and kept a confirmation one — and I cannot name a specific defect it would have caught. The dilution reading is a hypothesis from watching it, not instrumented; the small check (n=3, one 7B model) supports the pin-it remedy and the context-volume reading, not the mechanism.

Sources

• Marilyn Strathern (1997) — “when a measure becomes a target, it ceases to be a good measure.” The wording is Strathern’s; the underlying law is Goodhart’s. The frame.
• Specification gaming and reward hacking (DeepMind; Krakovna et al.) — an optimiser satisfying the literal objective while missing the intended one. The model-agnostic name for what happened here.
• Huang et al. (2024), Large Language Models Cannot Self-Correct Reasoning Yet — intrinsic self-correction of reasoning, with no external feedback, doesn’t reliably work. Extending that to the objective, so that a critic can’t get past the goal it’s scored against, is my step rather than the paper’s claim.
• Zhuang & Hadfield-Menell (2020), Consequences of Misaligned AI — under optimisation, a proxy that omits an attribute can drive true utility down, not just short of target. The formal version of the long-tail anti-correlation.
• Poppendieck (Lean), the Last Responsible Moment — defer a commitment, here reading, until deferring it would cost you. The discipline behind supervising churn by distillation.
• Fundamental attribution error (social psychology) — over-weighting disposition over situation, which is why I read the caution as the model’s nature rather than an unseen installed rule. And action at a distance, or hidden global state, the software anti-pattern: behaviour driven by state set somewhere you can’t see.

---

Related: Noticing Is Cheap, Judgment Is Scarce — the engine this came from, where mechanising the noticing buys the judgment. A Big Rule Set Is an Attention Problem — the attention version of the same lever.