Noticing Is Cheap, Judgment Is Scarce

“The art of being wise is the art of knowing what to overlook.” — William James, The Principles of Psychology (1890)

A code-review engine I built, run on a fourteen-line mechanical rename, cost about 788,000 tokens across seven agents and four minutes of wall-clock. The change renamed a symbol. There was nothing to judge; a scanner could have confirmed it was safe in milliseconds.

The cost was fixed per pull request, not proportional to the judgment the pull request asked for. The rename and a hairy 300-line refactor would cost about the same, because the boot sequence doesn’t care.

One thing to get straight before the numbers start, because the numbers will mislead you otherwise. Fidelity was the objective, and it isn’t a trade-off. It’s a constraint: find as much as is really there, while keeping false positives at zero. Not cost. Stated that way it’s a design brief rather than a dial — generous on the way in, because a dropped lead is unrecoverable and nothing downstream can judge what it never saw; ruthless on the way out, because nothing posts until something has tried to kill it and failed.

I’d already accepted that a staged reviewer costs more than an unstaged one, and took that trade on the assumption that unit prices fall over time while a missed bug doesn’t. What bothered me about the rename wasn’t that it was expensive. It was that the expense bought nothing: 788,000 tokens to establish that a symbol rename was a symbol rename. That’s waste, which is a different complaint from cost, and worth removing whatever your budget.

It’s also where I part company with DoorDash, who are explicit about taking the other road. “The single biggest decision in building this was refusing to optimize for ‘catch everything’,” they write, because the cost of catch-everything “is predictable: PR spam, muted notifications, ignored comments.” They’re right about the failure mode. I just don’t think the only cure is to give up recall — the spam comes from posting unverified candidates, not from finding them.

So I mechanised the noticing and metered it, expecting to find savings. What I found was where the money actually was.

BLUF: Bottom line up front

Most of what a review agent does is noticing — scanning changed lines against a big rulebook, checking whether a pattern appears. A deterministic scanner does it for free. The model’s real job, its scarce one, is judgment: given the surrounding code, does this actually matter. I’d been paying model rates for both.

• Mechanise the noticing. A scanner over the syntax tree, the types, the lint rules and the import graph notices; the model only judges. Measured, the noticing part works: fresh model tokens on the rename fell roughly sevenfold, and across a census of eighteen real pull requests the detectors ran clean.
• But mechanising didn’t cut the bill. It relocated it. Noticing went nearly free. The cost that’s left is the one judgment call, which is most of the ~$4 that fourteen-line change still ran. There’s no demonstrated total-cost win here, and that is the thesis: you can’t cheap out on judgment.
• The recall trap is real, and it bit me. Mechanising silently demoted the rule scan, and on the first honest comparison the new pipeline returned nothing where the old one had flagged five. I fixed the demotion. I still can’t prove the pipeline lost nothing.
• Cheaper overall was never the goal. Fidelity was. Cost is the thing most likely to fall on its own; a missed finding isn’t.
• Zero false positives isn’t politeness, it’s the only safe setting — because a finding has two readers and only one is a person. Increasingly the machine doesn’t relay the finding, it acts on it, and for that reader a false positive is a bad commit to code that was fine.

The problem is fixed context, not the diff

Each stage in the old pipeline started from scratch, reloading somewhere between 100k and 130k tokens of overlapping context — the rule index, the briefs, the diff. Seven stages, seven reloads, most of it identical. The diff itself was a rounding error against that fixed cost.

So the fourteen-line rename paid for seven full context loads to discover there was nothing to say. Cost tracked the pipeline running, not the change being hard.

The fix: notice cheaply, judge expensively

The split isn’t mine and neither is the word for it. DoorDash published this architecture first — “a staged architecture that separates noticing from verifying,” a lead scout flagging suspicious areas and deep reviewers confirming or dropping them — and they got there the way I did, by watching one agent fail at too many jobs at once. They also measured it at a scale I can’t: on a 105-case benchmark the staged reviewer cost five times the unstaged one and roughly tripled the real findings. So “staging makes it cheaper” was already known false before I started. The quotes, the figures and where the rest came from are in the sources below.

What’s left that’s mine is one substitution. In DoorDash’s design the scout is a model — a cheap one, but a model, with a model’s costs and a model’s moods. Alibaba’s open-code-review moves part of the work off the model, splitting “the review into a deterministic pipeline that handles what must never go wrong and an agent that handles judgment.” I’m pushing it further: the noticing shouldn’t be a model at all. A parser walks the syntax tree, the types and the import graph, and the model never sees a line it didn’t flag.

A rule you can test is a rule a parser can evaluate, and that’s the whole chain. “Don’t swallow an error” is a sentiment; “a catch block whose body neither rethrows, logs, nor assigns” is a predicate, and it costs nothing to check.

That substitution has three consequences I haven’t found written up. The bill doesn’t drop, it relocates, and where it lands tells you what you were paying for. The scanner becomes a single point of silent failure in a way a model scout isn’t. And the reason to prefer a parser over a cheap model turns out not to be cost at all.

When I audited the rulebook, a large share of the rules turned out to be mechanically detectable. Not “a model can probably spot this” — a parser can detect the candidate. Banned import here, missing await there, a type that doesn’t line up. The parser flags each as a candidate; confirming it is the judge’s call.

That splits the work cleanly. A deterministic scanner walks the syntax tree, the types, the lint rules and the import graph, and produces a shortlist of sites worth a second look. Then the pipeline sorts each flag by how much judgment it needs:

• Patterns nothing can excuse post straight out. No model involved; the scanner is judge and jury.
• Flags that are probably real go to a small cheap model, which confirms them in context for pennies.
• Real candidates that need the good model get it, but only on the shortlist rather than the whole file.
• Pure judgment — design smells, cross-module contracts, the actual bugs. No scanner finds these. This is what the expensive model is for.

Load the shared context once as a cached prefix, and the deep judgment pass only wakes up when a lead survives the cheap front. In that design the rename collapses to the scanner, finds nothing, and stops before the expensive pass. In the run I actually metered it didn’t: the deep pass still fired, which is why that rename still cost about four dollars. The short-circuit is what I’m aiming at. It isn’t what I measured.

What I measured: the bill moved, it didn’t shrink

I ran the same fourteen-line rename back through the mechanised version and metered it. The noticing nearly vanished. Fresh model tokens fell from roughly 788,000 to about 113,000 — the retrieval, the dedup and the broad rule scan all moved off the model into deterministic code.

That’s a fall of about sevenfold, and it’s a rough estimate rather than an exact delta: the “before” wasn’t instrumented tightly enough to certify every one of those 788,000 tokens was fresh.

On a census of eighteen real pull requests the detectors ran clean. Zero detector errors, meaning nothing crashed — a reliability check, not a claim of zero missed findings. The one tier that posts with no model behind it fired three times and was right all three. Three is a small number, so that’s a clean floor rather than proof of precision.

What I can’t show is a demonstrated total-cost win. That fourteen-line change still cost about four dollars and ran slower, roughly four minutes before against nine after. Two things drive the slowdown and only one is deliberate: I made the deep pass sequential for reliability. The more honest driver is that the detectors are tuned to over-fire so they miss nothing, which hands the judge more candidates than a tighter front would. Tightening them is the recovery path and it isn’t yet metered.

The before and after don’t even share an accounting basis, so I won’t pretend to a clean total delta. What the after does say is where the money now goes: almost all of that four dollars is one call, the deep pass carrying the full source so the model can judge in context.

Strip the noticing out and you don’t get a cheap review. You get a review whose entire bill is judgment.

So I won’t tell you mechanisation made it cheaper overall. On a small change it plainly didn’t. It made the cheap part cheap, left the scarce part scarce, and stopped me paying the scarce rate for the cheap work. Which was never the point, but it’s the part that shows up in a bill.

The recall trap, and how it bit me

Cheaper must not mean worse, and this failure mode is the dangerous kind: silent.

If the model only judges what the scanner shortlisted, a site the scanner fails to flag is a false negative the model never sees. A wrong flag costs a cheap check and gets caught. A wrong drop never enters the funnel, so nothing downstream can recover it. You can’t find a miss by looking only at what you found.

I don’t have to theorise about this one. I walked into it. On the first fair comparison — the mechanised pipeline against the old one on the same change — the new version returned nothing where the old one had flagged five rule findings. The cause was exactly the trap: mechanising had quietly demoted the pass that scans every changed line against the whole rulebook, down to a cheap floor that never ran the scan at all. The pipeline skipped the systematic scan, and because a scan that never runs produces no output, nothing complained. The warning sprang on the person who wrote the warning.

I fixed the demotion — restore the scan by default, shard a large diff so its size can’t trigger the skip — and confirmed the scan fires again. But I haven’t re-run the head-to-head, so I can’t yet say the pipeline lost nothing. I’ve fixed the cause, not certified the cure.

The reason I haven’t re-run it is its own finding. I pointed the fixed pipeline at a thirty-nine-file change and killed it after an hour with no output. On that diff, the deep pass carrying full source didn’t finish inside any window I’d wait through. The measurement that would settle recall is the one blocked by the thing that makes judgment expensive. I’m leaving that open.

Three rules fall out, and I’m holding to them:

• Every gate errs toward pass-through. That asymmetry is the whole design.
• The cheap floor must discover, not just verify. It reads the raw diff for what has no detector yet, because the scanner’s blind spots are where the misses hide. Not free, but a cheap model with no rulebook to reload is a small flat floor rather than the old explosion.
• Ship it provisional and in shadow. The mechanised front runs behind the existing scan on real changes until a non-inferiority check shows it loses no finding the old path caught. If it drops a real bug to save a dollar, it isn’t shipped.

The quieter failure

The demotion was a loud failure: the scan didn’t run, and the zero was obvious the moment I looked. There’s a quieter kind and I hit that too, where the scan genuinely works but only sometimes — which turns out to be worse than never, and is its own post.

Where this stands

One claim here is a bet rather than a result: that taking the rote noticing off the model’s plate makes it judge better, with better recall on the hard findings and less anchoring on cosmetic nits, because attention is finite and I’d been spending it on the wrong things. I believe it. I haven’t isolated it, and the precision census doesn’t speak to it. Treat it as why I expect this to be worth it, not as something I’ve shown.

The other open edge is recall non-inferiority: fixed at the cause, unproven at the pipeline, blocked on that large-diff run.

So, plainly. Noticing is cheap is measured, and so is one deep call dominating this after-run. Judgment is scarce is what I argue from those, not a second measurement. Cheaper-overall is false on small changes. Better-judgment and no-lost-recall are owed, not shown.

The lesson generalises past code review. Any agent doing a “scan everything against a checklist” job is probably paying model rates to notice what a parser would notice for free, and reasoning worse on the hard calls for the privilege. Find the noticing. Mechanise it. Spend the model on the judgment only it can do. And don’t expect the bill to drop so much as tell you what you were paying for.

None of which will collapse the bill, and I have stopped expecting it to. Cost is the problem I would bet on someone else solving; it has fallen every year I have been doing this. A reviewer that misses the bug, or cries wolf until you mute it, is the problem that stays mine.

---

How sure I am: The before and after costs, the eighteen-change census, the roughly sevenfold fresh-token drop, and the five-finding recall regression are metered runs of my own, genericised but real. The inverted-fixture run behind the quarter-of-the-time section is real too — thirteen fixtures, three runs, ten removal alerts across thirty-nine — but that corpus is unadjudicated: I haven’t ruled on which removals warranted a comment, so it establishes the clustering, not a recall rate. The cost-relocation reading is what the after-run supports; it explicitly does not support a total-cost win. Two things are unproven and marked as such above: that mechanisation improves judgment recall, which is a bet, and that the fixed pipeline is recall-non-inferior, which is blocked on a large-diff run that won’t complete. Those two are where I’d look first — which isn’t the same as knowing that’s where it’s wrong. The sibling post makes that argument against the reviewer, and it applies to me on the same terms: the misses you can name are the ones you already found.

Sources

• Metered before — my own instrumentation of the review engine: the ~788k-token, seven-stage run on a fourteen-line rename. Raw run logs, mine.
• Metered after — the same rename through the mechanised version: roughly 7× fewer fresh tokens, but ~$4 total and slower (~4 to 9 minutes), dominated by the deep judgment call. The write-up explicitly cautions against claiming a total cost or time win, since the before and after don’t share a token-accounting basis.
• Precision census — the mechanised detectors run read-only across eighteen real changes: zero detector errors; the tier that posts with no model behind it fired three times, three for three, no false positives (small n). The detectors also surfaced mechanical-rule issues the human reviews had missed, which is directional.
• Recall check — a planted-bug harness, clean on easy logic fixtures, plus a same-commit comparison where the mechanised pipeline returned zero rule findings against the old path’s five, root-caused to the demoted scan. Cause fixed and the scan confirmed firing; the strict head-to-head is not yet re-run.
• Prior art — where I got this. DoorDash, How we learned to trust our AI code reviewer and DoorDash built an AI code reviewer engineers actually listen to (careersatdoordash.com, 2026). The staged architecture that “separates noticing from verifying”, the lead-scout-then-deep-reviewer shape, and the term scout all come from here. Their v3 added the scout and they call it “the change that mattered most”, for attention rather than cost: “splitting noticing from verifying let us go deeper on the things that matter without spreading attention thin across the things that don’t.” Their 105-case benchmark has the staged reviewer at $3.91 per pull request against $0.75 unstaged, 504 real findings against 164, weighted recall 53.6% against 30.7%.
• Prior art — moving part of it off the model. Alibaba, open-code-review — a deterministic pipeline for file selection and rule matching, with an agent for judgment, plus independent positioning and reflection modules that verify each comment before posting. On their own benchmark against Claude Code: 385K tokens per review versus 5,664K, precision 33.9% versus 7.2%, and a conceded recall loss, 20.0% versus 28.9%, because the baseline “fired off thousands of findings”. Note it is a deterministic/agent split rather than a cheap-pass-then-deep-pass one, and the figures are totals rather than a decomposition.
• Prior art — deterministic matching feeding a model. CodeRabbit, Code Quality at Scale with AST Grep and LLMs (2024) — syntax-tree pattern matching feeding a model, which their author describes as using “AST Grep results as input for an LLM” to “provide contextual grounding”, extracting code elements “to constrain the LLM’s output”. Rules open-sourced at ast-grep-essentials. The original URL now redirects to their blog index; the quotes are the author’s on the Hacker News thread for it. Their matcher mainly grounds the model where mine filters what it sees, and the write-up carries no cost figures.
• Prior art — the pattern. Speculative decoding: a small model drafts, a large one verifies, because verifying costs less than generating. The same cheap-notice / expensive-judge shape, from inference rather than review.
• The typed-decision model class, and why I describe the shape rather than endorse a product. TypeSafe AI’s Jev (September 2026) is the prominent example: typed questions in, bounded decisions out. The launch claim was 193.6× faster and 444.6× cheaper than frontier models, but an independent benchmark ran it against nine cheap hosted models rather than frontier ones and the margins went: fourth of ten on accuracy for intent classification, sixth for injection detection, and third most expensive per thousand decisions at $0.0433 against Ling 3.0 Flash’s $0.0107 — because the typed-question format ships the full option list on every call, roughly doubling input tokens. Its genuine edge was tail latency, the only hosted model under 425ms at p95. It also cannot cache: cache, cache_control and store all return HTTP 400, which the benchmark calls “the only model here whose unit cost cannot come down”. n=200, so gaps under about five points are ties.
• Prior art — the gating. ncoevoet/claude-review-all, an open-source multi-agent reviewer. The borrowed pieces are the citation gate, the fail-closed completion gate, and the cheap batched verifier.
• Inverted-fixture removal probe — the thirteen planted-bug fixtures run with base and head swapped, so each diff becomes a removal. The model flagged the removal in ten of thirty-nine runs, clustered on the dramatic deletions and absent on eight of the thirteen. Reported deliberately as a shape rather than a rate: the corpus hasn’t been adjudicated for which removals deserved a flag.