You Can't Grade Your Own Misses

“Absence of evidence is not evidence of absence.” — Carl Sagan

Ask a code-review tool how good it is and you’ll get a precision answer: of the things it flagged, how many were real. That’s easy — every flag is a claim you can check. Recall is the other half, and — my inference — it’s the half nobody earns honestly: of the things it should have flagged, how many did it catch. A bug the reviewer missed leaves no flag, no trace, nothing to count. You can’t measure what didn’t happen by looking at what did.

So when a reviewer — human or model — is described as having “good recall,” the first question is: how did you see the misses? In my experience the answer is usually that you didn’t, and “good recall” means “I haven’t noticed it being wrong yet.” Optimise for the number you can see — precision — and you quietly select for a reviewer that says less. Information retrieval has known the shape of this for decades: you can’t compute recall without the complete set of relevant items, and for a reviewer that set — every bug that was actually there — is exactly what you never have.

The claim here isn’t that recall is unmeasurable. It’s narrower and more useful: you can’t grade it from your own output. You can get at it — but only with ground truth that isn’t yours, and even that runs out before the truth does.

Bottom line up front

• Precision is cheap to measure; recall is not. Flags are visible and checkable. Misses are invisible by construction — that’s the whole problem.
• You need ground truth that isn’t your own output. Three sources: bugs you planted (a floor on known cases, and the one that’s real ground truth), the human review that landed after yours (the field signal), and — the strongest — a second independent reviewer run on the same change. The latter two supply candidate misses you still have to adjudicate, not truth handed to you.
• A second reviewer suggests the truth is bigger than any one tool. Run two passes and treat what each catches that the other missed as a candidate miss (capture-recapture as a lens — the estimator’s independence assumption doesn’t hold for two models). In one real run they overlapped on one bug; each caught exactly two the other didn’t — so if those uniques hold up as real misses, the true set already exceeds either, and every run I added found more.
• A planted corpus can be shaped so the failure you care about can’t appear in it. Thirteen obvious planted bugs can show a reviewer sometimes drops one; they can’t show that a whole class is invisible to it. Running N cheap passes and unioning them shows a given miss varied across runs; it doesn’t show that nothing is blind.
• Define “miss” against a frozen snapshot. A miss is only a miss if the code was there when you reviewed. Anchor to the reviewed commit or you’ll score later edits as failures.
• Blind the adjudication and measure agreement. If the thing being tested helps decide what counts as a miss, you’re grading your own homework. Two blind raters and an agreement score (Cohen’s κ).
• Never tune and score on the same window. Freeze a holdout after a cutoff date. Tune before it, score after, or you’ve taught to the test.
• The denominator drifts, and has a floor you can’t see under. Mine misses from human review and you asymptote at human recall. Worse, the reference won’t hold still — one run is a noisy draw, and an evolving reviewer changes what counts as “the bugs” version to version. The bug nobody ever caught stays invisible to all of it.

Instrument one: plant the bug

The clean way to get a recall floor is to stop waiting for real misses and manufacture known ones. Take small changes, plant exactly one bug in each — a stale read, an inverted condition, an off-by-one, a dropped null-guard, a missing await — and run the reviewer. Ground truth is exact: you know where the bug is, so a miss is unambiguous. No PR, no posting, no waiting.

I did this. On five such fixtures, three runs each, the reviewer caught every one — fifteen for fifteen. And that buys almost nothing but a floor. Five easy, single-bug fixtures is a weak bar; 100% there proves the reviewer isn’t stochastically dropping obvious bugs, not that it out-recalls anything. It’s the pass/fail gate before the real question, not the real question. A recall claim that matters needs subtler bugs and real diffs — which sends you to the second instrument, and the second instrument is where it gets hard.

I’ve since grown the corpus to thirteen fixtures and run each three times against two models — a strong one, and a cheap one at about a ninth of the price. The strong model caught all thirty-nine. The cheap one missed one run of one fixture: the subtlest bug in the set, a stale read across an interleaved await. Then I did the arithmetic the price gap invites — eight cheap runs cost less than one expensive run — and unioned three cheap runs together. Union recall: thirteen for thirteen. The miss was stochastic, caught in two runs of three, and three draws recover it.

That number is worth less than it looks, and the reason is the useful part. Unioning the runs shows that this one miss varied run to run. It cannot show there’s nothing the cheap model is blind to. The union arithmetic needs p > 0; N runs of p = 0 is still zero. A blind spot would look like a fixture the strong model catches every time and the cheap one never catches across many runs — and my corpus can’t surface that class at all, because every bug in it was planted by me and planted obvious. Thirteen easy bugs can show that a reviewer sometimes drops one. They can’t show that a whole class of bug is invisible to it.

That’s the trap under the trap. The frozen holdout and the blind adjudication guard against measuring dishonestly. This one is about the shape of the ground truth: a corpus can be built so the failure you care about most cannot appear in it — and it will still hand you a clean number. Mine hands me 100%. The fix isn’t more runs, or more of the same fixtures; it’s harder ones — cross-file staleness, an invariant broken several calls away, an ordering dependency that only bites under a particular interleaving. I’m building that corpus now. Until it exists, “caught everything” means “caught everything I knew how to plant.”

Instrument two: mine the review that came after

Real recall lives in real PRs, and the ground truth is already there: the human review that followed your reviewer. If a person flagged something your tool didn’t, that’s a candidate miss — for free, from work someone did anyway. This is defect escape rate pointed at the reviewer: the field’s standard measure of misses is what escaped to the next stage — here, what a later reviewer caught that you didn’t.

The trap is the timeline. To count fairly you have to reconstruct the exact state the reviewer saw — fetch the PR at the commit it reviewed — and only count a human comment as a miss if the code it lands on existed in that snapshot. Comment on a line added later, and it was never yours to catch. Get this wrong and you’ll punish the tool for not seeing the future.

I nearly got this wrong in the first ten minutes. The command I reached for handed me the PR’s current head — which quietly included a later commit that had fixed the very bugs in question — not the state the reviewer actually saw. Score against that and every “miss” is a phantom, and every catch a lie. I only caught it by pinning the exact reviewed commit. The snapshot rule isn’t theoretical hygiene; it’s the first thing that bit me.

I’ve built the reconstruction primitive — pull the PR at the reviewed commit, map each later comment to that snapshot — but I want to be straight that this is a method I’ve built, not a recall number I’ve earned. No mined miss-corpus, no head-to-head yet. The design is the contribution here; the number comes later, and only if the rest of the discipline holds.

Instrument three: run a second reviewer

A stronger source of outside ground truth than a human who happened to comment later is a second reviewer you run on purpose. Point two reviewers at the same change; whatever each catches that the other missed is a candidate miss — surfaced deliberately instead of waited for. There’s a name for the idea — capture-recapture, the mark-and-recapture estimator borrowed for software inspection: from how much two independent passes overlap, estimate how many defects both missed. Borrow the intuition, not the arithmetic: the estimator assumes the two passes are independent with equal catchability, and two models reviewing the same code are neither — they fail on the same hard bugs (correlated errors). It’s a lens, not a calculator.

I ran it. A baseline reviewer and a deeper engine, same real change, pinned to the same commit. In isolation the baseline looked good — a handful of solid findings, nothing obviously wrong. Then the deeper engine found more, including logic bugs the baseline hadn’t. Overlap: one bug; each found exactly two the other didn’t — though “found” is doing work those uniques haven’t earned yet: some are false positives, not misses, until they clear the blind adjudication I demand two sections down. So if those uniques hold up, the true set exceeds either one — and since every method and re-run I added surfaced something new, I have no reason to think the union is complete either. I can’t prove a bug both missed exists — that would take the oracle I don’t have. That’s the point: from the inside, the incompleteness is unfalsifiable, which is exactly why “good recall” is unearnable.

And you never get a clean read on the number. For frozen code and a fixed definition the true defect set doesn’t move — but everything you’d use to estimate it does. A single pass is a noisy draw: same reviewer, same commit, same settings returned a different subset run to run — three passes, three different sets of three, two real bugs in none of them. The instrument changes too — the engine is under active improvement, so the version that finds N today isn’t next week’s. And for a judgment reviewer the definition of a defect isn’t fixed either: the rules are opinion, and they evolve. Pin the code, pin the reviewer version, freeze the rulebook — and you’re still reading one draw of a stochastic instrument. Not just “you can’t see the bug no one caught”: you can’t hold the target still long enough to aim.

The traps that turn a recall number into a lie

Three traps, each the gap between a measurement and a flattering story:

• Tune-and-score on the same PRs. If you adjust the reviewer against the misses you found and then report recall on those same PRs, you’ve fit to the test set. Freeze a holdout after a cutoff date: tune on the past, score on the future.
• Adjudicate with the thing you’re testing. If your tool (or you, invested in it) decides what counts as a real miss, the number is cooked. Blind the adjudication, use two raters, and report their agreement (Cohen’s κ) so “we agreed it’s a miss” isn’t one invested party’s call. (κ measures agreement, not truth — two raters can agree on a wrong definition — so it guards against solo bias, it doesn’t certify the call.)
• Forget the ceiling. Misses mined from human review asymptote at human recall — you can only find bugs someone eventually caught. The bug no human flagged either is invisible to this method too. Say so; don’t let “matches human recall” masquerade as “catches everything.”
• Assume “default” still means the baseline. I ran a two-arm comparison where the control passed no model flag and inherited the default — which a config change had switched to the treatment model the day before. Both arms were the same model. Nothing in the output said so, because the aggregate throws away which model each agent actually resolved to. The tell was the bill: cost per fixture came in at $0.119 against $0.106, where an eightfold gap was expected. Name the model in every arm explicitly and record the resolved version in every row. A control arm that has quietly become a second treatment arm returns a beautifully consistent result about nothing.

The point

Recall is the number everyone wants and — my inference — almost no one earns, because earning it means seeing what you missed, and missing is silent. If a tool — or a vendor, or you — claims strong recall, the honest follow-up is mechanical: against what ground truth, on what frozen snapshot, adjudicated by whom, on which held-out window, with which version of the reviewer? In my experience the questions usually have no answers, which is its own answer. Precision tells you whether to trust what a reviewer says. Recall tells you whether to trust its silence — and silence is much harder to audit.

---

How sure I am: The planted-bug results are real runs — fifteen for fifteen on the first five easy fixtures, and later thirty-nine for thirty-nine on thirteen — and I’ve flagged exactly how little either proves: a floor, not superiority. The cheap-model comparison (one stochastic miss, recovered by unioning three runs) is thirteen fixtures on one model pair in one sitting; read the union result as “that miss varied across runs,” never as “nothing is blind.” The subtle-bug corpus that would settle it doesn’t exist yet. The two-arm mix-up that cost per fixture caught is a real mistake I made, not a hypothetical. The snapshot-reconstruction harvester is a design plus a built primitive, not a mined recall corpus. Instrument three — the two-reviewer run — is real but small: a baseline reviewer against a deeper engine on one change, one config, a handful of passes. Read it as directional — a hint that each output under-covers if those uniques hold up, and that the reference drifts; the union’s own completeness is unknown, and it’s not a measured recall rate. The run-to-run and version-to-version drift I watched first-hand; the frozen-snapshot trap I actually tripped on before catching it. The disciplines — frozen holdout, blind adjudication, Cohen’s κ, the survivorship ceiling — are standard measurement practice I’m adopting, not numbers I’ve published. If you take one thing, take the questions, not a number.

Sources — pointers to the relevant literature and my own working notes, not exact citations I’ve verified line by line.

• My own recall-eval notes — a planted-bug harness (first five fixtures, three runs, all caught; later thirteen fixtures × three runs per model arm) and the design for a snapshot-reconstruction harvester that mines the later human review for misses against the reviewed commit. Working documents, genericised; no production recall figure claimed.
• My own cheap-vs-strong model comparison on that thirteen-fixture corpus — one stochastic miss on the subtlest bug, recovered under a union of three cheap runs; and the two-arm mix-up where the control inherited a changed default and only cost per fixture gave it away. Genericised working notes. The subtle-bug corpus that would separate a stochastic miss from a blind spot is designed, not built.
• Cohen’s κ — inter-rater agreement, the standard guard against a single invested adjudicator.
• Survivorship bias (Abraham Wald, WWII aircraft) — you can only study what came back; here, you can only mine the misses someone eventually caught.
• Held-out / time-forward validation — the standard fix for teaching to the test.
• The relevant information-retrieval literature on precision/recall — recall is undefined without the complete relevant set; estimating it without ground truth is a known hard problem. The core of why self-measured recall is suspect.
• The relevant software-quality literature on Defect Escape Rate / escaped defects — the metric for misses: what slipped to a later stage. The harvester is DER aimed at the reviewer instead of the release.
• Capture-recapture / mark-recapture (the Lincoln–Petersen estimator) and the software-inspection literature applying it to fault content from independent inspectors (e.g. Eick et al.) — the basis for instrument three: the overlap between two independent passes estimates how many defects both missed.
• A two-reviewer cross-methodology run — a baseline AI reviewer vs a deeper engine on one real change, pinned to the reviewed commit; genericised. Overlap of one, each exactly two uniques (unadjudicated) — so each output under-covers the other if those uniques hold; the union’s own completeness is unknown. Findings drifted run-to-run and version-to-version. Directional, not a recall rate.

---

Related: Evals Aren’t the Moat (green ≠ correct; this is the recall version — measured ≠ measured honestly), Noticing Is Cheap, Judgment Is Scarce (the recall trap, where a silent miss nearly shipped), A Pool of Models Narrows the Overlap (coverage isn’t a verdict; a recall number isn’t either until it’s adjudicated).