Buy Back the Coverage

“To the curious incident of the dog in the night-time.” “The dog did nothing in the night-time.” “That was the curious incident.” — Arthur Conan Doyle, “The Adventure of Silver Blaze” (1892)

I had already published the warning: if a model only judges what a scanner shortlisted, a site the scanner fails to flag is a false negative nobody ever sees. A wrong flag costs a cheap check and gets caught. A wrong drop never enters the funnel.

Then I mechanised the review pipeline I’d built, to cut the bill. And mechanising quietly demoted the rule scanner — the pass that reads every changed line against the whole rulebook — down to a cheap floor that never ran the scan at all.

I found out by accident. I ran the new pipeline against the old one on the same change, for an unrelated reason, and the new one returned nothing where the old one had flagged five rule findings. The warning sprung on the person who wrote the warning.

The lesson is not “be more careful.” I had the general rule. What I did not have, anywhere on disk, was a line saying this build gave up the systematic whole-rulebook scan in exchange for cost. That specific exclusion was never written down as a decision, so nothing was watching it, and nothing could have complained. A skipped notice produces no output. The zero looked exactly like a clean run.

Bottom line up front

• Every method that narrows your view has a bill: the coverage it excluded. A scanner shortlist, a gate, a sample, a proxy metric, a filter, a comparison you chose not to run. The excluded part doesn’t stop mattering. It stops being visible.
• So treat the exclusion as a decision, not a side effect. Do two things: aim a compensating check at what the method cut out, and record the gap beside the decision that created it — not in a notebook, and not in your head.
• Aiming is the part you control. Whether the second check reaches the excluded region, or just shares the first one’s blind spot at twice the price, this post does not settle.
• Write the gap down. An unnamed gap produces no signal. A wrong output complains. A missing one doesn’t. In both episodes I have, whether the gap got found came down to whether some unrelated check happened to cross it.
• The same move is in four of my published posts. I did it once. I prescribed it three times. I never named the rule. Naming it is what lets you carry the move to a surface you haven’t filtered yet. Recording the gap is what stops the version I shipped.
• Limit: I can show that named gaps get placed in a durable record. I cannot show that anyone comes back for them.

The move I keep reaching for and never named

Going back through my own posts, the same shape is in four of them. Each time a method narrows the view, and each time the post reaches for the excluded coverage — without ever saying that’s what it’s doing. What it doesn’t do is finish, three times out of four, and that turns out to be the interesting part.

Where	What the method cuts you off from	What checks the excluded coverage	Did I actually do it?
Augmented Reading	Everything the machine pass said nothing about	Route the deep read there — and carry the limiter: “a flag marks a point, not a neighbourhood”	I did this. The post describes it as current practice: “it’s caught things I missed.”
Skills Are a Codebase	When the skill fires; gating the eval to grade output “says nothing about when the skill fires”	Shouldn’t-fire cases alongside should-fire — “only a shouldn’t-fire case can” tell you a skill is waking up where a sibling should own the task	I prescribed this. The post says what the harness needs; I have not built it.
Noticing Is Cheap	Sites the scanner never shortlisted, which the judge therefore never sees	“The cheap floor must discover, not just verify… because the scanner’s blind spots are where the misses hide”	I prescribed this after the failure. It is one of the three rules that “fall out”.
Build the Operator	Whether training bought anything over pasting the rule into the prompt	The comparative held-out-rule eval — “that delta is the whole claim”	I have not run it. The post ends: “it’s the eval I haven’t run.”

That last column is the one I didn’t expect when I went looking, and I’d been about to publish the table without it. Four surfaces, one move, done once. The other three are promises I wrote down and did not keep.

That sharpens it. If I had done this four times, “I never named the general rule” would be a curiosity. Having prescribed it three times and completed it once, the missing name looks like the reason: there was no rule to carry. Each post re-derived it, made a local promise, and left it there. Nothing accumulated. When the fifth surface arrived — the mechanisation — there was nothing to apply.

My bet: an unnamed practice doesn’t transfer. You rediscover it per surface, and the rediscovery is expensive enough that you’ll skip it somewhere. One episode is what I have — I failed to transfer a rule I knew. That’s a demonstration that it can happen, not evidence about how often.

Why the writing-down is the half that matters

I went looking for the receipt I expected: a gap left unnamed that surfaced later as a user complaint or an incident, and felt at the time like bad luck. I didn’t find one. What I found were two unnamed gaps that both cut the other way, and they turn out to make the better argument.

The first is the demotion above. The second is from the same pipeline: the reviewer had no check at all for behaviour a diff removes — a deleted call, a dropped guard, a catch that quietly stopped reporting. I found that one by inverting my planted-bug fixtures on a hunch. The model flagged the removal in about a quarter of runs, and the quarter clustered hard: three for three on a deleted await store.flush(), three for three on a removed error-swallow, and nothing whatsoever, in any run, on eight of the thirteen fixtures.

Neither of those arrived as bad luck. Both arrived as findings from deliberate work I happened to be doing for another reason. That kills the claim I started with. Leaves this:

Both gaps were silent, and each was found only because an unrelated check crossed it. The demotion: “because a skipped notice produces no output, nothing complained.” The removal class: “it doesn’t present as a gap. It presents as coverage.” That’s the same property twice. In both, discovery had nothing to do with how bad the gap was. It came down to whether something else I was doing happened to pass through it.

“You’ll eventually get unlucky” can’t be checked against anything. “Nothing will tell you” can: point at a gap and ask what output it would produce if it were costing you. If the answer is none, then nothing in the system is going to raise it, and you write it down beside the decision.

The demotion removes the excuse. I wasn’t ignorant of the failure mode; I had published it. Knowing the rule in general did nothing, because the decision that created the gap happened in a different place from the post that named the danger, and the two never met.

“Written down” has to mean somewhere in particular

Not a limitations section. Not a chat log. The test is: the artifact someone opens when they reopen the decision.

For a design choice, that’s the design doc. For a filter in code, it’s next to the filter. For an eval, it’s the eval’s own description of what it does and doesn’t measure. The gap has to live beside the decision, because the moment the gap matters is the moment somebody is changing the thing that created it.

I have one receipt for this working as intended, and it’s a positive control rather than a save. Auditing an instrument mid-run, I wrote three known gaps into the durable design record instead of keeping them in my head — the note I left myself was literally “known gaps stay in the RFC rather than in my head” — alongside a correction to a catalogue row that had been claiming evidence that didn’t exist. Those gaps are now available to whoever reopens that design.

Nobody has come back for them. They are available, not revisited. And I recorded the gaps; I did not record a check beside them.

The strongest objection

“How do you know the check you bought reaches any of the excluded coverage — rather than sharing the original method’s blind spot, and just costing you more?”

I don’t. This decides whether the title is a rule or just a phrase. A second reader who learned to read the same way misses the same things. A discovery pass built on the same model as the scanner inherits the model’s blind spots wholesale. Nothing here establishes that any of my checks reached new ground. I need a defect log to know whether two nominally-different checks miss different things. I am still building it.

The smaller claim survives: name the gap so a check has somewhere to point. Without the gap, you cannot ask whether the check reaches it — you have two checks and no account of what either is for.

The second objection: “this is ‘document your limitations,’ which every engineer already knows, and every LIMITATIONS section is a graveyard nobody reads.”

Half of that lands, and it’s the half I conceded above: I have no evidence anyone reads the record. If the record is never reopened, writing it buys nothing.

But it misses the trigger. The instruction isn’t note things you’re unsure about — that’s a judgement call, and it’s the call I didn’t make: the mechanisation registered as a cost win, not as a coverage decision, so there was no moment where I felt uncertain and declined to write it down. The instruction is mechanical: any time a method narrows the view, it has excluded something; name what. A shortlist excludes the un-shortlisted. A pass/fail on output excludes the trigger. A non-comparative eval excludes the comparison. It runs without a judgement call. You just have to notice you added a filter.

And the other half isn’t documentation at all. Aim the check first, record second. My four posts all reached for the check and none of them recorded the gap — and that combination is precisely how the demotion shipped. The reverse combination, recording without checking, is the graveyard the objection describes. A note with no compensating check is a confession. A check with no note is the scanner I lost: it was doing its job right up until a rebuild had no reason to keep it.

You can’t buy back all the coverage, that’s just “do more work.” No — the buy-back is scoped to what this particular filter excluded, which is a much smaller and much more specific set than “everything,” and it’s usually a different kind of check rather than more of the same. Shouldn’t-fire cases aren’t more should-fire cases. A discovery read isn’t a bigger scanner.

What this isn’t

Two things look like this. They are not the same.

Not the claim that a filter makes a defect class invisible. That was the strong version and I retired it. I went to Basili and Selby’s testing-strategy comparison (TR-1501, 1985) expecting support and it doesn’t give any: two of their six defect classes come out flat across techniques, and what they vary is technique, not the surface you’re looking at. The honest claim is about signal, not invisibility.

Not the question of what gets to say no. Naming the positive term an instrument must be pointed at, and naming the gap it therefore isn’t pointed at, look like one idea and aren’t. They come apart both ways: you can validate a target properly while never recording the remaining gap, and you can record a coverage gap without having any instrument capable of saying no. Adjacent, separate.

Whether the check is independent of the one it compensates for is different again. It belongs with the objection above, not here — this post’s imperative depends on the answer.

How sure I am: Two runs are real. The demotion returned zero findings where the old path returned five, root-caused to a scanner silently demoted by mechanisation. The removal probe produced ten alerts across thirty-nine runs on thirteen fixtures, but I have not judged which removals warranted one, so read it as a shape and not a rate. The four posts are a reading of my own corpus, verifiable line by line, not a measurement: one performed, three prescribed. One design record holds three named gaps, and that is the whole of the positive control — co-recording a check beside a gap is my prescription with nothing behind it. I cannot show that anyone reopens the record, that a second check sees what the first missed, or that naming the move makes it transfer; I have one failed transfer, which shows the failure is possible and nothing about its rate. I am also not claiming an unnamed gap will eventually cost you something — my own two receipts point the other way, and that is why the claim here is about signal rather than consequences. One thing is retired outright: that a filter makes a defect class invisible. If this post is wrong somewhere, it’s in the assumption that the record gets reopened.

Sources

• The recall regression — my own instrumented run: the mechanised pipeline returned zero rule findings where the old path had flagged five on the same change, root-caused to a rule scanner demoted to a floor that never scanned. Cause fixed and the scanner confirmed firing; the strict head-to-head is not yet re-run. Genericised, author’s own. Written up at greater length in Noticing Is Cheap.
• Inverted-fixture removal probe — thirteen planted-bug fixtures run with base and head swapped so each diff becomes a removal; the model flagged the removal in ten of thirty-nine runs, clustered on the dramatic deletions, absent in all runs on eight of the thirteen. Author’s own, genericised, deliberately reported as a shape rather than a rate — the corpus has not been adjudicated.
• The positive control — my own mid-run instrument audit: three known gaps written into the durable design record rather than held in memory, alongside a correction to a catalogue row claiming evidence that didn’t exist. Working notes, author’s own. Note what it does and doesn’t cover: gaps recorded, checks not recorded alongside them.
• The four-instance reading — the status column is quoted from the posts themselves, not remembered: augmented-reading.md:74 describes the routing as current practice; skills-are-a-codebase.md:105 states what the harness needs; noticing-is-cheap.md:76 introduces its rule with “three rules fall out, and I’m holding to them”; build-the-operator.md:37 ends “it’s the eval I haven’t run.”
• Basili & Selby, “Comparing the Effectiveness of Software Testing Strategies” (University of Maryland TR-1501, May 1985 — the tech-report precursor to IEEE TSE 13(12), 1987; cite the report, which is the version read) — read primary, and cited here for what it doesn’t support. The two flat classes are data and cosmetic faults, at α>.05; the significant effects run the other way (code reading found more interface faults, functional testing more control faults, both α<.01). The comparison varies technique, not the surface under view. It is not evidence that a vantage makes a class invisible.
• Conan Doyle, “The Adventure of Silver Blaze” (The Strand Magazine, December 1892; the story is also cited simply as “Silver Blaze”) — the epigraph, quoted with Holmes’s setup line intact, because without it “the curious incident” has no antecedent. I use it as the canonical example of an absent signal being the whole finding.

---

Related: Noticing Is Cheap, Judgment Is Scarce — the pipeline this post’s two receipts come from, and one of the four instances; Augmented Reading — the same move on a reading surface, and the one I actually performed; Your Skills Are a Codebase, Not a Junk Drawer — the same move on a trigger; Build the Operator, Keep the Operands as Data — the same move on an eval, prescribed and not run; You Can’t Grade Your Own Misses — why the coverage you excluded is hard to see from inside.