Train on What the Gate Throws Away

“We learn from failure, not from success.” — Bram Stoker

My code reviewer does far more work than it ships. Scouts cast a wide net — anything that might be a violation. A deep pass then refutes most of those leads, and a precision gate ships only the survivors: the few findings I’d actually put in front of a person. The gate is deliberate — a reviewer that cries wolf loses the right to be read, so it over-generates coverage inside and ships hard-filtered outside.

But look at what the gate throws away. Every refuted lead is labelled: “looked like a violation, wasn’t” is a hard negative. Every confirmed one is a positive. The reviewer computes its whole candidate surface — every lead its scouts raised, each adjudicated with a reason — and then bins almost all of it. That bin is the labelled dataset you’d want to train a better reviewer (bounded, as we’ll see, by what the scouts surface in the first place). And you already paid for it.

Bottom line up front

• Your reviewer’s reject pile is labelled training data. Every lead it refutes is a hard negative; every one it confirms, a positive-with-confidence. The gate discards this whole adjudicated candidate set after computing it.
• It’s the constructive flip of survivorship bias. Don’t train on merged code (the survivors); train on your reviewer’s rejects — the adjudicated leads the gate throws away.
• The labelling is nearly free. You spent the compute adjudicating when you ran the review; you’re just not keeping the verdicts. (Storing and training on them isn’t free — the labels are.)
• Capture the pre-gate lead→verdict pairs, not the shipped findings. The findings are the survivors; the training signal is the whole pile.
• Design-forward, honestly: I run the harvesting loop (drops → recorded exceptions → mechanised) and tighten the scout against the drops with a recall floor (measured, modest); I have not trained a model on the pile. Those uses are lived; the model-training is argued.

The gate computes recall, then spends it

Picture the pipeline as three stages. Scouts raise leads — wide, cheap, tuned for recall, because a lead never raised is a bug never caught. A deep pass verifies each lead and refutes the ones that don’t hold. A precision gate ships only what survives. Recall lives inside the pipeline; precision is enforced at the exit. That split is on purpose: over-generate coverage where it’s cheap, then gate hard so the human only ever sees findings worth their attention.

The gate is also where the recall you computed gets spent. Everything below the line — every refuted lead, with the reason it was refuted — is calculated and then dropped. You ran a classifier over your own codebase and threw away every label except the handful you shipped.

Refuted isn’t worthless — refuted is labelled

The part that’s easy to miss, because “refuted” sounds like “noise”: a refuted lead is one of the most valuable kinds of training example there is — a hard negative, something that looked enough like a violation to get raised, and wasn’t. Hard negatives are what you normally pay annotators to find; a naive dataset is all easy negatives and a few positives, and it teaches a model nothing about the boundary. Your reviewer manufactures hard negatives for free, every run, at exactly the decision boundary you care about — and then deletes them.

Training a verifier on labelled generations is a standard move (Cobbe et al. trained verifiers on model outputs to great effect). The analogy isn’t exact — their labels were externally checkable right/wrong answers, the reject pile’s are the deep pass’s own fallible verdicts — but the shape is the same: labelled generations, pre-built. Lead → verdict, with a reason, at the boundary. You are sitting on the training data and shipping it to the bin.

One caveat, and it’s load-bearing: those verdicts are the labels, and they’re the deep pass’s own — fallible. A real violation the pass wrongly refuted becomes a mislabelled negative, so training on the pile teaches this reviewer’s judgment, misses and all. The pile is only as good as the adjudicator that built it: it compounds your reviewer, it doesn’t transcend it. That’s a ceiling to respect, not a reason to keep binning the labels.

The flip side of survivorship

This is the constructive complement to a hazard I’ve written about before. Merged code — the final snapshot on the main branch — is survivorship-biased: it’s post-cleanup, so the violations that got caught are missing from it (they may still live in intermediate commits; it’s the merged snapshot that’s scrubbed) — train on that and you learn from the bombers that came back. The reject pile is the other direction entirely. It’s precisely the caught-and-adjudicated leads — the violations and the near-misses — that merged code no longer contains. So the reject pile fills the caught hole: the labelled recall surface the obvious data can’t give you.

Only the caught hole, though — and that bound matters. A violation no scout ever raises never enters the pile, so the pile is capped by your scouts’ recall: its own survivorship line, one level up. It’s the honest signal for everything your reviewer surfaced, not for what it never saw. Two loops, opposite signs: the merged snapshot is the biased data you should be wary of training on; the reject pile is the honest-but-bounded signal you’re throwing away. Adjacent, not the same.

The loop I actually run

The lived part, so I’m not selling you a plan as a practice: I run a harvesting loop now. Run the deep reviewer across the codebase; for every lead it drops, capture the reason; encode that reason as an exception on the rule that fired (and mechanise it where it can be made deterministic — the how of that migration is its own post). What the loop harvests is the point here: a growing, labelled set of adjudicated leads, captured instead of binned.

And the labels already do more than seed exceptions. A refuted lead is a negative, and negatives are exactly what you need to make the scout fire less without firing worse: I run the drops through a prompt optimiser that tightens each rule’s one-line scout summary so it stops raising the pattern that kept getting refuted — with a hard recall floor, so any tightening that would silence a known true catch is rejected outright. Across the two-dozen-odd rules I ran it over, a couple moved precision from about a third to a half with recall held; roughly half didn’t budge, and two were rejected outright for dipping below the recall floor. The eval sets are tiny — one curated positive plus eight-to-ten mined negatives per rule — so this is directional, not a proof, and the easy headroom looks mostly spent. Modest, but real and measured — the pile tightening the reviewer that produced it.

So the pile already pays off twice, both lived: exceptions encoded, and scout summaries tightened against the negatives. What I have not done is train a model on the accumulated pile — the obvious next step, and still argument, not result. Prompt-optimising a summary against the drops is a long way from moving the reviewer’s judgment into weights: the first is running, the second is designed. I’d rather say that plainly than imply a trained model exists.

What you keep

Stop throwing away the labels. Your reviewer’s real product isn’t the findings it ships — it’s the labelled recall surface it computes to decide what to ship. Send the survivors to the human; keep the whole adjudicated pile for the factory that makes the reviewer better. The gate is the right place to be strict with a person’s attention and the wrong place to be strict with your own data.

---

How sure I am: the pipeline (scout → deep → gate), the precision gate, and the harvesting loop are real and running, genericised here. Two uses of the pile are lived: exceptions encoded from the drops, and the scout summaries tightened against those drops with a recall floor — the tightening was measured but modest (a few rules improved, the broad pass mostly a null result). What I have not done is train a model on the pile: “the reject pile is model-training data” is a sound argument from how the pipeline works, not a demonstrated result. The generator-verifier framing is real prior art; the specific yield of this pile as training data is unmeasured. Treat the model-training claim as designed, not demonstrated.

Sources

• Cobbe, Kosaraju, Bavarian, Chen et al., Training Verifiers to Solve Math Word Problems (2021), arXiv:2110.14168 — train a verifier on labelled model generations (GSM8K). Cited by analogy: their labels were externally checkable, the reject pile’s are self-adjudicated — the shape of the move, not proof this reviewer improves.
• Your Merged-PR History Is Survivorship-Biased Training Data — the hazard this is the constructive flip of (the author’s own post).

---

Related: Your Merged-PR History Is Survivorship-Biased Training Data (the biased data to avoid; this is the honest data you discard); You Can’t Grade Your Own Misses (recall is what the gate spends); Rules Are the Alpha and A Big Rule Set Is an Attention Problem (the corpus + the exceptions the harvest feeds); Keep the Factory, Ship the Result (the pile feeds the factory, not the shipped output).