Addressed Is Not Reviewed

“Nemo iudex in causa sua” — no one is judge in their own cause. — Principle of natural justice

I built a re-review mode into the reviewer: run it again after the author has responded, and report whether the findings were addressed. It is the obvious thing to build. It is roughly what you’d ask a colleague to do.

Then I read the procedure it was actually running. Per acknowledged finding, three ordered steps, none of them skippable, each one careful: find the original violation pattern in the new diff and confirm whether the code was removed or rewritten; name the invariant the finding was really about rather than the code shape it happened to take; trace the affected value from producer to consumer until you can confirm the old failure mode is no longer possible.

Every step aimed at the bug that was already dead.

And the code it wasn’t looking at had been written to satisfy round 1’s own instructions — which had not merely named the defect, but specified the shape of the fix.

The pass came back clean. Every finding addressed. That report was true, and I had no idea how little it meant. A human read the same fixed code by hand afterward and found four defects the pass hadn’t raised.

Bottom line up front

• “Addressed” and “reviewed” are different words and the pipeline conflates them. A second pass can tell you the concern you raised is gone. It cannot tell you the code that removed it is any good, and it reports both in the same word.
• The hole was a stopping condition, not a missing stage. The procedure ran three careful, ordered, non-skippable steps per finding — and every one of them was satisfied the moment the old defect was provably gone. Run it perfectly and you still close without reading the replacement. Stopping conditions are the part of a spec nobody reviews, because they read like definitions instead of decisions.
• It also had no rule saying a prior suggestion of its own carries no authority. So the second pass arrived holding its own prescribed fix plus the author’s agreement to it, and that was the strongest signal available about the new lines. It was a judgment made before the code existed, about the defect in the code it replaced.
• One recorded run cost four defects. The author applied the prescribed fixes; the second pass reported everything addressed, accurately; a human then read the fixed code by hand and found four things the pass hadn’t raised. Three of the four were in code the reviewer had itself prescribed. Ordinary correctness bugs, the class a first pass catches all day.
• Not “the verdict is worthless” — I tried that framing and it’s wrong. A scoped re-review does real work: it can reject a fix that fails tests, breaks a caller contract, or mishandles an error while superficially following the suggestion. What it can’t do is find what the fix broke somewhere it was never told to look.
• The author is the laundering step, and their acceptance is a thinner check than it looks. Your prescription leaves as prose and comes back as code with someone else’s name on the commit, so it reads as independently authored. The author can reject it, test it, argue — a real check — but it thins the more the comment designs rather than points, and nothing downstream can tell that the code and the grader share a source.
• The remedies are dull. The trigger isn’t. End a step somewhere other than the old defect; write down that your own prior suggestion isn’t a warrant. Both obvious once said. The part worth the post: a prescriptive comment is the signal that you need them, and almost nobody reads one as anything but helpful.

The spec was satisfiable without reading the code

The procedure was not thin. That’s the thing I had wrong when I first wrote this up, and correcting it made the finding sharper rather than smaller.

Look again at where those three steps stop. Step one hunts the original violation pattern. Step two abstracts the finding to its invariant, so that a fix in a completely different shape still counts as a fix. Step three walks the affected value from producer to consumer through the new diff. That is real investigation, specified in order, with skipping disallowed. And all three steps are satisfied the moment the old defect is provably gone, because the terminal condition is the old failure mode is no longer possible — a fact about code that isn’t there anymore.

So a pass can run every step correctly, completely, in order and in good faith, and close the finding without the replacement having been read as code.

That is not a missing stage. I’d have spotted a missing stage. It’s a procedure that describes genuine work, gets executed properly, and still doesn’t require the one thing anyone would assume it covered. The gap isn’t in the rigor and it isn’t in the execution. It’s in the stopping condition — the part of a spec nobody reviews, because a stopping condition reads like a definition instead of a decision.

There is a published sibling on the general shape: noticing-is-cheap, on a pipeline that only verifies never discovering, and on the cheap floor of a first pass having to read the raw diff for what has no detector yet. I’m citing it rather than rebuilding it, and this isn’t it. That post is about a stage that was never built. This is a stage that was built with care and stopped one question early. The remedy there — give the floor a discovery pass — leaves this exactly where it was, because the procedure here already investigates. It stops investigating once the corpse is confirmed cold.

And this is the last stage. It’s the one you’d point to if someone asked how you know the fix was safe.

The grader wrote the spec

The other half I hadn’t seen coming.

Round 1 didn’t just say this value can be null. It said what to do about it, in enough detail that the author could apply it without deciding anything. That’s what a good review comment is supposed to look like. Actionable, no homework.

And nothing anywhere in the procedure said that a prior suggestion of its own carries no authority. Written out, that sounds like a strange thing to need on paper. It is exactly the thing that needs to be on paper. The second pass shows up holding its own prescribed fix and the author’s agreement to it, and with no rule to the contrary, the strongest available signal about the new lines is that we already decided they were correct. We decided that before the code existed, judging a defect in the code it replaced.

There’s a published sibling here too — cant-self-measure-recall says you need ground truth that isn’t your own output. This is a turn of that screw: in that post my own output was the yardstick. Here it’s the object. The reviewer didn’t supply a standard to measure the code against; it supplied the code, in prose, and then measured it.

The route it took is the thing I’d want a reader to sit with. The prescription left as a review comment, passed through a person, and arrived as a commit. Somewhere in that trip it stopped looking like the reviewer’s output and started looking like the author’s work. The author is the laundering step — not deceiving anyone, just typing — and after they’ve typed it, nothing downstream can tell that the code and the grader have the same source.

What it missed

For a while the honest position here was that I had an argument and no casualty. The mechanism said a miss was possible. I couldn’t show you one.

Then I went back through the recorded runs.

One re-review, one change, one author. Round 1 raised a blocker and a set of smaller findings. The author fixed the blocker and four of the others, mostly by applying what the comments had told them to apply. The second pass ran and reported everything addressed, and that report was accurate.

A human then read the fixed code by hand and found four defects. The pass had raised none of them. Three of the four were sitting in code written to satisfy the reviewer’s own comments.

And none of the four were exotic. They were the ordinary correctness class the first pass catches all day — the kind of thing a reviewer with a rule set and a diff is genuinely good at. The second pass didn’t miss them because they were hard. It missed them because it wasn’t looking, and it was least likely to look exactly where it had done the most prescribing.

That’s one run, and one run tells you the miss is real, not that it’s the rate. But it is the difference between this could happen and this happened, and it concentrated where the mechanism said it would.

What the pass can and can’t establish

Here’s the part where I have to be careful, because the strong version of this is wrong.

A re-review checking a prescribed fix is not comparing the code only against the suggestion. The tests are still there. The types are still there. The caller contracts, the original issue, the surrounding code — all independent objects, all things a scoped pass can legitimately use to reject an implementation that followed my suggestion badly. I wanted to say the verdict was a tautology. It isn’t. That framing made a limited, useful check sound fraudulent, and it didn’t survive review.

What’s actually true is narrower and still worth knowing:

It can establish conformance. Did the concern I raised go away? Yes or no, and the answer is real.

It cannot establish that the change was good. The judgment of whether this is the right shape was made in round 1, by the same step that’s now grading. That question isn’t being re-asked, it’s being confirmed, and confirmation by the party that decided it adds nothing.

It cannot see collateral. Anything the fix broke that the stopping condition doesn’t reach is invisible, and the stopping condition reaches exactly as far as the old defect.

Three different things, and the report says “addressed” for all of them. (The same move in a different place — you can’t verify a self-generated field from the output that generated it, but you can falsify it from the architecture — is self-reported-basis; it’s the cheap check in both cases, and I’m citing it rather than deriving it twice.)

What “addressed” hides

So here’s the reframe, in the form that survives.

A prescribed fix is code that reached the diff without an independent read. Not “reviewed by nobody” — the author read it, chose it, and could have refused it, and that choice is a real review action. But it’s a thinner one than review, for two reasons worth naming.

The first is that the author is implementing the suggestion of the person who decides whether their change lands. It takes some nerve to reply your fix is wrong to whoever’s holding the approve button. I don’t know how often people do it. I’d guess less than they should, and I’m guessing — but you don’t need a rate for the point to land, because the check being thin in the cases where it matters most is already enough to stop calling it review.

The second is that the thinness scales with the comment. This can be null here leaves the author to solve it, so their judgment is genuinely in the loop and there’s something independent to grade. Wrap it in Option and return early hands over the answer. The more the comment designs, the less the author decides, and the less their acceptance is worth as a check.

Then the re-review says addressed, and that word covers all of it — the concern is gone, the author agreed, the prescriber confirmed its own shape came back. Three weak signals stacked into one confident-sounding noun.

The objection I didn’t have an answer for

Not the ones I expected. This one:

You’re confusing a contaminated oracle with no oracle. My first pass’s suggestion isn’t the sole specification — the tests, the types, the caller contracts and the original issue are all still there, and a scoped re-review can use any of them to reject a fix that followed my advice badly. It doesn’t have to discover an unrelated bug to do real work. If you mean a verification-only pass can’t catch collateral damage, say that, and stop implying the whole check is theatre.

That’s right, and it’s why the section above is shorter and duller than the one it replaced. The check is contaminated on one axis — is this the right shape — and sound on the others. Saying so costs the post its most quotable line and leaves it more useful.

Two objections I did expect, briefly:

“This is just ‘add a step.’ Ordinary engineering.” The remedy is dull, yes. But look at what the objection assumes — that the step was missing and somebody forgot it. It wasn’t missing. It was three specified steps nobody would read as inadequate, and I wrote them, and I reviewed them, and the hole was in where they ended. The contribution isn’t the fourth step. It’s the two things that point at it: a stopping condition is a design decision wearing the costume of a definition, and a prescriptive comment is the signal to go read yours — because a review comment reads as part of the machinery that catches bad code, not as a new source of it.

“Human review has worked this way forever.” Mostly, and I have a hypothesis about why rather than a finding. Humans tend to point rather than design — less patience for specifying a whole fix — which leaves the author something to solve. And a human looking at your fix probably re-reads the code around it; eyes wander. If that’s right, it’s a discovery stage nobody planned, and the machine’s verification pass is precisely what strips it out. I haven’t measured either half, and I’d bet on this part of the post least.

What to do about it

1. Read your re-review’s stopping condition, not its stage list or its output. Ask what fact ends each step. If every answer is a fact about the code that used to be there, the pass is complete and blind at the same time, and no amount of quality in those steps changes it. What I added was a fourth step that ends somewhere else: the replacement is new code on changed lines and gets the review any changed line gets — the full rule set, not a compliance check.
2. Write down that your own prior suggestion isn’t a warrant. It feels too obvious to state, which is why it goes unstated, and unstated it defaults the other way. The sentence that closed it for me: we proposed it before it existed, judging the old code’s defect, not the new code. The author implementing it faithfully establishes that they did what you asked, which is a different question from whether the result is right. Make the pass ask the second one out loud.
3. Don’t let the step that specified the shape be the only one that grades it. A different pass, a different model, or a human. The other axes — tests, contracts, types — are fine; the was this the right shape question is the contaminated one, and it needs somewhere else to go.
4. Treat a prescriptive comment as the trigger. When you write one, you’ve added code to the diff and removed a decision from the author. That’s the moment to point something independent at the result.

The last one costs nothing and survives outside any pipeline. It’s just noticing that the suggestion you typed is going to become a commit.

---

How sure I am: The architectural half I’d stand behind, and you can falsify it by reading a procedure instead of trusting me: if every step of your re-review terminates on a fact about the code that got replaced, the pass can run to completion without reading the replacement. That’s not a frequency claim and a bigger sample wouldn’t make it truer. I had the mechanism wrong the first time I wrote this up, which is worth saying because the error flattered me. I described a verification-only pass with no discovery stage — a thing somebody forgot to build. It wasn’t that. It was three careful mandatory steps that stopped one question early, and I found out by going back to what the procedure actually said rather than what I remembered it saying. What the argument does not buy, and I over-claimed this in the first draft: it doesn’t make the verdict worthless or tautological — a scoped pass still checks the fix against tests, types and contracts, and can reject it on any of them. The contamination is on one axis only, was this the right shape. The receipt is one run: four defects found by hand in code the pass had just cleared, three of them in code the reviewer had prescribed. That establishes the miss is real and that it clustered where the mechanism says it should. It does not establish a rate, and three-of-four is four data points, not a signal I’d defend. The gap I’d attack first if I were you: one run, one build, so how often is wide open. What I’d most want is the same comparison on a re-review where round 1 only pointed at defects instead of designing the fixes — that’s the experiment that separates the prescription effect from the stopping-condition effect, and I haven’t run it. The social claim — that authors under-challenge a reviewer’s fix — is a guess I’ve labelled as a guess, and the human-review explanation is a hypothesis, not a finding.

Sources: No external sources. The receipt is my own build, described and not named.

---

Related: noticing-is-cheap — verification without discovery, and what the two halves cost. cant-self-measure-recall — why your own output can’t be the ground truth. agent-invents-the-bar — the other referent failure, where nothing was there at all.