The Basis Is Generated Too

“Nothing is so difficult as not deceiving oneself.” — Ludwig Wittgenstein, Culture and Value

I wanted the reviewer to tell me how it knew. So I put a field in the output schema — basis — with values like traced-flow, read-definition, pattern-match. Every finding would arrive carrying its own provenance, and I could sort by it: trust the traced ones, discount the pattern matches.

It worked for about a day. Then I noticed a stage that only ever sees the diff — no file reads, no definitions, no call graph — emitting basis: traced-flow. It had not traced a flow. It could not have traced a flow. The field wasn’t reporting the run. It was a word that fit the sentence.

Bottom line up front

• A provenance field looks like telemetry and isn’t. basis: traced-flow comes out of the same generation step as the finding it describes. It’s a claim about the run, written by the run.
• You can’t resolve a generated basis the way you resolve a citation. A rule citation has a referent — paste the sentence or the rule doesn’t exist. A generated basis has nothing to paste. You can give it one by logging the run instead of asking the model about it, and you should; that’s telemetry, not a basis field, and it isn’t what most schemas ship with.
• The reported reading is wider than the reading that happened. Hand-adjudicating thirty high-severity findings against source turned up four false positives, and all four are the machine on the correct line, graded without one or two lines of neighbourhood. Each one’s basis implied it had read the function.
• Those four also killed my own story about where machines go blind. I’d been saying the machine’s edge is a wider window. The window was wide and the context still didn’t land. I have a replacement — the unit it grades is narrower than the unit it reads — and four cases, which makes it a suspicion, not a finding.
• Check the basis against the wiring, not against the output. A diff-only stage structurally cannot trace a flow, so basis: traced-flow from that stage is refuted before you read a word of the finding. Free, and it runs once per stage instead of once per finding.
• The honest limit: this needs you to know the wiring. Often you don’t. Then the field is decoration, and you should read it as decoration.

A citation resolves. A basis doesn’t.

I’ve written before about an agent inventing the bar it was measuring against — citing a rule that was in no file, and the fix being mechanical: resolve the citation to its text before acting on it. Paste the sentence. If you can’t paste it, the rule doesn’t exist.

That check works for one reason. The thing being cited exists outside the model. There’s a file, a line, a sentence; the model’s claim and the sentence are two different objects, and you can hold them up against each other.

A generated basis has no outside. There is no artifact called “the flow I traced.” The reading either happened during the run or it didn’t, and the field describing it is not instrumentation — it’s the next token after the finding. The same step that decided the code was broken also decided how it knew, which is reason enough to expect the two to fail together. In the cases I’ve hand-ruled, they did, and the basis still read as confident.

So the sibling fix doesn’t transfer. You can’t resolve a basis to its text, because there is no text. That gap is the whole problem: the field most likely to make you trust a finding is the field with no referent.

Here’s where a good engineer stops me, and they’re right. Nothing forces the field to be generated. Log the tool calls. Record which files were opened, which definitions resolved, which edges of the call graph got walked, and derive basis from those events. Now it has a referent and you resolve it exactly the way you resolve a citation — against the log.

Take that, because it’s the better fix. Notice what it does, though. What you’ve built at that point isn’t a basis field; it’s telemetry, and the model has been cut out of describing its own run. That’s the lesson in one move: don’t ask the model how it knew — record it. The field only looks like provenance for as long as the model is the one filling it in.

Which leaves the case this post is about. Someone else’s tool. A stage you’re not re-instrumenting this quarter. A schema already in production with the field sitting in it. For those, the model is still filling it in, and there’s still a check. It just doesn’t look anything like reading the finding.

The reading it claims is wider than the reading it did

Here’s the version that cost me something to learn.

I took thirty high-severity findings the reviewer raised that no human reviewer had commented on — all from public open-source pull requests — and ruled every one against the source, read at the exact commit. Sixteen were genuine at the severity claimed. Seven were genuine but over-graded. Three were duplicates of each other. Four were false positives.

The four false positives are one defect wearing four coats. In every case the machine landed on the correct line and graded it without reading one or two lines either side.

Two shapes. In the first, the author had already written the dismissal into the code. A floating promise chain flagged as a silent failure — with the comment // silently fails for now on the very next line. An empty else branch flagged as incomplete coverage — where the branch’s own comment states why it’s empty and points at the validation downstream that catches it. Both times, a human reads the comment and moves on for free. It’s not even a judgment call; it’s a line of English sitting right there.

In the second shape, something worse sat within three lines and went unread. A soft complaint about a hook’s API shape, raised three lines above a genuine hook-ordering violation — an early return before a useEffect, which is the actual bug in that function. A simplification nit filed on a line that begins it.only( — which silently disables every other test in the file.

Now the part that matters for the basis field. All four arrived with a basis implying the surrounding code had been read. That’s not a lie the model told. It’s a claim that never had a mechanism behind it, in a field designed to look like it did.

The story I had was wrong

Those four cost me something past the adjudication time.

I’d been telling a story about where the two readers go blind. The human reads what’s in front of them and makes assumptions about what’s off screen; the machine actually pulls the other file up. Wider window, blind spots somewhere else.

The window half survived. The conclusion didn’t. In all four cases the context was already on screen and never entered the verdict. Whatever went wrong, it couldn’t see far enough isn’t it.

I have a replacement story and I’m going to undersell it, because I have four cases and no instrumentation. The suspicion is that the machine grades a smaller unit than it reads — roughly a line, with the diff hunk as decoration — so context can sit in the window and stay out of the judgment. That’s a claim about attention and granularity. It needs a designed comparison, not four retrospective examples, and it’s a different post than this one. Take it as the reason I stopped believing the window story, not as the thing that replaces it.

It does earn its place here on one point. read-definition is often literally true. The definition was in the context. It just wasn’t in the verdict — and a field written after the fact has no way to tell those two apart.

The hedge that ships as a verdict

One more shape, same root. A finding that read, near enough: this library might return a string in a different format.

That’s not a finding. It’s a hedge. The uncertainty is right there in the sentence — the model is telling you it doesn’t know. But the output schema has nowhere to put “I don’t know.” There’s a severity, there’s maybe a confidence score, and the confidence score is generated by the same step as the hedge. So the hedge ships in a high-severity slot, at the same confidence as a bug someone actually traced, and a reviewer has to spend the time to find out which one it is.

The false positive is the receipt. A real trace can honestly end in I don’t know — tracing something isn’t the same as concluding about it, and a basis that meant anything would still permit an uncertain finding. What it shouldn’t permit is that finding arriving at high severity anyway. The basis didn’t stop it. The confidence score didn’t stop it. Both were written by the step that wrote the hedge.

Check the basis against the wiring

You can’t verify a basis from the finding — that’s circular. You can falsify it from the architecture, and that check is nearly free.

Write down, per stage, what that stage can actually see: which tools it has, what context it’s handed, whether it can read a file at all. Then take the list of basis values the schema permits and delete, for that stage, every value its wiring cannot produce. A diff-only scout gets pattern-match and nothing else. If the model can’t say traced-flow there, it can’t claim it.

Three things make this worth doing:

It runs once per stage, not once per finding. You pay for it at build time. Every finding afterwards inherits the constraint.

It’s falsification, not verification. You’re not checking whether the reading happened. You’re checking whether it could have. That’s a question about your own wiring, which you can answer with certainty, unlike the question the field appears to answer.

It catches the whole class at once. Before I looked at a single finding, I knew every traced-flow from that stage was wrong. No adjudication required.

None of which is clever. It’s capability-aware schema validation — the same discipline as not letting a read-only path hand back a write receipt, and if I’d had the field in front of me with that framing I’d have caught it in an afternoon. The part that’s easy to miss isn’t the technique. It’s that a field called basis reads as a measurement when it’s a generation, so it never occurs to anyone to validate it at all.

Then one rule for the values that survive. A basis may lower severity. It may never raise it. pattern-match on a finding is useful — the model has no reason to invent a weaker basis than it has, so a modest claim is roughly trustworthy. An ambitious one isn’t evidence of anything. Grading confidence up on the strength of a self-reported basis is exactly the move the field invites and the one it can’t support.

What this doesn’t fix

It needs you to know the wiring. For a tool you built, that’s a morning’s work. For a closed product, someone else’s agent, or a model behind an API, you don’t have the stage map — and without it, a basis field is decoration. Read it as decoration. Sort on it if you like; don’t gate on it.

And constraining the enum does nothing about the neighbourhood defect. The machine will still grade a line without reading the comment under it. It’ll just stop telling you it read the function. That’s a smaller lie, not a better reviewer — and fixing the actual reviewer is a different post.

The nearest published result isn’t mine, and it doesn’t quite reach this. Turpin and colleagues put it better than I would: chain-of-thought explanations “can systematically misrepresent the true reason for a model’s prediction.” Reorder the multiple-choice options so the answer is always (A), and models rationalise the biased answer while — their words — systematically failing to mention the bias. Accuracy falls by as much as 36% across thirteen BIG-Bench Hard tasks. That’s a result about reasoning-trace text, under an intervention the authors designed, on their own task suites, on two 2023-era models. It isn’t a result about schema fields, and I’m the one extending it — that a basis enum fails the same way is my inference from their mechanism, not their finding.

What I’d add is narrow and operational. In a tool, stated reasoning gets a field name, a schema, and a column in a table, and starts looking like telemetry. It isn’t. But because it’s a tool, you own the wiring, and the wiring is checkable for free.

---

How sure I am: The diff-only stage emitting basis: traced-flow is a thing I watched happen in my own build, repeatedly — that part I’m certain of. The thirty-finding adjudication is my own hand-ruling against source at the recorded commit, one person, one pass, no second adjudicator; the 16/7/4/3 split is a judgment call I’d defend but not a measurement, and a second reader would move at least one of those. That all four false positives share the same root is the strongest empirical claim here and rests on four cases — a clean pattern at small n, not a rate. The concession that matters: a basis can be made resolvable, by logging the run and deriving the field from the events rather than asking the model. That’s the better fix, it’s in the post, and it means the no-referent problem is a property of generated provenance, not of provenance — everything after that section is about the pipelines where nobody did it. Demoted on this pass: the unit-of-judgment reframe used to be a section making a claim; it’s now a suspicion I flag and route onward, because four retrospective cases can’t carry an account of model attention and I haven’t instrumented one. I can show the misses, not the cause. The hedge-as-verdict example is one finding, remembered by shape rather than exact words. The fix — constrain the basis enum per stage — is ordinary capability-aware validation, I’m confident in the reasoning, I’ve only partly implemented it, and I haven’t measured what it buys. On the prior art: Turpin et al. is about chain-of-thought under deliberate input bias, on their own task suites, on GPT-3.5 and Claude 1.0. Applying it to a structured schema field is my extension, stated as mine in-body; their paper does not test it. I’ve read the abstract at source and quote it directly — I haven’t read the full paper or reproduced their runs, and nothing here rests on a number I didn’t take from their own words.

Sources

• Turpin, Michael, Perez & Bowman, Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (arXiv:2305.04388; NeurIPS 2023) — chain-of-thought explanations “can systematically misrepresent the true reason for a model’s prediction.” Adding biasing features to the input — reordering the multiple-choice options in a few-shot prompt so the answer is always “(A)” — produces explanations that rationalise the biased answer and that models “systematically fail to mention”, dropping accuracy “by as much as 36% on a suite of 13 tasks from BIG-Bench Hard”, tested with GPT-3.5 and Claude 1.0. A separate social-bias task shows explanations justifying stereotype-aligned answers without mentioning the bias. The authors’ own summary: CoT explanations “can be plausible yet misleading.” Scope, and I’d rather say it than have you catch it: that is chain-of-thought text, under interventions the authors designed, on their own task suites, on two 2023-era models. It is not a result about output-schema fields, and the step from their mechanism to a basis enum is mine. Quoted from the paper’s abstract, read at arXiv; I have not read the full paper or reproduced their runs.
• My own adjudication of thirty high-severity findings from a reviewer I built, ruled against source read at each pull request’s head commit: 16 genuine at severity, 7 genuine but over-graded, 4 false positives, 3 duplicates.
• Jin & Chen, Are LLMs reliable code reviewers? Systematic overcorrection in requirement conformance judgement (Automated Software Engineering, 2026; doi:10.1007/s10515-026-00638-5) — on standard benchmarks with unified prompts, “LLMs frequently misclassify correct code implementation as non-compliant or defective”, and “more detailed prompt design, particularly with those requiring explanations and proposed corrections, leads to higher misjudgment rates.” Why it’s here and what it does and doesn’t cover: my four false positives are one person’s hand-ruling, so it’s fair to ask whether they’re a fluke — this says the phenomenon is measured and systematic, which is the part I’d otherwise be asserting from n=4. It does not back the mechanism. They measure code against a natural-language requirement; my four cases are about a fact in the neighbouring code never reaching the verdict, and that account is still mine and still untested. Their second finding cuts against the obvious fix, which is worth knowing before anyone reaches for a richer prompt. Quoted from the abstract, read at the publisher; I have not read the full paper.

---

Related: The Agent Invents the Bar (the citation version of this failure — and the fix that works there because a citation has a referent); Confidently Wrong Is the Failure Mode (asks the model to mark the seam between seen and guessed; this post is why the mark is the same generated token as the claim).