The Labour Was the Check

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman, Cargo Cult Science

I don’t write to think.

I’ve tried. People I respect swear by it — get the idea onto the page and the page tells you whether you had one. For me it’s a tax. Holding a single idea still long enough to word it properly costs me the next three, which were queued behind it and are gone by the time I look up. So the way I work now is to blurt: dump everything, unordered, half-finished, and have it gathered into something coherent.

It works. What comes back is better sequenced than anything I’d have ordered by hand, in a fraction of the time.

One thing is missing from it, and it took me a while to name: nothing pushes back.

That sounds like a complaint about tone. It isn’t. The pushback wasn’t a bonus that careful writers earned on top of the words. It was the only error-detection prose ever had — and unlike most of what AI has taken off our hands, when this one goes it leaves no mark in the output.

Bottom line up front

• The effort to explain is a measured error-detector, not a writerly virtue. Being made to explain something makes you more accurate about what you knew. Twelve studies, 2002. I’m citing that, not claiming it.
• What’s new is the price, not the trick. Buying an explanation you didn’t produce is ancient — ghostwriters, templates, copying a good one out of a book. What’s new is that it’s now free, instant, and pointed at your own half-formed material. The benefit those studies measured sits in the half that’s become optional.
• Code likely survived AI authorship because it ships with an adversary. Compiler, types, tests, the thing actually running. Partial, but real and uninterested in your framing. Prose has none. I think that’s the reason; treat it as a hypothesis, because nobody has tested it.
• Handing the writing to a machine raises your guard on foreign output and lowers it on your own. The discriminator is whether what comes back reads as your thinking. A synthesis of your own bullets is the one artifact you won’t be sceptical of.
• “Will I still need this skill in a year?” won’t catch this. That’s a retention question. This failure lands in today’s artifact with your skill fully intact.
• The fix has to be a real external check, not a resolution to be careful. Intending to justify something later has been tested. It made people more confident, not better calibrated.

Somebody already measured the effort

In 2002 Leonid Rozenblit and Frank Keil published twelve studies on what they named the illusion of explanatory depth. The shape is simple. Ask people how well they understand how a zip fastener works, or a helicopter, or a flush toilet. They rate themselves high. Then ask them to write the explanation out, step by step. Then ask them to rate themselves again.

The second rating is lower. Reliably.

People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion—an illusion of explanatory depth.

There’s an obvious cheap reading of that drop: people get sheepish when an experimenter puts them on the spot, and the lower number is modesty rather than knowledge. Rozenblit and Keil went after that reading directly and ruled it out.

These findings support our interpretation of the drops in self-ratings shown in Studies 1–4: the participants are becoming more accurate in assessing their knowledge, not merely less optimistic or more conservative when confronted by the experimenter.

More accurate. The effort to explain isn’t a memory aid and it isn’t a confidence tax. It’s a measurement. You find out what you actually had.

So the interesting half of this post isn’t mine. Producing the explanation is the error-detector has been sitting in print since 2002. Worth saying plainly, because before I went looking I’d written down the opposite prediction — that the literature owned “difficulty helps you remember” and not “difficulty catches your errors right now.” Half right. The retention results held. The detection claim I thought I was contributing was already measured, with the confound I’d have raised already tested and excluded.

What’s new is the price

In every one of those studies the explanation and the effort to produce it were the same event. Rate your understanding, write it out, rate it again — the writing was the manipulation. There was no version of the task where the explanation turned up without the work.

Now the obvious objection, and it’s a good one: buying an explanation you didn’t produce is ancient. Ghostwriters. A co-author who writes up your half. A template that shapes the argument for you. Copying a clean explanation out of a textbook. None of that is new and none of it needed AI. If unbundling the artifact from the effort were the whole story, we’d have had this problem for a century.

What’s new is the price, and the price is the argument.

Every one of those predecessors costs something that makes you notice you’re using it. A ghostwriter costs money and a briefing — and the briefing is itself an explanation you have to produce. A co-author asks questions. A template shapes an argument you still have to fill. Copying gets you someone else’s explanation of their idea, not a fluent version of your own unformed one. Dictation doesn’t unbundle anything at all; you still have to say the thing in order.

AI synthesis is the first version that’s free, instant, private, aimed at your own half-formed material, and available at the granularity of a paragraph rather than a chapter. Blurt the bullets, get the coherent version back. No money, no briefing, nobody to explain yourself to.

That’s a claim about friction, not about kind — and friction is what decides whether a check runs at all. A check you have to pay for gets run on the things worth paying for. A check that costs nothing to skip gets skipped by default. Those studies measured what the effort buys you. They never had to ask what happens once the effort is optional.

The claim is narrower than “AI makes you shallow.” Your skill is intact. Your knowledge is whatever it was. What’s gone is the moment where trying to say it properly would have told you that you couldn’t — and nothing in the finished paragraph is different for having skipped it. That’s what makes it dangerous rather than merely regrettable. A coherent draft of a wrong idea is a coherent draft.

Why code got away with it

Code has been written by machines for a few years now and the sky hasn’t fallen. The usual explanation — code is functional, prose is communicative, so code has a ground truth — is close but points at the wrong thing. What saves code is that it ships with an adversary: the compiler, the type checker, the tests, the thing actually running in front of a user. None of them care how confident you feel and none of them are working from your framing. Review is cheap on code because the code was already argued with before a human read it.

In degrees, not as a binary. give-away-the-writing draws the real boundary: tests and types check the properties you managed to externalise, and can’t cover the intent you never wrote down. The adversary is partial. It is still more than prose has, which is nothing.

And the reason isn’t just intuition. Rozenblit and Keil went looking for why explanations are uniquely prone to this and landed on something that reads almost as a description of source control:

By contrast, we often retrieve facts, tell narratives of events, or perform procedures; hence it is often easy to assess our average level of knowledge in these cases by inspections of past performance.

Past performance you can inspect. That’s exactly what code has and an argument doesn’t. Every function you wrote has a history of working or not working, and you can go look. A paragraph has no runs.

Treat this section as a hypothesis about why code escaped, not a result, because it has two limits and they’re both mine. Their “procedures” means tying shoes and making an omelette, not shipping software — the parallel is suggestive, not measured. And the things they asked about have ground truth you can go and check: a zip exists, you can take one apart and find out you were wrong. A technical argument often has no equivalent, which means the effort to explain may be carrying more weight in my case than in theirs, or less, and I don’t know which. The weak step is the jump from code has an adversary to prose’s only one was the labour.

The precondition on giving away the writing

There’s a wager in give-away-the-writing that I still believe, and this post is the condition it runs on rather than an objection to it. The wager:

And here’s the part that isn’t obvious — call it a wager. For a lot of engineers the guard is stronger in review than it ever was on their own writing. Your own code you trust; you cut corners, you let things slide. Code you didn’t write, you’re sceptical of in a way you never manage with your own. Hand the writing to the machine and you’ve moved the work into the mode where you actually pay attention. The catch: the guard only guards if the review is real.

That holds when what comes back reads as foreign. The scepticism is triggered by this isn’t mine.

Now feed the machine your own bullets and ask it to organise them. What comes back is recognisably your thinking, in better order. The trigger never fires. You get the review posture for a stranger’s draft and the credulity you reserve for your own — on the same artifact, which is the worst available combination.

So the discriminator isn’t machine-written versus hand-written. It’s whether the output reads as someone else’s. That’s a wager stacked on top of his wager, and I’m labelling it as one: it’s testable in principle — run the same person’s scepticism against foreign output and against a synthesis of their own notes — and nobody has run it, including me.

The gate that doesn’t catch this

the-productivity-mirage gives the best triage I know for what to let a tool take: do I want to still be able to do this without the tool a year from now? Keep the difficulty whose absence would cost you the skill. Automate the rest.

Answer that gate honestly about prose sequencing and the answer is no. I don’t want to get better at ordering paragraphs. It’s the part I’m worst at and least interested in, and handing it over is the right call by that test.

And I still ship a wrong argument this afternoon.

That’s the gap. The retention gate is per-career: it asks what you’ll still be able to do. This failure is per-artifact: it asks what’s checking the thing you’re about to publish. A difficulty can be correctly classified as not-worth-retaining and still have been the only thing checking the current output. That’s a third column on a shorter clock than the other two, and the residue test can’t reach it — residue is measured periodically, after the fact.

What it cost me, today

Earlier today I was working on another post that rested on a claim about the mechanism of a re-review procedure — a procedure in a system I built. I wrote the claim from my own recap of it. Not from the procedure’s text. It was wrong, and it had been wrong on disk before the draft existed.

Two things make that a receipt rather than just an error.

It was wrong in the flattering direction. My version was somebody forgot a stage — a tidy gap with an obvious fix. The source said something harder: three careful, ordered, non-skippable steps whose terminal condition was the absence of the old defect, so the procedure could run perfectly and still close without anyone reading the replacement. The comfortable reading survived because nothing contradicted it.

And nothing in the artifact showed the gap. The draft was coherent, argued, structured, and reviewed against our own publish bar, and it shipped a wrong mechanism. Cohesion carried no signal, which is this post’s claim about itself.

What fixed it was resistance imported on purpose: go read the primary source, and treat my own recap as a claim rather than a fact. We already have that rule written down — treat any summary as a claim, not a fact: a subagent’s report, a test digest, your own recap. The rule existed and I still needed an external prompt to apply it to myself.

Then it happened twice more the same day, on the record file for this post. I wrote confident claims about two of our own published posts from my recollection of them. Both wrong, both wrong in the direction that made my contribution look bigger. Reading them fixed it.

Grade this honestly. It’s one working loop, mine, self-observed, and not independent of the claim — I noticed it because I was already thinking about it. the-productivity-mirage already published a near-twin of the same incident: the author couldn’t reconstruct the reasoning behind one of his own decisions, and the first story he told himself was the wrong one. So this isn’t newly noticed, and presenting it as new would be the exact failure the post is about.

Now the part the receipt doesn’t do, because it would be easy to let it stand for more than it does. It establishes that a coherent artifact can carry a wrong claim with nothing on its surface to give it away, on someone who had already written the rule down. It does not establish that skipping the effort to explain is what let the claim through — what I actually did was not read a source, which is a related miss and a different one. Cohesion carrying no signal is the half I’ve got. The causal half needs a comparison nobody has run: take the same half-formed claim, write it out longhand once and have it synthesised once, and find out which version you catch. Until someone does that, the mechanism in this post is an inference from a measured effect, not a measured effect.

What to do, and one thing not to

Put the checks that can actually reject you first, and they are all outside your own head. The primary source instead of your recap of it. The procedure’s text instead of your memory of the procedure. A test, a type, a constraint, a person with no stake in the idea. What those have in common is that they can come back with no, and none of them care how the paragraph reads.

One narrowing to add, and it’s the finding that cuts against the version of this advice I’d have written myself. Rozenblit and Keil ran a condition where people were told in advance they’d have to produce the explanation. That did not make them start out humbler:

We might have expected the manipulation to result in the lower initial (T1) ratings of knowledge, but it did not. … Instead, the magnitude of the effect was reduced because T2 and T3 ratings were higher in this study compared to Studies 1–4.

Anticipating the test anchored them. They ended up more confident, not better calibrated. So keep in mind you’ll have to justify this later is not the fix — it’s been tried and it goes the wrong way. The check has to be something outside you that can actually say no.

The model-as-adversary version is second-rate, and worth naming as such rather than leading with. the-productivity-mirage has the mechanism and the ask: a fluency-maximiser validates whatever you hand it, so pressure-test this is a leading question — invert it, and demand a verdict rather than an opinion. Useful. Not a check. A model asked to object to your idea is still working from your framing, which is the limit models-narrow-the-overlap sets out. Call it triage: weaker than an adversary with no stake, better than nothing, and not a substitute for going and reading the thing.

The last thing to be straight about: the remedy costs exactly what blurting bought. Imported resistance is slower. That’s the real trade, and I’m not going to pretend the fix is free.

If you’re one of the people for whom wording it properly is the thinking, none of this is aimed at you — you never unbundled anything, and you’re getting the check for free. My argument is about what goes missing when you change modes, not that one mode wins.

---

How sure I am: High on the foundation — the illusion of explanatory depth is twelve studies with the modesty confound tested and excluded, and I’m citing it rather than proposing it. Medium on the price claim: that buying an explanation without producing it is old, and that AI made it cheap enough to become the default, is reasoning rather than measurement. Low-to-medium on the code-adversary section, which I’d call a hypothesis — it has Rozenblit and Keil’s own mechanism behind it and a parallel between everyday procedures and software that nobody has tested, in either direction. Low on generality: one self-observed incident, non-independent, no rate, and it establishes that cohesion carries no signal rather than that the unbundling is what let the error through. The comparison that would settle that is named in the post and unrun.

Sources

• Leonid Rozenblit and Frank Keil, “The misunderstood limits of folk science: an illusion of explanatory depth,” Cognitive Science 26(5), 2002, 521–562.
• Robert A. Bjork and Elizabeth L. Bjork, “Desirable difficulties in theory and practice,” Journal of Applied Research in Memory and Cognition 9(4), 2020, 475–479.
• Norman J. Slamecka and Peter Graf, “The generation effect: delineation of a phenomenon,” Journal of Experimental Psychology: Human Learning and Memory 4(6), 1978, 592–604.

---

Related: Give Away the Writing, Keep the Reading — the wager this post puts a precondition on. The Productivity Mirage — the retention gate, the two ledgers, and the adversarial framing. A Pool of Models Narrows the Overlap — It Doesn’t Erase It — why the adversary you can summon is a weak one. A Description Can’t Replace the Code — the same gap on the other side of the handoff.