Something Has to Be Able to Say No

“Nullius in verba” — take nobody’s word for it. Motto of the Royal Society, adopted 1662, after Horace.

My skill-evaluation harness told me a whole set of skills was useless. I believed it, and the reason I believed it is the thing worth examining: it had come back negative. It could have told me the skills helped. It didn’t. That felt like evidence.

It was wrong. The harness injected every skill into every case, including cases the skill should never touch, so it graded them on noise. Gate injection on the actual trigger and the same skills came out clearly helping.

Sit with the shape of that. The instrument was falsifiable in the only sense people usually mean — it was able to return a verdict I didn’t want, and it did. That property is why I trusted it. And that property had nothing whatsoever to do with whether it was measuring the right thing.

So there are two questions, and I had been asking one.

Bottom line up front

• Vibe coding’s tell was never distance from the code. Plenty of legitimate work happens at a distance. The tell is that nothing in the loop is permitted to come back negative. Read Karpathy’s line as a claim about distance and you draw the wrong lesson; read it as a claim about accountability and it’s exactly right.
• So distance is fine. Unfalsifiable distance isn’t. If you can’t point at the thing in your loop that is allowed to tell you no, you don’t have a vantage. You have a mood.
• But a no is not a check. Necessary, not sufficient. My harness said no and was wrong, and the no is why I believed it. Negative-capability buys credibility. It does not buy validity, and it is very good at feeling like it does.
• Ask both questions. Can this come back negative? — and is it negative about the thing I actually care about? The first is a property of the instrument. The second is a property of the fit between the instrument and the question, and nothing about the first will tell you about the second.
• The sharpest instance: a check that can only fail on output is silent about whether it was pointed at the right case. Which means the case you most need is the one where the thing under test shouldn’t do anything — and a pass/fail gate on output can’t contain it.

The foil, stated precisely

Karpathy’s original framing is worth quoting rather than paraphrasing, because the paraphrases have drifted: fully give in to the vibes, embrace exponentials, and forget that the code even exists.

The lesson everyone took is about distance — that reading the code is the discipline and not reading it is the sin. That reading doesn’t survive contact with how this work actually goes. Steering a factory instead of reading everything it produces is not a moral failing; it’s the only way to operate at any scale, and it has always been how people run teams. If distance were the sin, every engineering manager would be vibe coding.

Read the same line as a claim about accountability and it lands. “Forget that the code even exists” isn’t describing a reading habit. It’s describing a loop with no term in it that can come back and say no, that didn’t work. The vibes can only ever agree with you. That’s the actual failure, and distance is a red herring that happens to correlate with it.

Which is a testable difference rather than a vibe about vibes. Recall measured against planted bugs can come back negative — you know the bugs are there, so a miss is a miss. Precision measured against leads a deeper pass refuted can come back negative. A held-out corpus with a frozen cutoff date can come back negative. Every one of those works fine while you never read a line of the output.

A sense that things are going well cannot come back negative. Neither can a metric you’re allowed to redefine after seeing the result — which is the same failure in a lab coat, and the reason you never tune and score on the same window.

A no is not a check

Here’s the part I’d have skipped a month ago, and it’s the part that makes the rest safe to use.

Having worked out that falsifiability is the guard, the tidy conclusion is: audit your loop, find the thing that can say no, ship. My own harness is the counterexample, and it’s mine, which is the only reason I take it seriously enough to put here. It could say no. It said no. It was measuring the wrong configuration entirely.

And the negative verdict was actively load-bearing in my believing it. A tool that only ever confirms you is obviously suspect; everyone knows to distrust that. A tool that contradicts you feels like it has skin in the game. That intuition is doing unearned work. Being able to disagree with you and being pointed at the right question are independent properties. An instrument can have the first in full and none of the second, and when it does, the disagreement is the thing that stops you looking.

That failure has a specific and common shape, and I’ve written up the narrow version of it before: gating an eval so a skill only runs where it belongs fixes how you grade its output, and says nothing about when the skill fires. A pass/fail gate on output alone can’t tell you a skill is waking up where a sibling should own the task — only a shouldn’t-fire case can. That was about skill triggers. The general form is the point of this post: an instrument’s ability to fail tells you nothing about the range of things it could fail about, and the questions outside that range are invisible in exactly the way a passing test is invisible.

There’s a cousin worth keeping distinct. Some unfalsifiability isn’t your fault — from inside your own review loop, whether a defect exists that every pass missed is unanswerable, which is why “good recall” is unearnable no matter how honest you are. That’s a property of the question. What this post is about is a property of a loop you built, which means it’s your fault and it’s your fix.

Two questions, and where the second one bites

So the guard is two-part, and only the first part is the famous one.

Can this come back negative? Cheap to check, and most loops fail it. Name the term. If the answer is “well, I’d notice if it got worse,” you have a mood.

Is it negative about the thing I care about? This is the one that needs work, and here is the move that actually finds things: write down what the instrument would have to be pointed at for its verdict to mean what you want it to mean, then go and confirm it’s pointed there. For the harness, that sentence was “the skill fires the way it fires in a real run.” One look at the injection logic would have killed it. I never looked, because it had already told me no.

The concrete version, and the thing to steal: your check needs cases where the right answer is to do nothing. Should-fire and shouldn’t-fire. A suite made only of cases where something is supposed to happen can be fully green while the thing under test is firing constantly on work it should never touch. Absence of action is a verdict, and if your instrument can’t express it, there’s a whole class of wrongness it will never be negative about.

And one structural note: whatever says no has to sit outside the thing being judged. A model asked to check its own reasoning without external feedback doesn’t reliably improve it (Huang et al., 2023) — a loop where the generator is also the sole judge has the form of a check and none of the independence. Self-critique is a useful move. It is not a no.

The point

Distance from the code isn’t the problem. A loop that can only agree with you is the problem, and distance just makes it easier to build one without noticing.

So name the term in your loop that is permitted to tell you no. Then do the harder thing, which is to check what it’s pointed at — because a negative verdict from an instrument aimed at the wrong question is worse than no instrument. No instrument leaves you uncertain. That one leaves you confident.

---

How sure I am:

• From experience, and it’s the spine of the post. The harness that lied is my own log, genericised. It’s one instance. One instance is enough to show that negative-capability and validity come apart, because that’s an existence claim — it can’t tell you how often they come apart, which is the number I’d actually want.
• Prior art, scoped narrowly. Huang et al. is about intrinsic self-correction of reasoning without external feedback. I’m using it for one thing only: the judge has to sit outside the generator. It does not say models never improve, and it isn’t evidence for anything else here.
• Mine, and argued rather than measured. The whole distinction — falsifiability rather than distance as vibe coding’s tell — and the necessary-not-sufficient sharpening on top of it. The second half generalises something I’d previously only claimed about skill triggers, and a generalisation is exactly where an argument like this would break without my noticing.
• The weakest joint. “Loops with nothing negative in them produce worse work” is the premise the whole post rests on and the one thing here I haven’t measured at all. It’s plausible, it matches everything I’ve seen, and it is not a result.

What would change my mind: a comparison across teams where the loops with no negative-capable term in them show the same defect-escape rate as the loops that have one. That would make the distinction decorative — a nicer vocabulary for the same work — and I’d want to know, because I’m currently using it to decide what to build.

Sources

• Royal Society, motto nullius in verba, adopted in the First Charter (1662), from Horace’s Epistles I.1 — “take nobody’s word for it,” adopted to signal that claims get settled by experiment rather than authority. Borrowed here with the pronoun turned around: the expensive word to take on trust is your own.
• Karpathy, 2 February 2025 — “fully give in to the vibes… and forget that the code even exists.” The foil: cited to be disagreed with, on the grounds that the operative failure is unfalsifiability rather than distance.
• Huang, Chen, Mishra, Zheng, Yu, Song & Zhou, Large Language Models Cannot Self-Correct Reasoning Yet, arXiv:2310.01798 — intrinsic self-correction without external feedback doesn’t reliably improve reasoning. Cited only for the independence requirement.

---

Extends Skills Are a Codebase — that post’s shouldn’t-fire case is the narrow instance of this argument, and this generalises it from skill triggers to any check. Distinguish from You Can’t Self-Measure Recall: the unfalsifiability there is a property of a question nobody can answer from the inside; the unfalsifiability here is a property of a loop you built. Sibling to Pick Your Blind Spot — that one asks which check to reach for, this one asks whether what you have is a check.