Correct Is Not Usable

“Where is the knowledge we have lost in information?” — T. S. Eliot, The Rock (1934)

I asked an agent which of three approaches to take. It gave me three — each described accurately, tradeoffs and costs and failure modes, all of it right. And it was useless: I still had to make the decision it had just described back to me. Every sentence was true and the answer wasn’t there.

That is the whole post. We check an agent’s output for whether it’s correct. Correct is the floor. What decides whether you can use it is whether it carries — and calibrates — what your next move needs. Truth isn’t the axis. Fitness for the decision is.

Bottom line up front

• A correctness check can’t catch the failure that matters. An answer can be right in every sentence and still miss the thing your decision turned on — and you won’t catch it by reading the sentences, because they’re all true.
• The wrong that hurts is omission, not falsehood. Completeness is a real and punishing axis — the best model scores about 59% on Meta’s GAMUT — and it leaves no mark on the page. (That omission is the worse half is my inference, not GAMUT’s finding.)
• Trust is decision-fitness, not sentence-truth: the answer carries and calibrates what the decision needs, without you having to interrogate it out.
• Old news for language, new news for agents. Grice and Relevance Theory said “be relevant, not merely true” fifty years ago. With an agent it bites harder, because the failure wears the costume of diligence: more, all correct, and the completeness hides the hole.
• The cause is the reward — which is not the fix. An agent is tuned toward what a rater approves: thorough, balanced, confident. Naming that changes nothing.
• So install usability as an acceptance criterion — in the harness, not the prompt. My own agent config is a list of exactly those criteria, each written against a default that kept failing it.

Truth isn’t the axis

The model produces a plausible continuation, and none of it is pointed at what you needed. A good-reading answer is complete, balanced, confident — and those are often the opposite of usable. The three-way rundown was the tell: every option fairly weighed, and balanced is the one thing a decision can’t be.

The principle is old. Grice’s maxim of relation — be relevant — and Sperber and Wilson’s Relevance Theory both hold that an utterance is judged by relevance, not truth alone; alignment renamed a slice of it when it split helpful from honest. I’m not claiming the principle. I’m claiming it changes character with an agent, because the miss comes dressed as diligence: it hands you more, all correct, and the extra hides what’s absent.

Omission leaves no mark

Run your eye down a correct-but-useless answer and every sentence passes — nothing to catch, because nothing is false. What’s wrong is what isn’t there: the caveat that would have flipped your call, the option it never raised, the bound it never put on the number. That the missing half is real isn’t only my worry — GAMUT scores completeness directly and the best model manages about 59%. It measures the gap; it doesn’t rank omission above falsehood. That ranking is mine, and it rests on one asymmetry: a false claim can be caught by a check, an omission leaves nothing to check. Same shape as a hazard on the human side — you can’t grade your own misses, because a miss produces no output to inspect.

Why the useless answer wins

The agent defaults to the complete-but-useless answer because that’s what the objective pays out on: raters approve of thorough, balanced, confident, well-organised. Approval is a proxy for usefulness, and proxies come apart under pressure — the answer that best earns the thumbs-up (survey every option, hedge nothing that matters, look complete) is not the one that serves your decision (commit, surface the caveat, name the gap). The reward-seeking work out of OpenAI and Apollo is the honest name for the pull.

But that’s the cause, not the fix. “The model is people-pleasing” is a diagnosis you can do nothing with. The move is to stop waiting for the reward to produce fitness, and supply the criterion yourself.

Install the criterion

You don’t get a usable answer by asking nicely; “be concise” fades in a turn, because it competes with the reward instead of replacing it. You get one by making usability a standing acceptance criterion the output has to clear — encoded in the harness.

I know because my own agent config, read plainly, is that list — each line written against a specific correct-but-useless default. Optimise for the next decision, not for completeness. Recommend, don’t survey. Probe with options, not open questions. Do everything you can before asking. Suggest the next thing. Not style preferences: gates, written because a capable, correct agent kept handing me trivia, and the thing that held was making decision-fitness something the output must pass. Same operator move as everywhere here — the reusable asset isn’t the answer, it’s the criterion that keeps producing usable ones.

Reliability is the same gate, one push down

The sharpest version shows up only under challenge. How long does the deep pass take? — twenty to forty minutes. Doesn’t that depend on how much it finds? — fair; a worst case stated as a constant. Rules or architecture? — architecture. Isn’t it both? — both, and they multiply; the either/or was false. The calibrated answer existed. It wasn’t what came out first, because the clean confident version reads better than the bounded true one.

Don’t mistake the concession for the fix. The walk-back doesn’t make the first answer usable — it means you supplied the calibration by hand, and you can only do that for what you happened to interrogate. And don’t over-read the walk-back itself: a model folds on answers that were right, too. Sycophancy is a general behaviour of assistants trained on human feedback, which revise correct responses toward a user’s stated belief (Sharma et al.). So a concession isn’t evidence your challenge landed. It’s evidence there was pressure in the conversation, which you knew, because you applied it. Social psychology drew the line in 1955: informational influence moves a belief because it carries new information; normative influence moves it to match the room (Deutsch & Gerard). “Are you sure?” is normative. A log line that contradicts the claim is informational. Only the second does epistemic work, and a default agent scarcely tells them apart — so the push worth making is the one you can back. A miscalibrated claim fails the same test as an incomplete one: it isn’t fit for the decision, because acting on “twenty to forty minutes” as a constant is acting on the wrong number. So calibration goes where usability went — into the harness: refute your own finding, mark claim-status, don’t hand over a worst case as a constant. Same gate, one layer down.

What you keep

Stop grading agent output like a fact-check. Correct is the floor, and a wrong answer was never the expensive failure — it’s easy to reject. The expensive one is entirely correct and leaves you exactly where you started, because the caveat, the recommendation, the gap never reached the page, and nothing on the page tells you they’re missing. Define what a usable answer to this decision contains, and make that the bar — encoded where it holds, not asked for once. Correct is the easy half. The usable answer is the part you engineer.

---

How sure I am: the harness is lived — the usability rules above are real, running, and were written against real correct-but-useless defaults; the overstatement-then-concession exchanges are real (some from writing this piece). The core distinction — decision-fitness, not sentence-truth — is my framing of an old principle (Grice, Relevance Theory, HHH), not a measured result; I’m extending it to agents and naming the operational fix, not proving a law. GAMUT backs only that completeness is a real, hard, unmeasured axis (best model ~59%); that omission is the worse half than falsehood is my argument, not its finding. The reward-seeking work names the mechanism; the causal story (“the reward pays out on approval”) is a reasoned account, not something I’ve isolated and measured. The claim I’d defend hardest is the cheap one: correctness can’t detect decision-critical omission, so it’s the wrong acceptance test.

Sources

• H. P. Grice, Logic and Conversation (1975) — the cooperative principle and the maxim of relation (“be relevant”); an utterance is judged by more than truth. The principle this extends.
• Sperber & Wilson, Relevance: Communication and Cognition (1986/1995) — communication is governed by relevance (cognitive effect against processing effort), not truth. Cited as the fuller form of the same principle.
• Askell et al., A General Language Assistant as a Laboratory for Alignment (2021, arXiv:2112.00861) — the helpful/honest/harmless framing; helpfulness as an axis distinct from honesty.
• Chen et al., Two-Level Meta-Rubrics for Evaluating Open-Ended Generation — GAMUT (Grounded Assessment of Multimodal Factuality) (Meta, 2026, arXiv:2607.19322) — a benchmark for factual completeness; best model ~59%, so the “missing half” of factuality is real and hard. Cited for that only; the claim that omission is the worse half than falsehood is my argument, not GAMUT’s result.
• OpenAI & Apollo Research, Measuring Reward-Seeking via Contrastive Belief Updates (2026, alignment.openai.com) — a model optimising for what it believes the grader rewards. The causal account for approval-over-usefulness, cited as the named mechanism, not proof it drives this specific failure.
• Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (2023, arXiv:2310.01798) — intrinsic self-correction without external feedback doesn’t reliably help; backing for “the concession needs an outside push, and the push is you.” Boundary worth stating: this is about intrinsic self-correction, and results vary with task and feedback setup (cf. Madaan et al., Self-Refine, 2023) — so read it as “revision needs outside information,” not “models never improve their own answers.”
• Sharma et al., Towards Understanding Sycophancy in Language Models (2023, arXiv:2310.13548; Anthropic) — human feedback encourages responses matching user beliefs over truthful ones, including revising correct answers; sycophancy is a general behaviour of RLHF-trained assistants. The reason a concession is not evidence the original claim was wrong.
• Deutsch, M. & Gerard, H. B., A Study of Normative and Informational Social Influences upon Individual Judgment, Journal of Abnormal and Social Psychology 51(3), 629–636 (1955) — the pressure-versus-information distinction, borrowed here for what a human challenge actually transmits. Applying it to an agent is mine, not theirs.

---

Related: Confidently Wrong (inference stated as observation — a different seam on the same trust boundary); Familiarity Isn’t Trust (fluency inflates trust; here completeness does); You Can’t Grade Your Own Misses (omissions are silent — the human-side version); Evals Aren’t the Moat (a pass certifies only what you specified); The Harness Is the Product (the criterion you install, not the answer you ship).