A Big Rule Set Is an Attention Problem

“A wealth of information creates a poverty of attention.” — Herbert A. Simon, Designing Organizations for an Information-Rich World (1971)

A single clay reading stone holds open one leaf from a quiet fan of receding rule papers.

Running my code reviewer’s full rule set through the model on a nine-file change took about sixteen minutes and sixteen dollars, and turned up four low-severity nits. Most of it went on scanning for rules rather than judging the code.

An opinionated rule corpus is the asset. But the bigger it gets, the more it threatens to turn back into a liability, and the reason is attention rather than cost.

BLUF

A big corpus can fit the context window and still overflow the model’s attention. Those are two different limits, and the second is the one that quietly bites.
That’s lost in the middle, and it’s documented — models use the middle of a long input least. A rule set inherits it too, so more rules mean quietly worse reviews — and that extension is mine, not Liu’s result.
The fix isn’t a better prompt. It’s spending the attention budget. Move the deterministic noticing into code: a detector flags the candidate sites, and the model’s attention lands on judging them instead of scanning every rule against every line.
Mechanisation is an attention lever, not only a cost one — and mostly it moves the noticing, not the rule. Only about a third of my rules fully leave the model’s context; the rest still need a judge, but no longer a scan.
Structure the rules that stay — target, exceptions, examples, why — and split them across passes so no single context holds the whole corpus.
A live loop shrinks the in-context set over time. Every lead the deep pass drops becomes a recorded exception, mechanised where it can be.

Rules in context, and the wall

A large corpus has an obvious problem: you can’t stuff every rule into the model’s context and expect equal attention to all of them. Language models use the middle of a long input least and the ends most, a measured positional effect. Liu et al., Lost in the Middle (2023), arXiv:2307.03172 — a measured positional effect on what gets used, not a claim about attention. It names where information gets used rather than “attention” directly. But the consequence for a big rule set is real. Pile a hundred prose rules in, and the judgment call you most needed — sitting at position fifty — is the one most likely to get skimmed. You get a review that technically saw every rule and acted on whichever ones landed where the model still looks.

A small check backs the direction, if not the tidy shape. I planted one violation and buried its matching rule in checklists of 5, 20, and 80 rules. Recall of that violation fell as the list grew (≈50% → 44% → 33%) and as the rule sat lower in it (≈56% at the top → 28% at the bottom). A bigger list does bury a rule. The check was noisy — n=6 per cell, one 7B model — and it didn’t reproduce Liu’s specific middle-dip cleanly, so it reads more as “longer and lower both hurt” than as a clean U. The extension held directionally.

The sample size is the lesser doubt. Here’s the bigger one: this ran on a 7B model and the pipeline runs a frontier one. Whether a small model’s attention behaviour transfers upward is unestablished, and nothing in my check speaks to it. I hedged the number because the number had a unit, which is the exact failure another post here is about.

Spend the attention budget

The fix is not a better prompt. It’s to spend the attention budget deliberately.

Some rules can be made deterministic: the pattern matches, the syntax-tree checks, the mechanical detectors. Those get compiled out of context into code that runs without the model at all. A large detector set already runs that way.

Pulling those out isn’t only a cost lever. It’s an attention lever. Every rule you move to code is context the model no longer has to read, so its attention lands only on the rules that genuinely need judgment. Mechanisation is how the routing keeps working as the corpus grows. A live loop pushes it further: every lead the deep reviewer drops becomes a recorded exception, mechanised where it can be, so a known non-issue stops costing model attention next time round.

The broader principle is to disclose the right context at the right time instead of cramming it all in, and people are starting to measure it directly. A controlled study of progressive disclosure for long-context agents found it “buys context, not intelligence”: near-zero gain when the material is small enough to navigate, decisive once it’s too big to read through, with one level of disclosure beating deeper nesting (He et al., 2026). Compiling to a detector is the deterministic end of that same move, which is my extension of their point rather than their finding. The rules a detector fully owns don’t get disclosed to the model at all.

Now the honest shape of it. That nine-file run up top is the “before”, and mechanisation is messier than “move the rules to code”. Of about a hundred and fifty rules, only a third fully leave the model’s context; the rest still need a judge.

What a detector usually removes isn’t the rule. It’s the noticing. It flags every candidate site deterministically, and the model’s attention goes to judging those candidates instead of scanning every rule against every line. Even when the rule text stays, the expensive part — the scan — doesn’t. That’s the attention lever, said honestly: mechanise the noticing, not necessarily the rule.

A third route: delete the rule

There’s a route that isn’t mechanisation at all. Anthropic reported removing “over 80%” of Claude Code’s system prompt for its more advanced models “with no measurable loss on our coding evaluations”, and the diagnosis they give is worth borrowing: they “were overconstraining Claude Code” across the system prompt, the CLAUDE.md files, and the skills.

Their transcripts showed directives clashing inside a single request — “leave documentation as appropriate” sitting next to a blunt “DO NOT add comments.” That’s a second way a corpus eats attention, and it isn’t positional. It’s interference: the model spending budget working out which instruction wins before it can act on either. Their multi-clause comment prohibition became one judgement line — “Write code that reads like the surrounding code: match its comment density, naming, and idiom.”

Alex Ewerlöf argues the general version from the other side, and it’s the part that should worry you: a natural-language rule set has no consistency checker. Linters and compilers catch a fraction of the ways one instruction can contradict another, and the only reliable way to surface a clash is to run the agent and watch it fail, which costs far more than a static check. So the corpus can be internally inconsistent and nothing in your toolchain will tell you.

Two things to keep straight. The scope depends on the model — the deletion was for the more advanced ones, so it’s a claim about what a model can now carry on its own, not a claim the rules were always dead weight. And the check behind it was an eval, which climbs only as high as what you managed to specify; “no measurable loss” is evidence, not proof, that nothing left with the deleted text.

The route is still real, and it’s cheaper than mechanising. Before you spend engineering on a detector, check whether the rule is one the model no longer needs told.

Structure what stays

The rules that stay in front of the model are structured to earn their attention. Each carries a target, its exceptions — the field most rule products I’ve looked at skip — examples that feed the deep verification pass, and a why. They’re split across passes rather than dumped in one blob, so no single context holds the whole corpus at once.

That why does double duty. It’s unpacked into the review comment to teach the author, so the rule doesn’t just catch the mistake, it explains it.

Without routing, attention fades across a long rule field; a detector selects candidate sites so model judgment concentrates only where it is needed. — A long rule list dilutes attention, especially through its middle. A detector routes candidate sites into a much smaller judgment surface; the rule may remain, but the scan leaves the model.

The point

A rule corpus scales only if you spend attention deliberately. Mechanise the deterministic noticing so the model’s scarce attention lands on judgment, even where the rule text itself has to stay. The corpus is the asset; this is how you wield a large one without drowning the model that has to read it.

How sure I am: The detector set and the exceptions-pass loop are lived — both running now, genericised. Lost in the middle is prior art I’m citing, not claiming. The extension to a rule set — more rules, worse recall of the buried one — now has a small directional check (Qwen2.5-7B, n=6 per cell): recall fell as the list grew and as the target rule sat lower, though it was noisy and didn’t reproduce the specific middle-dip cleanly. That mechanising rules out improves the judgment, rather than only cutting cost, is reasoned from the attention argument and not something I’ve isolated and measured. The receipt is real — a nine-file run, roughly sixteen minutes and sixteen dollars for four nits, genericised — with two honest bounds: only about a third of the rules fully leave the model’s context, and whether mechanised noticing lifts recall is unproven, since a detector protects the verdict rather than recall, so a pre-filter miss is silent. The Anthropic deletion figure is theirs, on their harness, checked against their own coding evals; I cite it as a practitioner receipt that the route exists, not as a measurement that transfers to a rule corpus. No such ablation has been run against mine, so “delete before you mechanise” is a prompt to check rather than a result.

Sources

Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni & Liang, Lost in the Middle: How Language Models Use Long Contexts (2023), arXiv:2307.03172 — models use information in the middle of a long input least: a measured positional effect on what gets retrieved and used. The extension to a rule set (“more rules → skimmed → worse reviews”) is mine, not Liu’s measurement.
He, Zhao, Wang & Chen, Is Progressive Disclosure All You Need for Long-Context Agents? (2026, arXiv:2607.17598) — a controlled study finding progressive disclosure “buys context, not intelligence”: near-zero gain when a strong harness can already navigate, decisive once the corpus is too large, and one level of disclosure beating deeper nesting. Corroborates the disclose-don’t-cram principle; mechanising to a detector is its deterministic end. Read pre-publish and cited from the abstract’s qualitative findings, not its numbers.
Thariq Shihipar, The New Rules of Context Engineering for Claude 5 Generation Models (claude.com blog, 24 July 2026) — Anthropic removed “over 80%” of Claude Code’s system prompt for its more advanced models “with no measurable loss on our coding evaluations”, having found they “were overconstraining Claude Code” across the system prompt, CLAUDE.md files, and skills. Transcripts showed directives clashing inside a single request, and a multi-clause comment prohibition was replaced with a single judgement line. Cited for the deletion route and the interference mechanism, both theirs, with their capability scope (more advanced models) intact. Reading it as a third disposal route beside mechanising and structuring is mine.
Alex Ewerlöf, Coding Is Not Solved (blog.alexewerlof.com, 26 September 2026) — explicitly opinion-led, cited for one argument: a natural-language instruction set has no consistency checker, linters and compilers catch only a fraction of the contradictions, and the dependable way to surface one is to have the agent attempt the build, which costs far more than a static check. Argued rather than measured; cited as a second, independent statement of the interference mechanism, so the claim doesn’t rest on one vendor’s transcripts.

Related: Noticing Is Cheap, Judgment Is Scarce — the same move, costed.