A Flip Is a Missing Rule
A colleague sent me the cleanest bug report I've had all year. He'd taken an open change, squashed it, and opened it again — same content, byte for byte, as…
Tagged evaluation
-
-
Buy Back the Coverage
I had already published the warning: if a model only judges what a scanner shortlisted, a site the scanner fails to flag is a false negative nobody ever sees.
-
Intermittent Competence Looks Like Coverage
My reviewer caught a deleted flush call three times out of three, and said nothing about eight of thirteen removals in any run.
-
A Constraint You Can Quietly Lift Isn't One
I had a harness measuring whether a set of skills helped a cheap model. It came back negative: no lift worth the tokens. I believed it.
-
Keep the Factory, Ship the Result
I built an eval to tell me whether my agent's skills were earning their place. For a while it said they weren't — clean, green, confident.
-
Rules Are the Alpha
Two reviewers, the same commit — a fat change, a few thousand lines. One was a deep reviewer I own, running a portfolio of judgment rules tested across a…
-
A Cue Finds More of What It Can Already See
An AI reviewer finds three bugs on a real PR. Tell it to expect four or five and it finds more — but only more of what it could already see.
-
Shrink, Reorganise, Relocate: Why Synthetic Wins Fail on Real PRs
I spent a stretch trying to make an AI code reviewer cheaper without letting it miss more. Cost and recall, two dials pulling against each other.
-
You Can't Grade Your Own Misses
Ask a code-review tool how good it is and you'll get a precision answer: of the things it flagged, how many were real.
-
Your Merged-PR History Is Survivorship-Biased Training Data
In the Second World War, the Statistical Research Group was asked where to add armour on bombers, and handed the damage data from the planes that came back …
-
Evals Aren't the Moat
A fast eval told me a restructured module was clearly better. Comfortable margin. I ran it again to be sure, got the same verdict, promoted the new version…
