You Can't Fine-Tune Yourself

“We can know more than we can tell.” — Michael Polanyi, The Tacit Dimension (1966)

I wanted a code reviewer that reviewed like me. The obvious move was to train one on my own outputs — years of review comments, commits, code. Every one of those artifacts is a decision I made, and a fine-tune is exactly the tool for given this, produce that.

I never ran it. Not because fine-tuning is hard — at the scale I needed, it’s a weekend. I didn’t run it because the artifacts don’t hold the thing I was trying to copy. What made those reviews mine was mostly not in them.

That sounds like a soft claim about tacit knowledge. It isn’t. There’s a test, and it’s the one I run now before reaching for a training run. It has three questions, I got to them in the wrong order, and the one I reached last is the only one that mattered.

Bottom line up front

• The weights hold the transform. The input holds the task. Weights are how you get from A to B. The input is which A, plus the facts that decide which B is right. Judgment is the check on what comes out.
• Three questions decide it, and the third is the one that bites. What does the verifier cost. How big is the target. And — the one I reached last and should have reached first — do your examples contain the facts that selected them. The first two are about feasibility in general. The third is about your corpus, and it’s the one that stopped me.
• “Write like Hemingway” works; “write like me” gets the surface and misses the selection. A model can match your measured style to three nines and still not be choosing what you’d choose. The difference isn’t prose versus code.
• Your artifacts record the decision, not the facts that selected it. Fit a model to your outputs perfectly and it still hasn’t got the half that mattered. This is the question the title rests on, and it’s the only one of the three that no budget fixes.
• Rules and weights aren’t rival mechanisms. Prompt tuning optimises the input of a frozen model. A rule corpus is the discrete, readable version of the same move — worse at optimising, and you can edit one line of it.
• Rules are the available knob, not automatically the better one. Most of us run one model we didn’t train, so the input is the only thing that moves. That is precisely why the input is where the differentiation lives: everybody has the same weights.

Three parts, one of which you can move

Strip the setup down and there are three parts. The input is the task and the facts that bear on it. The weights are how you get from the task to an answer. Judgment is the check: is this output good enough? If it isn’t, exactly two things can change — the input, the weights, or both.

Most teams never change the weights. You run a model somebody else trained, on a schedule you don’t set, and you swap it for a newer one twice a year. The teams that do train are almost always training something narrow and well-specified, which is the whole point of what follows. For everyone else the input is the only knob.

I want to be straight about what that means for Rules Are the Alpha, because it’s the honest version of that argument. Rules aren’t the alpha because encoding beats training. Rules are the alpha because the weights aren’t yours to move and the input is. That reads like a downgrade and it isn’t: your competitor is running the same weights you are, from the same handful of labs. The weights are the part nobody differentiates on. Whatever you know that they don’t has to enter through the input, because there’s nowhere else for it to go.

The objection is mine, and it’s the good one

Here’s the argument against everything above, and I’m the one who made it.

The weights are a generator. Given the previous tokens, what comes next — that’s the whole model. Training is encoding how to generate. And you can tell a model write like Hemingway and it will, without a single gradient step, on a frozen model, from conditioning alone. If a whole authorial voice fits through the input, and generation fits in the weights, then “transform in the weights, task in the input” is a distinction without a mechanism. Choosing words with a consistent tone and choosing a design with a consistent shape are the same operation on the same machine.

That objection is right about the mechanism. There is no separate faculty. It’s input and output the whole way down, which is the same thing The Raw Is the Trust Boundary says about reading versus writing.

So if the mechanism is identical, the difference has to be in the target.

How big is the target, and what does the verifier cost

Hemingway is not a point. It’s an enormous region of output space. Thousands of different sentences are all recognisably Hemingway, any one of them passes, and you can check in under a second by reading it. Huge target, free instant verifier. Of course conditioning gets you there.

The architectural call I’d have made in that review is close to a point. There’s one shape I wanted, the near misses all compile and pass the tests, they’re wrong in ways that don’t show up for a while, and the verifier is six months of maintenance. Tiny target, enormously expensive verifier.

Those two aren’t prose and code. They’re both just outputs. What separates them is how much of the output space counts as correct, and how much it costs to find out.

Half of that is not new. Cheap, checkable rewards are why maths and code got the post-training attention they got — if you can grade an answer for free, you can train against it. What I’d add is the second column, and what it’s for: not a rival explanation, but the thing that tells you whether a cheap verifier is even likely to exist. You can grade Hemingway by reading. You cannot grade an architecture by reading.

Write like me sits in the middle of that axis, and closer to the architecture end than to Hemingway. Hemingway works because Hemingway is already in pretraining — thousands of pages of him, plus a century of people describing what he does. You aren’t.

When researchers tested prompted imitation properly — more than 400 real authors, over forty thousand generations per model — the models did reasonably well on structured genres like news and email and struggled at the informal, personal end, blogs and forums, which they call a fundamental gap in personalized adaptation (Wang et al., 2025). Be exact about the setup, because it limits what I can claim: that’s in-context learning from a handful of the author’s samples. Nobody in that study fine-tuned on a personal corpus. It tells you prompting doesn’t get there; it doesn’t tell you training wouldn’t. A separate result cuts finer. Continuing a human-written prefix, prompting reached 99.9% agreement with the author’s measured style on that paper’s own benchmark — and the output still wasn’t human-like, because it stayed statistically far more predictable than the person it was copying (Jemama & Kumar, 2025). So the surface transfers, and transfers well. What doesn’t come with it is the thing that made the author pick that sentence over the other one. Matching the style is not being the author.

So the test, before you reach for a fine-tune:

What you want	Target	Verifier	Trainable?
Write like Hemingway	huge region — thousands of outputs pass	free, instant (read it)	yes, and you don’t even need training
Apply a rule and try to refute yourself	broad — the skill generalises across rules	cheap but loose — a citation check grades shape, not application	yes — this is the case for training the operator
Write like me	narrow — one person, not in pretraining	slow and subjective	partly, and worst where it’s most personal
Make this architectural call the way I would	a point	the outcome is months out, and every cheaper proxy is weak	no — and the reason isn’t the verifier

The rule falls out of the table. Broad target and a cheap verifier: train it. A point with an expensive verifier: it belongs in the input, and it belongs there permanently, because it changes.

Row two deserves its own asterisk, because it’s the row I’d most like to be true. Did it cite a line is cheap, and cheap is the point — but it grades citation-shaped compliance, not whether the rule was applied correctly, whether the counterexample was any good, or whether the model should have abstained. A verifier that cheap will happily train a model to cite fluently and reason badly. Cheap and aligned are different properties, and only the first one is free.

The two columns aren’t independent in the way I first wanted them to be, and the counterexample is a backdoor. When this exact trigger appears, emit this exact string. That’s a point target — one output passes, everything else fails — and it trains in almost no data, because the verifier is string equality and costs nothing.

So target size doesn’t cause verifier cost, and I shouldn’t pretend it does. They come apart in both directions. Exact targets verify for free all the time: a parser either accepts or it doesn’t, a test either passes, a migration either round-trips. And broad targets can be brutally expensive to check — make this memo persuasive has an enormous passing region and no cheap grader anywhere in sight. What makes a verifier cheap is having a specification or a proxy that actually lines up with the thing you want. Target size is a tell, not a cause: when a huge fraction of outputs pass, reading one is usually grader enough, which is why Hemingway is free to check.

Which leaves the verifier doing most of the work across the first two questions — and I should say plainly that this half isn’t news. Cheap graded signal is the premise of the entire verifiable-rewards line of post-training. If the test stopped here, it would be familiar advice in new packaging.

That’s the step Build the Operator leaves implicit. That post says train the verb and keep the rules as data, and it’s right, but it argues from the economics of a review system — you don’t want to retrain to edit a standard. The reason I think the verb trains and the rule doesn’t is that the verb is a broad target with a cheap verifier and the rule is a point with an expensive one. Which means the split isn’t a fact about review systems. It’s the test, and it applies to anything you were thinking of training, including yourself.

Where a practitioner stops me

Someone who fine-tunes models for a living will push on the last row of that table, and the push is good.

An expensive outcome verifier doesn’t make a thing untrainable. It makes it slow to iterate on and hard to be confident about. But the training signal doesn’t have to come from the outcome. Thousands of reviews I’ve already written are labels — they exist, they’re mine, and supervised fine-tuning doesn’t need to know whether the call turned out right six months later. You’d evaluate on held-out later judgments, an adjudicated offline suite, a shadow run. The verifier costs six months quietly assumes the deployment outcome is the only available grader, and it isn’t.

The second push is sharper. The architectural call is a point is only true once you delete the input. Nobody trains a model to emit one architecture. The learnable object is a conditional policy: given this repo, this team, these constraints, this diff — make this call. A policy over thousands of situations can be a very broad target even when every individual answer is singular. Define the target at the wrong level and you can prove almost anything unlearnable.

Both land. And both point at the thing I’d left out, because if what you’re training is a conditional policy, it can only be learned from examples that carry the conditions.

Mine don’t.

The trace is a residue

So here’s the third question, and it’s the one the title is about: do your examples contain the facts that selected them? Unlike the first two, no budget moves it.

Take the best case: you fit a model perfectly to your outputs. Every review comment you ever wrote, reproduced on demand. You still haven’t got what you wanted, because the artifact records the decision and not the facts that selected it. The comment says make this a lookup. It doesn’t say that you’d seen this shape grow a fourth branch twice before, that the team that owns the caller was about to change its contract, or that you’d have said the opposite in a service where the branches genuinely differed. None of that is in the trace. It never was.

Peter Naur got there in 1985: the theory of a program lives in the people who built it, and the text is what’s left over. His line is that the death of a program happens when the team possessing its theory is dissolved — the code survives the theory, and being able to read the code doesn’t reconstitute it. Training on my outputs is training on the residue and hoping the theory comes with it.

Be precise about what that does and doesn’t rule out. It doesn’t say no fine-tune could ever learn to review the way I do. It says no fine-tune can learn it from my artifacts, because the selecting facts were never written down — and my artifacts are the entire corpus I was proposing to use. That’s the claim the title is making.

Which is Polanyi’s point at the top, made concrete. I can’t tell you all of what I know, so I can’t write it into a training set either.

What’s actually worth putting in the weights

My father taught maths. When students asked why they were learning something they’d never use, he gave the same answer every time: learning maths isn’t about solving those problems. It’s about teaching you to think.

That’s the split, and it was sitting in front of me the whole time. The problems are the task. The way of seeing is the transform. You put the transform in the weights and you keep feeding it problems.

And here’s the part that should have stopped me before I ever considered the fine-tune: the model already has the transform. It can spot a missing guard, duplicated state, an assertion so broad it promises nothing. Those patterns are learnable and they’re learned. What it doesn’t have is the instance — the thing in your codebase, disguised as reasonable code, that you catch because you’ve been caught by it. The weights were never the thin part. The input was, and I was about to spend a training run on the half that was already working.

Rules and weights were never rivals

The framing itself is a false opposition. Prompt tuning learns a soft prompt — a real parameter, learned by backpropagation — over a frozen model, and across the T5 sizes it closes the gap with tuning the whole thing as scale grows (Lester et al., 2021). One model family, one set of tasks; take the shape of the result, not a law. The gradient goes into the input. Nobody had to touch the weights to change what the model does.

A rule corpus is not that parameter. A soft prompt is a continuous embedding; it isn’t tokens and you can’t read it. A rule corpus is the same move made discrete — optimise the input rather than the weights, by hand, in words. As an optimiser it’s far worse. It has one property the learned version hasn’t: you can read it, and you can change one line of it on a Tuesday.

One caution before you go rule-shaped

Don’t read this as the model can’t, so write a rule. When a model doesn’t do the thing, the usual cause isn’t a missing capability, it’s a failure to ask for it.

I’d like to hand you a clean citation for that and I can only hand you half of one. Password-locking deliberately hides a capability behind a trigger, and the capability comes back under a small amount of the right elicitation — a handful of demonstrations, or RL (Greenblatt et al., 2024). The follow-up cuts the other way, in exactly the setting this post is about. Circuit-breaking was built because password locks were too easy to open; prompting still recovered the capability on multiple-choice, activation steering didn’t, and in code generation only fine-tuning got it back (Hofstätter et al., 2025). Their own recommendation is that where you can fine-tune, you should. So the honest version is narrower than I wanted: capability that looks absent often isn’t, and the cheap way of asking usually finds it — on code, sometimes it doesn’t.

The caution survives that, because it doesn’t need the citation. The other usual suspects all sit on the input side of the line anyway. The context it needed wasn’t there. Two instructions contradicted each other and it picked the wrong one. Retrieval didn’t return the file. Every one of those is an input failure, and none of them is fixed by a training run. Rule those out before you conclude the weights are short.

What you keep

The loop. The model brings the transform. You bring the task, the facts that select the answer, and the check on what comes back. That last one doesn’t move into the weights either — and the name on the commit is still yours.

---

How sure I am: the mechanism claims are backed, the decision rule is mine, and the third question is the only one I’d defend hard. Backed: soft prompts are learned parameters over a frozen model and close the gap with full tuning as scale grows, on T5 (Lester et al.); prompted imitation of an ordinary author’s implicit style is measurably hard, worst on informal personal writing (Wang et al., 400+ authors — in-context, no fine-tuning in that study); style fidelity and human-likeness come apart under completion prompting (Jemama & Kumar). Cited against me: in code generation, prompting and steering failed to recover a hidden capability and only fine-tuning did (Hofstätter et al.) — which is why the caution in that section rests on input failures rather than on elicitation. Mine, argued not measured: the three-question test, the reading of Naur as your artifacts are a residue, and the claim that a missing-conditions corpus is what actually blocks the fine-tune. Cheap graded signal is not my idea — it’s the premise of the whole verifiable-rewards line, and I say so in-body. Two concessions I’d make before you make them: an expensive deployment outcome doesn’t make a policy untrainable, because thousands of already-written reviews are supervised labels and you’d evaluate offline; and “the architectural call is a point” is only true if you delete the input, since the learnable object is a conditional policy and that can be a broad target. Both are in the post. Target size is demoted accordingly — it’s a tell that a cheap verifier probably exists, not the cause of one. Not done: I did not run the fine-tune. This is the argument for why I didn’t, not a report of it failing — a negative result would need the training run and a held-out set of my own judgments, and I have neither. Treat the table as a design heuristic with prior art under it, not a measurement.

Sources

• Michael Polanyi, The Tacit Dimension (1966) — “we can know more than we can tell”; the book opens by reconsidering human knowledge from that fact. The epigraph, and the reason the trace is incomplete in principle rather than by accident.
• Peter Naur, Programming as Theory Building, Microprocessing and Microprogramming 15 (1985) — the theory has primacy over the program text; “the death of a program happens when the programmer team possessing its theory is dissolved.” Backs artifacts are a residue. The application to training data is mine.
• Brian Lester, Rami Al-Rfou & Noah Constant, The Power of Scale for Parameter-Efficient Prompt Tuning (EMNLP 2021, arXiv:2104.08691) — soft prompts are learned by backpropagation over a frozen language model, and close the gap with full model tuning as scale grows across the T5 sizes. Backs the input is a parameter. Two limits: it’s one model family on one benchmark suite, so the scaling trend is a shape not a law; and a soft prompt is a continuous embedding, not tokens — reading a hand-authored rule corpus as the discrete, human-optimised version of the same move is my extension, not the paper’s.
• Zhengxiang Wang, Nafis Irtiza Tripto, Solha Park, Zhenzhen Li & Jiawei Zhou, Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors (EMNLP 2025 Findings, arXiv:2509.14543) — 400+ real authors, 40,000+ generations per model; reasonable results on structured genres, a “fundamental gap in personalized LLM adaptation” on informal personal writing. Scope, and it matters: the imitation is in-context, from a small number of the author’s own samples. Nobody in that study fine-tuned on a personal corpus, so it’s evidence about prompting, not about training. Backs prompted imitation being hard; it does not back the stronger claim that training wouldn’t get there.
• Rebira Jemama & Rajesh Kumar, How Well Do LLMs Imitate Human Writing Style? (IEEE UEMCON 2025, arXiv:2509.24930) — completing a human-written prefix, prompting reaches 99.9% agreement with the author’s measured style on that paper’s own essay benchmark, yet “high-fidelity imitation does not imply human-like unpredictability” (perplexity 29.5 human vs 15.2 model). One benchmark, one verifier, and the setup is completion rather than cold generation. Cited in both directions.
• Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov & David Krueger, Stress-Testing Capability Elicitation With Password-Locked Models (NeurIPS 2024, arXiv:2405.19550) — a capability hidden behind a password comes back under modest elicitation: a few demonstrations, or RL.
• Felix Hofstätter et al., The Elicitation Game: Evaluating Capability Elicitation Techniques (arXiv:2502.02180) — cited against my own convenience. Circuit-breaking is introduced as a harder-to-open alternative because password-locking proved weak. Prompting recovers the capability for both organisms on multiple-choice; activation steering fails; in code generation “only fine-tuning can elicit the hidden capabilities” of their novel organism, and the paper recommends fine-tuning as the default elicitation method. That’s the reverse of a cheap-elicitation-always-works story, in the domain this post is about, so the caution stands on the input failures instead.

---

Related: Rules Are the Alpha (why the input side is where the differentiation is); The Raw Is the Trust Boundary (it’s input→output all the way down, and how tightly the input constrains the answer); Build the Operator, Keep the Operands as Data (the architecture this gives the test for).