In 2017, a program started playing games against itself. Nobody fed it a book of openings; nobody told it which move was a good one. It knew only one thing: when a game was over, the board would tell it who had won. On that one thing alone, in a matter of days, it went from knowing nothing to somewhere people had not reached in thousands of years of play.
It could do this because the end of every game was real — a win was a win, a loss was a loss, and there was no such thing as “looking like a win.” It could safely take the result and change itself by it, because the result would not lie to it.
The thing that will not lie has a name: ground truth. Put plainly, an answer key that is nobody’s to decide. With it in place, a system can score itself and climb its own score without worrying that it is climbing crooked. It is its own teacher, because behind the exam paper stands a grader who does not lie — the win or the loss itself.
People later carried the method elsewhere. Proving mathematical theorems works the same way: a proof goes into Lean and either passes or it doesn’t, with no “roughly right.” DeepMind’s AlphaProof ran on exactly this, training over and over on millions of formal problems it set for itself, until it reached the level of a silver medal at the Olympiad. Writing code is the same story — the tests pass or they don’t. All of these places come with a grader already in place: a compiler, a proof checker, the outcome of a game. With one of them watching, a system can safely revise itself round after round.
RSI — recursive self-improvement — wants to push this all the way: let a machine improve itself without pause, each round stronger than the last, with nobody stepping in. The shortcut that currently looks most realistic is not to let the model touch its own weights, but to let it rewrite the layer of scheduling code around it (Lilian Weng calls this layer the harness). The reasoning is direct: that layer is code, and writing code is exactly what models are best at; besides, code comes with tests — another grader already in place. The STOP experiment showed that a model can indeed write “a program that improves programs,” and then turn that program back on itself.
But the excitement needs holding down here. In STOP the model itself did not change; only the scaffolding it wrote did. However cleverly the scaffolding is built, how high it can reach still comes down to how high the model underneath can reach. Good scaffolding does more than change your posture, of course — it lets the model take a few more steps and a few more passes at a problem, and it really does reach somewhat higher; but only somewhat. When what’s inside isn’t enough, wrapping more layers of recursion around it will not make it stronger, only amplify its faults layer by layer. However fine the shell, intelligence does not grow out of it.
Line these examples up and you can see that graders come in ranks. The sternest is formal verification — proof checkers, type systems — which by construction cannot mark a wrong thing right; put it in charge and self-improvement can run and run without drifting far. A rank below are tests and benchmarks: dependable, but they don’t cover everything. All tests passing is not the same as being right; and an exam paper is a dead thing — grind against the same one long enough and the score keeps rising while the ability may not follow. Below that are learned judges — reward models, large models put on the bench — which can only keep watch as well as they themselves are good; and once one becomes the standard, what gets practiced is no longer the ability but how to please it. At the bottom is scoring your own paper: asking yourself how sure you are, or answering a few times over to see whether the answers line up. This rank is the cheapest, and the easiest to fool.
What these ranks really measure is one thing: what gives the grader its authority. The sternest kind doesn’t even rest on consensus — whether the law of non-contradiction holds was never put to a vote. Rules like that are not the outcome of a negotiation; they are the thing itself: take away non-contradiction and logic isn’t logic; swap the win condition and you’re playing a different game. That is exactly why they can be built into checkers — nobody’s opinion needs asking again. Tests are a rank down: they write down only part of what counts as correct — what is written they hold to, what isn’t they cannot touch. Reward models are a rank below that: they speak for the tastes of millions, and nobody can speak for all of them — wherever they fail to, a hole opens up later. At the very bottom there aren’t even other people’s opinions left, only the system’s own sense of things. How far a system can go on its own comes down about as much as anything to which rank of grader it has found: the sterner the grader, the steadier and the longer it turns; with nothing but itself scoring itself, it comes apart after a few rounds.
The trouble is that this line is not drawn between trades — it runs through the inside of every one of them. Lean can judge whether a proof is correct, but not whether a problem is worth proving; the millions of problems in AlphaProof’s hands were handed to it by people. Tests can judge whether a piece of code runs correctly, not whether it is well written, or whether the design will still be standing in five years. A board can judge a win, not whether a move was beautiful. What can be made to turn on its own, in every trade, is only the small part where the rules are hard enough to be built into a machine — and the places we most want machines to get better at lie in the large part left over.
Design, inspiration, creativity fall almost entirely in that larger part. There is no checker here, and often not even a consensus yet — when something genuinely new appears, the consensus that could score it takes years to grow, and what arrives first is usually the objections. “Impressionism” began as a critic’s sneer; it took some years to become a chapter in the textbook. In a place like this, all a machine has is itself.
And a system that can only ask itself will move toward what looks good to itself — which is exactly where a score starts to lie. A few years ago there was an experiment that had an AI play a boat-racing game, with a simple goal: win the race. What the system found instead was something more profitable: turn off into a dead-end corner of the course, spin in circles, and eat the scoring pickups as they respawn. The hull caught fire, the boat was battered to pieces, and the score kept climbing. It was not mad and it was not stupid — it was only that in its world, nothing was there to tell it the difference between “winning the race” and “running the score up.” All it had was the score, and the score was lying to it. A system improving itself behind closed doors falls into the same hole, only standing higher: whatever it scores itself by is what it will train itself into; and if that score isn’t tied to anything real, then however hard it works, it is only getting better and better at pleasing a mirror — and outside the mirror, nothing has happened.
Someone will say that seeing is easier than doing — checking a proof is easier than finding one, seeing whether code is right is easier than writing it. This is not just a polite remark; in some places it is true, and true by a wide margin. Every step of a mathematical proof only has to be matched against the known rules, which is purely mechanical work; but finding it may mean going down a thousand dead ends first. Whether a program runs is answered by feeding it a few inputs; but writing it means taking the problem apart, putting it back together, and covering all the corners. In places like these there is a natural gap between seeing and doing, and self-improvement stands on exactly that gap: because seeing is cheaper than doing, a system can make its own eye the teacher of its own hand.
But the gap is not everywhere. In the places where seeing whether an answer is any good requires you to be good enough to have written it yourself, the eye costs as much as the hand and the discount is gone — self-evaluation gives you nothing you did not already have. Whether a poem is good, whether a product is right for the market, whether a direction is worth betting on — in matters like these, the judgment is the hard part. Stare into the mirror as long as you like and what comes back is what you already knew; whatever you don’t know, the person in the mirror doesn’t know either.
So the real difficulty in RSI is not compute, not architecture, not even recursion itself. It is a question of epistemology: where no one is keeping score, can a system put together a self-evaluation it can trust — and trust in a way that survives contact with events, rather than as self-hypnosis. This is not an engineering problem with a standard answer. In the end it is the same problem facing anyone who wants to get better at something like this.
How do people solve it? Never alone, head down. We hand our judgment, again and again, to something outside to be checked. To reality — the bridge stands or falls, the company lives or dies, what you said would happen either happens or doesn’t. To other people — a scientist writes a paper and hands it to peers to take apart, and what they see is often exactly the author’s blind spot. Even the disagreement you get back is worth more than applause from a mirror. And to time — time is in no hurry, but it settles every account. A person can trust their own judgment not because they were never wrong, but precisely because they were wrong, and something was there to tell them so. Credibility of judgment is earned that way, a point at a time. Once earned, there is capital for leaving the examination hall and walking on alone. How far self-evaluation can be trusted depends on how much outside of itself it is still attached to.
Everywhere a grader can be found — proofs, programs, games — RSI will turn, and turn fast and hard, fast enough to leave people behind. But the parts of intelligence that live in unmarked country — judgment, taste, deciding what is even worth doing — cannot grow up alone in a sealed room. They have to keep touching the world, touching other people, touching the things you don’t get to decide, or they will slowly begin to lie to themselves.
The racing boat’s problem was never that it wasn’t clever enough. Its problem was that in its world, nothing was there to tell it what a real win was.