It’s Not That Hard to Build Recursively Self-Improving AI
There’s a question that comes up constantly in AI discourse, usually framed with appropriate gravitas: How far away are we from recursively self-improving AI?
The implied answer is always “far enough that we don’t need to panic yet.” The concept gets discussed in the context of superintelligence timelines, foom scenarios, and the kinds of capabilities that would require massive breakthroughs in architecture, training, or reasoning. It’s treated as a threshold, a bright line between “interesting tool” and “existential concern.” And most people, even people who think about AI risk seriously, tend to place that line comfortably in the future.
I want to offer a small piece of evidence that the line is closer than you think. Maybe not in the “superhuman intelligence recursively bootstrapping itself to godhood” sense. But a sense that matters for updating your models about what’s coming.
The Experiment
I have no formal software engineering background. I’m a person who finds AI interesting and has access to Claude Code. Over a weekend, I built a system called Autofiction that does the following:
It writes a piece of fiction based on an autogenerated prompt.
It analyzes the fiction it just wrote, evaluating prose quality, narrative structure, thematic coherence, and other dimensions.
It uses that analysis to rewrite its own prompt, attempting to correct for the weaknesses it identified.
It loops. Indefinitely.
That’s it. That’s the whole system. Write, evaluate, improve the instructions, repeat. Each cycle, the prompt that generates the fiction is slightly different — informed by the system’s own assessment of what went wrong last time.
This is, by any reasonable structural definition, recursive self-improvement. The system is modifying the parameters that govern its own behavior based on its evaluation of its own outputs. It doesn’t require human intervention. It runs continuously.
Yes, I Know
I can hear the objections forming already, and I want to be clear: they’re mostly correct.
This is a toy model. The “self-improvement” is happening at the prompt level, not at the level of weights or architecture. The evaluation function is itself an LLM, which means it inherits all the limitations and blindspots of current language models. The fiction probably converges on some local optimum rather than improving forever. There is no risk here. Nobody needs to worry about Autofiction taking over the world.
All of that is true. And all of that somewhat misses the point.
The Update
Here’s what I think the takeaway should be: the scaffolding for recursive self-improvement is not hard to build. A person with no software background, using publicly available tools, can create a system that exhibits the core loop — act, evaluate, modify, repeat — in a weekend.
The usual story about recursive self-improvement involves a long list of prerequisites: the system needs to understand its own architecture, it needs to be able to modify its own code or weights, it needs a reliable evaluation function, it needs to avoid reward hacking, and so on. And for the scary, world-ending version of RSI, those prerequisites are real. But the structural pattern? The loop itself? That part turns out to be trivially achievable with current tools.
This matters because of a general heuristic I think is underappreciated in forecasting: things that are easy to do in toy versions tend to become easy to do in serious versions faster than expected. The gap between “proof of concept” and “real capability” is often smaller than the gap between “impossible” and “proof of concept.” We’ve crossed the second gap. The first gap is a matter of degree, not kind.
I think the right Bayesian update from a project like Autofiction is this: if your model of “how hard is recursive self-improvement” was calibrated based on the assumption that the basic loop is an unsolved engineering problem, you should move your probability mass. The mundane version of a thing often precedes the transformative version by less time than we expect. Keep your eyes open.
I’d call what you’re describing iterative self-improvement instead of recursive, where the ability to self-improve is itself improved. Loosely speaking, your approach might get you “linear” progress, but not “exponential.”
I’m not sure we should call it self-improvement at all because the prompt is not part of the model, so actually the model (the “self”) is not improving at all. It seems more like practice to improve on a skill, not improving the author itself.
Yeah. Though strictly speaking, only something like self-play (mentioned by Karl Krueger below) is strictly speaking a model improving itself. The more classic example of RSI is a model acting as an AI researcher, working on better ML algorithms (optimizer, architecture, objective function, etc), which doesn’t directly improve the model (the AI ML researcher) itself, only ancestor models which are then trained using this improved ML algorithm. The latter form of RSI is nonetheless more powerful than something like self-play, which is stuck with its likely suboptimal architecture, meaning that it will eventually plateau. An automated ML researcher can theoretically improve up to technological maturity: the best ML algorithm possible.
Agreed. There’s really a continuum here, where the rate of self-improvement can range from sublinear to fully exponential. Processes that lack a mechanism for feeding improvements back into the system’s underlying intelligence are unlikely to achieve exponential growth; they’ll tend to plateau or progress linearly at best.
One takeaway is that an effective regulatory regime aimed at limiting fast-takeoff risk might focus precisely on this distinction — restricting processes that iteratively improve at greater than a linear rate.
Long before modern ML, it was not difficult to write a program which would generate programs, test those programs on some task, and make random changes to the programs in an attempt to find programs which perform better on the task.
It was also not difficult to apply such a program to the task of self-improvement, i.e. score how quickly it finds new-and-better programs on a basket of object-level tasks (including the self-improvement task itself).
All that would have been straightforward in the 1970s, if one had access to a machine to run it all. By the 1990s, a bright CS undergrad could do it on their home computer. The loop you’re talking about was easy to code long before modern ML.
By this standard, does all self-play count as “recursive self-improvement”? Or only self-play where the two players’ roles are different, or not entirely rivalrous (like author and critic) rather than the same and purely rivalrous (like white and black in Go)?
Self-play is the foundational element of recursive self-improvement — it’s what allowed AI systems to bootstrap to superhuman performance in Chess and Go through effectively unbounded iteration. That said, there’s likely a meaningful spectrum within self-play that depends on the independence and algorithmic sophistication of the evaluating role.
Not all self-play is created equal in this regard. Purely rivalrous, symmetric self-play (like white versus black in Go) represents one end of that spectrum, while asymmetric arrangements with structurally distinct roles (like author and critic) may occupy a qualitatively different position — particularly when the critic brings independent evaluative criteria rather than simply optimizing against the same objective from the opposing side. The degree to which the feedback signal is decoupled from the generative process likely matters a great deal for how “recursive” the improvement truly feels.
In case you’re curious, here’s the current output:
February 15th Story
The chart said Adaeze Okonkwo, fifteen months, and Dr. Rao already knew what was coming before Patty leaned into the doorframe of her office.
“Mrs. Okonkwo is in Room 2. She’s here for the MMR.”
Patty said it the way she’d been saying things lately, with the emphasis shifted just enough to signal trouble. She held a printout against her chest, the state medical board letterhead visible at the top.
“I know what it says,” Dr. Rao told her.
“I’m just making sure you saw the updated memo. ‘Administering non-recommended vaccines without documented clinical justification may constitute grounds for license review.’” Patty read it like a weather report. “‘Physicians are advised to exercise caution consistent with the revised federal schedule.’”
Dr. Rao took the paper. She’d read the original version three weeks ago when the CDC’s advisory committee had moved MMR from “routine” to “optional, parental discretion,” a category that hadn’t existed six months prior. The language was new. The meaning was not.
“She’s my last appointment,” Dr. Rao said. “I’ll handle it.”
Patty nodded and pulled the door shut behind her. Through the wall, Dr. Rao could hear the muffled syllables of a toddler’s voice, the particular pitch of a child discovering the crinkle of exam table paper.
She set the memo on her desk, squared it against the corner, and stood.
Grace Okonkwo sat in the molded plastic chair with her daughter on her lap. She wore a navy peacoat and had her purse on the floor between her feet, both hands occupied with keeping Adaeze from lunging toward the cartoon frog poster. The child had Grace’s wide forehead and her father’s dimpled chin, and she was reaching for the frog with the absolute confidence of someone who had never been told no about anything that mattered.
“Dr. Rao.” Grace’s voice was steady, rehearsed. “I’m here for her MMR. I called ahead. Your office confirmed you still carry it.”
“We do.” Dr. Rao sat on the rolling stool and opened the chart on her tablet. She could feel the shape of the conversation before it started, the way you feel weather change in your knees. “Mrs. Okonkwo, I want to be straightforward with you. The federal schedule was updated last month. MMR is no longer on the recommended list for this age group. It’s been moved to an optional category.”
“I know what they did.”
“If I administer it without a documented clinical justification, my license could be reviewed. That’s new as of this month.”
Grace shifted Adaeze to her other hip. The child grabbed a fistful of her mother’s braids and pulled. Grace didn’t flinch.
“Emeka was three,” she said. “My son. He got measles at his daycare in Silver Spring. A place with a waiting list. A place I thought was safe.” She paused, not for effect but because the next part still cost her something. “He had a hundred-and-five fever for four days. The infection spread to his ears. He lost thirty percent of the hearing in his left ear. He wears a hearing aid now. He’s seven.”
Dr. Rao set the tablet down.
“I am not here to argue about policy,” Grace continued. “I am not confused. I am not hesitant. I want my daughter vaccinated. I’ll pay out of pocket. I’ll sign whatever you need me to sign.”
“It’s not about the money, Mrs. Okonkwo.”
“I know it’s not about the money.” Grace looked at her directly. “It’s about whether you’ll do it.”
The room was quiet except for Adaeze, who had found the crinkle paper again and was tearing a strip of it with great satisfaction. The fluorescent light buzzed at a frequency Dr. Rao had stopped noticing fifteen years ago and was noticing now.
The government shipment of vaccines hadn’t come. It was 1987, and measles was moving through the slum east of her father’s clinic faster than anyone had predicted. Her father drove to a veterinary supply depot in Secunderabad and bought what he could, then spent three days sterilizing, reconstituting, dosing by hand with a chart he’d drawn himself on the back of an invoice. No authorization. No documentation anyone would accept. Fourteen children. No deaths.
He never talked about it. Her mother described it only once, late at night after his funeral.
Dr. Rao looked at Adaeze, who had stopped tearing the paper and was now trying to fit her entire fist into her mouth. She had four teeth, the front ones, and she was grinning around her knuckles.
The supply cabinet was three steps away. The MMR vials were on the second shelf, behind the varicella, exactly where they’d been for twenty-two years.
“Lay her down for me,” Dr. Rao said.
Grace exhaled and eased Adaeze onto the table. The child kicked once, twice, then went still as Grace placed a hand on her belly. Dr. Rao opened the cabinet, pulled the vial, checked the lot number and expiration. She drew the dose into a syringe, flicked it, pressed the plunger until a single bead appeared at the needle’s tip.
She swabbed the child’s left thigh. Adaeze looked at the cartoon frog.
Dr. Rao gave the shot, withdrew the needle, and pressed a cotton ball to the puncture. Adaeze’s face crumpled for two seconds, then reset. Grace scooped her up.
Dr. Rao dropped the syringe into the sharps container, heard it click against the plastic, and wrote the lot number in the chart.