Test Bio Please Ignore
Charlie Sanders
Karma: 98
Self-play is the foundational element of recursive self-improvement — it’s what allowed AI systems to bootstrap to superhuman performance in Chess and Go through effectively unbounded iteration. That said, there’s likely a meaningful spectrum within self-play that depends on the independence and algorithmic sophistication of the evaluating role.
Not all self-play is created equal in this regard. Purely rivalrous, symmetric self-play (like white versus black in Go) represents one end of that spectrum, while asymmetric arrangements with structurally distinct roles (like author and critic) may occupy a qualitatively different position — particularly when the critic brings independent evaluative criteria rather than simply optimizing against the same objective from the opposing side. The degree to which the feedback signal is decoupled from the generative process likely matters a great deal for how “recursive” the improvement truly feels.
In case you’re curious, here’s the current output:
February 15th Story
The chart said Adaeze Okonkwo, fifteen months, and Dr. Rao already knew what was coming before Patty leaned into the doorframe of her office.
“Mrs. Okonkwo is in Room 2. She’s here for the MMR.”
Patty said it the way she’d been saying things lately, with the emphasis shifted just enough to signal trouble. She held a printout against her chest, the state medical board letterhead visible at the top.
“I know what it says,” Dr. Rao told her.
“I’m just making sure you saw the updated memo. ‘Administering non-recommended vaccines without documented clinical justification may constitute grounds for license review.’” Patty read it like a weather report. “‘Physicians are advised to exercise caution consistent with the revised federal schedule.’”
Dr. Rao took the paper. She’d read the original version three weeks ago when the CDC’s advisory committee had moved MMR from “routine” to “optional, parental discretion,” a category that hadn’t existed six months prior. The language was new. The meaning was not.
“She’s my last appointment,” Dr. Rao said. “I’ll handle it.”
Patty nodded and pulled the door shut behind her. Through the wall, Dr. Rao could hear the muffled syllables of a toddler’s voice, the particular pitch of a child discovering the crinkle of exam table paper.
She set the memo on her desk, squared it against the corner, and stood.
Grace Okonkwo sat in the molded plastic chair with her daughter on her lap. She wore a navy peacoat and had her purse on the floor between her feet, both hands occupied with keeping Adaeze from lunging toward the cartoon frog poster. The child had Grace’s wide forehead and her father’s dimpled chin, and she was reaching for the frog with the absolute confidence of someone who had never been told no about anything that mattered.
“Dr. Rao.” Grace’s voice was steady, rehearsed. “I’m here for her MMR. I called ahead. Your office confirmed you still carry it.”
“We do.” Dr. Rao sat on the rolling stool and opened the chart on her tablet. She could feel the shape of the conversation before it started, the way you feel weather change in your knees. “Mrs. Okonkwo, I want to be straightforward with you. The federal schedule was updated last month. MMR is no longer on the recommended list for this age group. It’s been moved to an optional category.”
“I know what they did.”
“If I administer it without a documented clinical justification, my license could be reviewed. That’s new as of this month.”
Grace shifted Adaeze to her other hip. The child grabbed a fistful of her mother’s braids and pulled. Grace didn’t flinch.
“Emeka was three,” she said. “My son. He got measles at his daycare in Silver Spring. A place with a waiting list. A place I thought was safe.” She paused, not for effect but because the next part still cost her something. “He had a hundred-and-five fever for four days. The infection spread to his ears. He lost thirty percent of the hearing in his left ear. He wears a hearing aid now. He’s seven.”
Dr. Rao set the tablet down.
“I am not here to argue about policy,” Grace continued. “I am not confused. I am not hesitant. I want my daughter vaccinated. I’ll pay out of pocket. I’ll sign whatever you need me to sign.”
“It’s not about the money, Mrs. Okonkwo.”
“I know it’s not about the money.” Grace looked at her directly. “It’s about whether you’ll do it.”
The room was quiet except for Adaeze, who had found the crinkle paper again and was tearing a strip of it with great satisfaction. The fluorescent light buzzed at a frequency Dr. Rao had stopped noticing fifteen years ago and was noticing now.
The government shipment of vaccines hadn’t come. It was 1987, and measles was moving through the slum east of her father’s clinic faster than anyone had predicted. Her father drove to a veterinary supply depot in Secunderabad and bought what he could, then spent three days sterilizing, reconstituting, dosing by hand with a chart he’d drawn himself on the back of an invoice. No authorization. No documentation anyone would accept. Fourteen children. No deaths.
He never talked about it. Her mother described it only once, late at night after his funeral.
Dr. Rao looked at Adaeze, who had stopped tearing the paper and was now trying to fit her entire fist into her mouth. She had four teeth, the front ones, and she was grinning around her knuckles.
The supply cabinet was three steps away. The MMR vials were on the second shelf, behind the varicella, exactly where they’d been for twenty-two years.
“Lay her down for me,” Dr. Rao said.
Grace exhaled and eased Adaeze onto the table. The child kicked once, twice, then went still as Grace placed a hand on her belly. Dr. Rao opened the cabinet, pulled the vial, checked the lot number and expiration. She drew the dose into a syringe, flicked it, pressed the plunger until a single bead appeared at the needle’s tip.
She swabbed the child’s left thigh. Adaeze looked at the cartoon frog.
Dr. Rao gave the shot, withdrew the needle, and pressed a cotton ball to the puncture. Adaeze’s face crumpled for two seconds, then reset. Grace scooped her up.
Dr. Rao dropped the syringe into the sharps container, heard it click against the plastic, and wrote the lot number in the chart.
This story was inspired by considering what different possibilities the use of AI models might unlock. One potential area is ludicrously deep technical expertise, of the kind that eclipses the potential of even the most in-depth research. I think of these stories as having “hyperdetails”, details so specific that they transcend the fourth wall with the reader.
I was struck by the disparity in inflation results shown by https://www.visualcapitalist.com/cp/g20-inflation-tracker-august-2025/, with China being a clear outlier in global inflation performance. However, I struggled to take that data and visualize why deflation is generally regarded as economically destructive, or how. I think fiction is an underappreciated way to convey complicated topics, so here’s a story that tries to demonstrate the underlying mechanisms of deflationary spirals, also known as “Involution”.
LLM Disclaimer: LLM assistance was used in drafting this story.
Thank you! I think there’s a lot of value to be explored in increasing peoples’ awareness of AI progress via fiction.
There’s a parallelism here between the mental constructs you’re referring to and the physical architecture of the human body. For instance, each lobe of our brain has been associated with various tasks, goals, and activities. When you take a breath, your Medulla Oblongata has taken in information about levels of carbon dioxide in the blood via pH monitoring, decided that your blood has too much carbon dioxide, and has sent a request to the respiratory center to breathe. But you’ve also got a cerebral cortex that also gets a say in the decisions made by the respiratory center, and those two brain areas negotiate via highly complex, fully unconscious interactions to decide what directive the respiratory center actually follows.
To summarize: you’re now breathing manually.
I do not believe that 3a is sufficiently logically supported. The criticism of AI risk that have seemed the strongest to me have been about how there is no engagement in the AI alignment community about the various barriers that undercut this argument. Against them, The conjecture about what protein folding and ribosomes might one have the possibility to do really weak counterargument, based as it is on no empirical or evidentiary reasoning.
Specifically, I believe further nuance is needed about the can vs will distinction in the assumption that the first AGI to make a hostile move will have sufficient capability to reasonably guarantee decisive strategic advantage. Sure, it’s of course possible that some combination of overhang risk and covert action allows a leading AGI to make some amount of progress above and beyond humanity’s in terms of technological advancement. But the scope and scale of that advantage is critical, and I believe it is strongly overstated. I can accept that an AGI could foom overnight—that does not mean that it will, simply by virtue of it being hypothetically possible.
All linked resources and supporting arguments have a common thread of taking it for granted that cognition alone can give an AGI a decisive technology lead. My model of cognition is instead of a logarithmically decreasing input into the rate of technological change. A little bit of extra cognition will definitely speed up scientific progress on exotic technological fronts, but an excess of cognition is not fungible for other necessary inputs to technological progress, such as the need for experimentation for hypothesis testing and problem solving on real world constraints related to unforeseen implementation difficulties related to unexplored technological frontiers.
Based on this, I think the fast takeoff hypothesis falls apart and a slow takeoff hypothesis is a much more reasonable place to reason from.
My intuition is that a simulation such as the one being proposed would take far longer to develop than the timeline outlined in this post. I’d posit that the timeline would be closer to 60 years than 6.
Also, a suggestion for tl;dr: The Truman Show for AI.
Agreed. A common failure mode in these discussions is to treat intelligence as equivalent to technological progress, instead of as an input to technological progress.
Yes, in five years we will likely have AIs that will be able to tell us exactly where it would be optimal to allocate our scientific research budget. Notably, that does not mean that all current systemic obstacles to efficient allocation of scarce resources will vanish. There will still be the same perverse incentive structure for funding allocated to scientific progress as there is today, general intelligence or no.
Likewise, researchers will likely be able to make the actual protocols and procedures necessary to generate scientific knowledge as optimized as is possible with the use of AI. But a centrifuge is a centrifuge is a centrifuge. No amount of intelligence will make a centrifuge that takes a minimum of an hour to run take less than an hour to run.
Intelligence is not an unbounded input to frontiers of technological progress that are reasonably bounded by the constraints of physical systems.
One of the unstated assumptions here is that an AGI has the power to kill us. I think it’s at least feasible that the first AGI that tries to eradicate humanity will lack the capacity to eradicate humanity—and any discussion about what an omnipotent AGI would or would not do should be debated in a universe where a non-omnipotent AGI has already tried and failed to eradicate humanity.
In many highly regulated manufacturing organizations there are people working for the organization whose sole job is to evaluate each and every change order for compliance to stated rules and regulations—they tend to go by the title of Quality Engineer or something similar. Their presence as a continuous veto point for each and every change, from the smallest to the largest, aligns organizations to internal and external regulations continuously as organizations grow and change.
This organizational role needs to have an effective infrastructure supporting it in order to function, which to me is a strong argument for the development for a set of workable regulations and requirements related to AI safety. With such a set of rules, you’d have the infrastructure necessary to jump-start safety efforts by simply importing Quality Engineers from other resilient organizations and implementing the management of change that’s already mature and pervasive across many other industries.
As someone that interacts with Lesswrong primarily via an RSS feed of curated links, I want to express my appreciation for curation when it’s done early enough to be able to participate early in the comment section development lifestyle. Kudos for quick curation here.
How else were people thinking this ban was going to be able to go into effect? America has a Constitution that defines checks and balances. This legislation is how you do something like “Ban TikTok” without it being immediately shot down in court.
It’s verbatim. I think it picked up on the concept of the unreliable narrator from the H.P. Lovecraft reference and incorporated it into the story where it could make it fit—but then, maybe I’m just reading into things. It’s only guessing the next word, after all!
Just to call it out, this post is taking the Great Man Theory of historical progress as a given, whereas my understanding of the theory is that it’s highly disputed/controversial in academic discourse.
It would by definition not be bad thing. “Bad thing” is a low-effort heuristic that is inappropriate here, since I interpret “bad” to mean that which is not good and good includes aggregate human desires which in this scenario has been defined to include a desire to be turned into paperclips.
The ideal scenario would be for humans and AIs to form a mutually beneficial relationship where the furtherance of human goals also furthers the goals of AIs. One potential way to accomplish would be to create a Neuralink-esque integrations of AI into human biology in such a way that human biology becomes an intrinsic requirement for future AI proliferation. If AGIs require living, healthy, happy humans in order to succeed, then they will ensure that humans are living, happy, and healthy.
See, this is the perfect encapsulation of what I’m saying—it could design a virus, sure. But when it didn’t understand parts of the economy, that’s all it would be—a design. Taking something from the design stage to the “physical, working product with validated processes that operate with sufficient consistency to achieve the desired outcome” is a vast, vast undertaking, one that requires intimate involvement with the physical world. Until that point is reached, it’s not a “kill all humans but fail to paperclip everyone” virus, it’s just a design concept. Nothing more. More and more I see those difficulties being elided over by hypothetical scenarios that skip straight from the design stage and presuppose that the implementation difficulties aren’t worth consideration, or that if they are they won’t serve as a valid impediment.
There exists a diminishing returns to thinking about moves versus performing the moves and seeing the results that the physics of the universe imposes on the moves as a consequence.
Think of it like AlphaGo—if it only ever could train itself by playing Go against actual humans, it would never have become superintelligent at Go. Manufacturing is like that—you have to play with the actual world to understand bottlenecks and challenges, not a hypothetical artificially created simulation of the world. That imposes rate-of-scaling limits that are currently being discounted.
I’m a firm believer in avoiding the popular narrative, and so here’s my advice—you are becoming a conspiracy theorist. You just linked to a literal conspiracy theory with regards to face masks, one that has been torn apart as misleading and riddled with factual errors. As just one example, Cochrane’s review specifically did not evaluate “facemasks”, it evaluated “policies related to the request to wear face masks”. Compliance to the stated rule was not evaluated, and it is therefore a conspiracy theory to go from an information source that says “this policy doesn’t work” and end up with the takeaway “masks don’t work”. As other commenters have pointed out, it is physically implausible for facemasks to not work if they are used correctly.
The definitionally correct term to use for you is “conspiracy theorist” so long as this is a thing that you, after conducting your own research, have come to believe. Take your belief in the facemask thing as concrete evidence that your friends and family are correct and that you are indeed straying down the path of believing more and more improbable and conspiratorial things.
Agreed. There’s really a continuum here, where the rate of self-improvement can range from sublinear to fully exponential. Processes that lack a mechanism for feeding improvements back into the system’s underlying intelligence are unlikely to achieve exponential growth; they’ll tend to plateau or progress linearly at best.
One takeaway is that an effective regulatory regime aimed at limiting fast-takeoff risk might focus precisely on this distinction — restricting processes that iteratively improve at greater than a linear rate.