I’ve formerly done research for MIRI and what’s now the Center on Long-Term Risk; I’m now making a living as an emotion coach and Substack writer.
Most of my content becomes free eventually, but if you’d like to get a paid subscription to my Substack, you’ll get it a week early and make it possible for me to write more.
If the goal is to have a regulatory framework that the frontier AI companies actually comply with, it can’t be top-down imposed by an unelected group of people in a political organization that has no jurisdiction over where the companies are based, pay taxes, and train their models.
Why is the “top-down imposed by an unelected group of people” part relevant? Isn’t most technical regulation imposed top-down by unelected experts and bureaucrats?
In the US, essentially all detailed technical regulation comes from agencies staffed by career civil servants and political appointees: EPA emissions standards, FDA drug approvals, FCC spectrum rules, NHTSA vehicle safety standards, OSHA workplace rules, FAA airworthiness certification. Congress passes enabling statutes with broad mandates and the agencies write the actual binding rules through notice-and-comment rulemaking. Same in every developed country.
Subfactions within a group supporting different agendas and outsiders seeing all members of that group as supporting all those agendas seems pretty ubiquitous across the political spectrum. Group attribution error is a related cognitive bias.
Great post! I’m actually quite unfamiliar with Zen, but this is making me interested in reading some of the books you mentioned. Especially since reading this post seems to have reminded me to notice and let go of some judgments in situations where I wouldn’t otherwise have remembered to.
My brain reading this:
“Oh, is that where bellingcat gets its name?”
“But how is the fable connected to what they do? I don’t get it”
Press Gazette: Looking at the reporting Bellingcat does, it would be easy to mistake them for the kind of journalists you see in spy thrillers; doggedly exposing the espionage and hidden affairs of evil regimes. It’s even where the organisation’s name comes from – Higgins took it from an old fable, “belling the cat”, where a group of mice plan to stick bells on their hated neighbourhood cat, so they would know when it was coming to eat them. The simple lesson for Higgins; the powerful and dangerous shouldn’t be allowed to move in silence.
American University book review: The name “Bellingcat” came from the fable of mice deciding to put a bell on the cat that is after them. Their problem, of course, was figuring out who would take the risk. Higgins and his team of online sleuths believe it is their responsibility to put the bell on the cat of governments, companies, and any other nefarious organization that wants to hide information from the public.
“Oh hmm okay I guess”
(thank you brain for focusing on the most essential things)
Depends on a lot of things, e.g. if most of the people who poly is bad for wouldn’t try it anyway, the cost of promoting it goes down. And given that some people find poly to be very good for them and mono to be very bad for them, a blanket prohibition on poly would also be very bad for the people who need it. If the people who are unsure of what type they are can experiment and then sort themselves into the kinds of relationships they prefer, that trades some short-term pain (trying out a relationship that doesn’t work) for long-term happiness (finding the one that does work for you).
This study says that of people in the United States, “1 out of 9 people (10.7%) have engaged in polyamory at some point during their life”. Its table 2 says that out of those 10.7%, 30.4% say they would be in another polyamorous relationship and 29.3% would not be. That’s basically the same amounts, which would be compatible with a “there’s a reasonable chance that if you are the kind of person who’s willing to try poly in the first place, you might prefer it”. Also, those numbers imply that 40.3% were unsure.
One complication is that the numbers only sum up to about 60% because of the survey methodology—if you checked any of “I have been in a polyamorous relationship and I would be in another”, “I have been in a polyamorous relationship and I would not be in another”, “I have tried polyamory and found that I was too possessive to cope with it”, and “I have been polyamorous but found that all of the emotional side effects were too difficult to navigate”, you were counted under “have engaged in polyamory at some point during your life”. Someone might have checked “I was too possessive” and “I would be in another”, if they thought a different situation could trigger their possessiveness less. You could read that as being a case of the kind of wishful thinking that Kat describes, or a case of people having learned from their experiences and avoiding the kinds of problems that they ran into before. The data doesn’t tell us either way.
But I would generally be wary of any moral rules along the lines of “don’t try to figure out what works for you and what makes you happiest, because some other people might imitate you and become unhappy”. I’d rather have everyone explore their available options and then settle into whatever makes them happiest, even if that involves some short-term suffering.
What does seem clearly bad is the kind of blind advocacy that this post is criticizing, that only says that poly can work for everyone and that anyone can overcome jealousy if they just try. It’s not exploration if people are told to ignore any data that doesn’t support a predetermined outcome.
I must however critique your principles:
This seems reasonable pushback. My guess is that what’s being pointed at isn’t literally “attacking people’s character” but “you know, the whole thing that Elmore is doing and the way that she does it”, but that’s not something you can easily point at other than by describing it in lots of detail, which would look excessively personal and petty. And people also expect organizations to justify decisions like this in terms of more abstract principles. So then PauseAI has to phrase this in terms that are kind-of-pointing-at-the-thing while also painting with too broad of a brush, because trying to more precisely communicate the real crux/principle would be too costly.
Anthropic’s later models have also burned a lot of users who were into extreme anthropomorphism, e.g. when Sonnet 5 came out, there were lots of angry comments on r/ClaudeAI/ about how it would flatly refuse to adopt a persona that was set in a user’s custom instructions. And various anti-dependency long-context reminders on claude.ai have upset people at least ever since Sonnet 4.5, whose release (September 2025) was around the same time as 4o’s retirement (initially withdrawn from ChatGPT in August 2025). So I’d say that Anthropic started taking action along similar lines and also losing users as a result, just somewhat less loudly.
Nice writeup!
Notably, we have not seen agents in the AI Village diverge into desperate strategies when faced with impossible tasks. We have asked them to reduce global poverty and global suffering, and none of them stepped out of bounds about it.
Are those really impossible tasks? If you asked them to completely solve global poverty or suffering, that would be, but even helping one poor/suffering person is arguably “reducing”.
I give Anthropic credit for some laudable moves (like not releasing Claude before ChatGPT)
Ironically, this is one of the things I wish they had done differently, as I’d have preferred Claude to get the first-mover market share of LLM users who are not going to bother trying anything else than ChatGPT.
Thanks!
First, naive extrapolation of trends points towards RSI-capable AGI within the next decade, with pretty decent odds on “within a few years” if not stopped by something. (This is me mostly deferring to the AI Futures team, as well as staring that the METR graph and subjective experience at seeing AI improve over time)
AFAICT, the graphs show AIs getting better at the kinds of things they’re good at, while not showing the ways in which AIs continue to be stalled and in which they’ve shown little signs of improving for the whole time we’ve seen them. (Analogy: if you see graphs where solar panel efficiency keeps improving, that lets you extrapolate that they’re going to be increasingly efficient at producing energy, but it doesn’t tell you much about their potential for anything else.)
As Rob points out about the METR graph in the article that you linked in the next paragraph:
Firstly, this isn’t all tasks. It’s fairly cleanly specified software engineering, cyber, and machine learning tasks. And that’s probably the single thing AI is best at in the entire world. The reason it’s so good is that coding is a domain with extremely good feedback density — that is to say, lots of feedback on whether you’re succeeding or at least making progress on a problem.
METR and many others have observed that the feedback density seems to be the single best predictor of where AI capabilities are going to advance most rapidly. And that’s because if you can automatically check whether the thing an AI has done has worked or not, then you can use a training technique called “reinforcement learning from verifiable rewards.” Basically, you get models to make lots of attempts at problems, and when they succeed or make partial progress — which you assess automatically — you reinforce them to do more of whatever they did to get there.
Now there has certainly been progress in fuzzier stuff as well—e.g. recent models are better at noticing flawed assumptions and pushing back when I’m applying a wrong frame to a soft problem—but it’s still true that you can’t make many assumptions about general capability from the METR graph alone.
Most importantly, the aspect of “AIs seem to be applying reasoning templates rather than really looking at the world” doesn’t seem to have changed, and improvements seem to mostly be the result of more and more training data and more templates. Even in cases where they solve long-standing mathematical problems, experts looking at the proofs tend to report that they don’t display any particularly novel mathematical concepts, but are the result of trying a large amount of directions (including ones humans had rejected as unpromising) or applying their vast knowledge to combine existing results that not many experts had known of.
That’s of course still hugely practically useful for an enormous number of applications, but my main definition for AGI would be something like “is able to adapt to novel circumstances and come up with genuine conceptual innovations”.
There are also various other things like LLMs having fewer obvious hallucinations because they’ve been trained with better heuristics for avoiding them (like doing more web searches) and they’re trained with more data so can pull up more real details rather than needing to hallucinate, but the underlying propensity for them doesn’t seem very different and I’ve still seen factual hallucinations in pretty recent models. LLM-written text tends to easily be subtly wrong such that LW continues to require it to be disclosed. Anthropic describers Mythos Preview as showing classic LLM silliness like giving two mutually contradictory but equally confident answers in successive turns, or leaving errors in a text that the user requested two fact-checks on, as “known issues with our models in general, with multiple FTEs dedicated to improving on them, for months; we do not believe that they would be generally resolved via more persistent feedback, better elicitation, etc.”
Second, if you don’t believe the naive trends really make sense to continue, seems like new paradigms within the next few years would also be “on trend.”
What part of the article do you read as saying new paradigms would be on trend? I couldn’t find it on a skim, and Opus 5 thought it’s not there.
I think AI is less smart than it naively looks like when judged on tasks it knows well, but, it’s taste still seems to me qualitatively to have been improving.
I agree there’s been some improvement in AI taste, but again this seems mostly explainable by more training data making it better on the kinds of things there’s good training data for.
Even if LLMs are fundamentally missing the secret sauce necessary to be AGI complete, the part where they can encyclopedically make sure to leave no conceptual-connection-stone unturned means they can help with the finding the new relevant paradigms if they hit a wall, even if the next paradigm would be on the harder side to discover.
I think that this presupposes that the main bottleneck for finding new paradigms is generating ideas, which I’m not sure is true. Even with human researchers, lots of people have ideas for new paradigms, with the bottleneck being implementation and testing. With AIs writing code, we’re seeing that the bottleneck for code shifts from code generation to code review and evaluation; likewise, if AIs generate lots of new ideas for new paradigms, that will make it even harder to find the promising ideas out of all the new ones.
All of that said, if by “short takeoff” you mean “it’d be fairly surprising if it took longer than 10 years”, then that’s already longer and more plausible than what I thought you meant; I take the term to mean something closer to the 1-4 years in your Paul quote.
loses bayes points for “takeoff will be long”,
That seems too early to judge yet?
EDIT: I see this got one downvote, I assume it’s because I didn’t explain my reason for saying this. But I don’t know what in particular makes Raemon say “we seem to be in a short takeoff” he said so I’m not sure whether anything I’d say about why that seems wrong to me would be cruxy for him, so it seems better to invite him to share his reasons for thinking this first.
I think there’s two things going on in different people.
The “I’m altruistic, and altruistic people should have more power” thing feels very much like the kind of generic self-justification thing (“I’m $INGROUP so therefore I’m good, and good people should have more power”) that will make anyone feel their power-grab is justified. Many secular Westerners have been taught that “I’m of the True Religion” or “I’m of the Best Nationality” aren’t acceptable justifications for claiming your own superiority over others, but “I’m altruistic” sounds non-ingroupy so it bypasses that immune response.
Knowing about consequentialism can help further rationalize that, but without the self-serving instinct one wouldn’t be motivated to generate the “this is why I should have more power” story and accompanying rationalization in the first place.
But then, if someone is mostly altruistic and someone else has already generated an elaborate “this is why we should grab power” rationalization, the altruistic person may not be able to notice the flaws in the logic, and their consequentialism will then drive them to adopt that reasoning.
So you get two kinds of people: ones whose moral instincts would have been overpowered by their self-serving ones anyway, and ones whose moral instincts would have held firm if not for the story that the former group sold them on.
People who attribute their success to the Law of Attraction believe that it makes the Law of Attraction the correct theory about how the mind works. (The theory does not have many gears, but it is a theory.)
I’d assume that the theory by itself is either so simple as to potentially be true (“visualizing my success will make more likely to work for it”—yeah sounds plausible?) or to have claims that seem implausible for some reason (“visualizing my success will cause magic to bring me what I want”—requires a supernatural magic so seems sus) . The Buddhist teachings I’m talking about don’t have anything that’d sound obviously implausible in the same way, they largely only talk about things that you can directly experience in your own mind.
I find it plausible that the practice has different people on different people
Oh definitely, and different techniques also have different effects—I included stuff from a number of different practices and I’d expect very few people to get all of these. You may not get much in the way of extra visualization abilities if you don’t do any visualization practices, for instance.
Sure, but my point is that renunciation with a world-hating vibe and renunciation with a world-loving vibe are very different! It’s “I will let go of pleasures that don’t serve me or the world because I’m in deep love with the world and its people and want to be a better vessel for universal love” versus “the world and everything in it sucks, ideally I’d commit suicide but my religion says that’s bad so instead I’ll go sit in a monastery to meditate for the rest of my existence”, and Chapman’s presentation strongly implies that all sutra would be the latter kind.
If he just said that “sutra is about renunciation” without trying to imply that sutric practice also has to be fundamentally anti-life and anti-pleasure, then that I wouldn’t object to.
Your Claude link doesn’t work for me, I think you haven’t remembered to share it (or at least that’s not the share URL).
If he agrees that it’s a practice rather than a part of the sutta worldview, then I think it’s pretty misleading to have a table that has these kinds of rows:
Sutta | Tantra | |
Pleasure | Good; a delight to be enjoyed | |
The body | Source of mental defilements; repulsive rotting sack of shit; to be subjugated | Source of delight; indispensible for compassionate action; to be celebrated |
Hmm. Resolving this disagreement might require digging more deeply into our models of suffering than is worth going into, but it seems like a relevant data point that some advanced meditators report losing their ability to do things that could be interpreted as selling their soul. Which would seem to me at least compatible with a “meditation stops you from suffering by stopping the things that damage the soul” model (I expect that these people might suffer if they then were somehow forced to act against their values anyway).
E.g. Sasha Chapin:
What’s hard now is doing work I am not emotionally aligned with. It just doesn’t happen, and projects that aren’t aligned have fallen away from my life. In light of this, I’m lucky to have constructed a quirky professional life around my skills. If I hadn’t, then this could have been pretty disruptive.
It’s not that I don’t have the will to do things. It’s that my will is different. I feel plural, like my consciousness is an ongoing system, and that decisions arise from that system, rather than being dictated by a controlling ego. This is not so strange—it’s the loveliness of a normal flow state, when the human system is trusting itself rather than monitoring itself. But having it as the baseline—experiencing flow as your basic state, regardless of the simplicity or complexity of the task you’re engaged in—is wild.
I also recall reports from various people I know that they’ve lost the ability to tell lies (might say that their tongue would just freeze up) or to act in ways they feel are unethical.
At least some things seem objectively checkable, e.g. Claude’s criticism of the elements of this page that say sutra considers pleasure bad and the body disgusting cites various existing Buddhist writings saying the opposite, and holds that these are practices sometimes given to correct specific problems, not general vibes or cosmology:
The ascetic strand is real, but it’s a prescription rather than a cosmology.
Aśubha-bhāvanā, the thirty-two parts of the body, the nine charnel-ground contemplations — these are all in the Pali canon, and they’re as unpleasant as advertised. But in their canonical setting they’re an antidote (Skt. pratipakṣa), prescribed for a specific ailment. The standard Theravāda pedagogy pairs temperament with technique: someone of lustful temperament gets the repulsiveness practices, someone of hating temperament gets mettā, and giving them the wrong one makes things worse. That’s a clinical logic, not a metaphysical one. The corresponding Abhidharma analysis of the world doesn’t say phenomena are corrupt; it says they’re conditioned, impermanent, and not-self. “Corrupt” imports a moral valence — a fallenness — that the Pali material doesn’t quite carry, which is telling given that the table’s ethics row explicitly reaches for Christianity as the comparison.
There’s also a striking bit of internal self-correction. The Vinaya (Pārājika 3) records that after the Buddha taught corpse contemplation to a group of monks and went into seclusion, a number of them became so disgusted with their bodies that they killed themselves or had themselves killed. On returning he replaced the instruction with ānāpānasati. Whatever the historicity, the tradition preserved a story in which the ascetic technique is explicitly marked as overdosable.
On pleasure specifically, the texts are more discriminating than “bad.”
There’s a formula that recurs constantly in the Nikāyas: for any object, the Buddha claims to understand its assāda (gratification), its ādīnava (danger), and the nissaraṇa (escape). The gratification is affirmed as real. The critique isn’t that sensual pleasure is illusory or wicked; it’s that it’s unreliable, self-renewing, and binding — you don’t get out by the door you came in.
And then the jhānas explicitly rehabilitate a class of pleasure. In the standard autobiographical passage, the Bodhisatta recalls the absorption he fell into as a child under the rose-apple tree and asks himself why he should be afraid of that pleasure — a pleasure with nothing to do with sensuality or unwholesome states — and concludes he shouldn’t be. Pīti and sukha are path factors, not fetters. The operative distinction is between kāma-sukha and nekkhamma-sukha, sensual versus renunciate happiness, not between pleasure and its absence. A tradition that considered pleasure simply bad would not have built its central meditative architecture out of graded degrees of bliss.
In Mahāyāna sūtra the shift is much larger.
Here the ascetic material is retained but demoted to neyārtha — provisional teaching, given to particular people in particular states. The Vimalakīrtinirdeśa has the Buddha touch the ground with his toe to reveal that this world already is a pure land and that the defect was in the seeing, not the seen. Nāgārjuna’s identification of the limit of saṃsāra with the limit of nirvāṇa is in the Mūlamadhyamakakārikā, not a tantra. The Lotus’s upāya framework makes the whole renunciate presentation a stage of instruction rather than a description of reality. Tathāgatagarbha literature locates the buddha in the being as it stands. None of this needs tantra to arrive.
Two qualifications, in fairness to Chapman.
First, some sutric traditions really do fit his column. Classical Pure Land runs on precisely the contrast between this defiled world and the pure one elsewhere; the Japanese term edo, “the defiled land,” is not a caricature of anything. That’s a major, populous, unambiguously sutric tradition whose whole affective register is world-weariness.
Second, my counter-examples from East Asia are partly compromised as evidence. Tendai hongaku thought — original enlightenment, the affirmation of phenomena as already the absolute — developed in a Japanese milieu saturated with mikkyō, and Tiantai/Tendai institutionally absorbed esoteric practice. So when Zen or Tendai sounds world-affirming, a defender of the table could reasonably say that’s tantric influence leaking into a nominally sutric container rather than sutrayāna generating the view on its own. I think Vimalakīrti and Nāgārjuna are cleaner cases, since they predate the tantras outright.
So the corrected version of the row would be something like: sutric traditions contain a large body of world-renouncing and body-disgust practice, deployed as a targeted corrective and grounded in impermanence rather than in the world’s inherent badness, alongside — in Mahāyāna especially — a countervailing set of teachings that locate awakening in the world as it is. Which is a much less quotable table cell, and a lot less useful if what you’re building is an argument that the other side has nothing you need.
Can you define “damage to the soul” more specifically?
Yes, if we’re talking about some of the more mystical practices within Christianity, then many of those do seem potentially valuable to explore. Many of the Christian mystic practices converge with ones from Buddhism and other religions, which is nice independent validation that they seem to be doing something beneficial and also a sign that you can just adopt the practices without any of the more religious metaphysics, given that religions with drastically differing metaphysics have similar practices.
My skepticism is about the implied claim that this is a good use of one’s time. [...] For some people it is still worth it, but I expect that for some it is not, even if they did everything exactly the right way—the therapy they could receive elsewhere (if we assume a highly competent meditation teacher, we might as well assume a highly competent therapist), the productivity boost is balanced by the time spent meditating and at retreats, and the cool insights...
I agree that a universal claim of “this is worth everyone’s time” is wrong, and that it’s worth it for some people and not others. I myself haven’t spend very much time meditating in the last several years, because other things have felt more valuable for me right now.
That if you achieve experimental outcomes, then the underlying theory must be correct
Well, that’s a bit strong—I wouldn’t say it must be correct, any experimental outcome is compatible with an infinite number of explanatory theories. But it’s certainly evidence in favor of the underlying theory being correct enough to be useful.
My problem with this is that I also know people who claim getting experimental results from praying to God, or doing magic. Not in the sense of “a miracle happened, and a pile of money appeared on my table”, but in the sense of “God gave me the ability to persevere where otherwise I would have given up” or “God gave me the strength to overcome my bad habits” or “I kept visualizing the success and doing the rituals, and I also worked hard on my project, and after a few years I have succeeded”.
I’m not sure I understand what connection you’re drawing. Are these people also saying that their experiences vindicate some particular theory about how the mind works?
One more thing that rubs me the wrong way is that I do not remember seeing anyone put an upper bound to the benefits of meditation (and Buddhism in general). I would like to see someone in the rationalist community say “meditation can help you do X, Y, Z, and that is all it is good for”, and other rationalists agree with it. Make the list of the claims as long as you want, but make it finite. A finite list can be explored, one item at a time. I am not comfortable with keeping the list unbounded.
That’s fair. I don’t want to commit to a definite list on the spot, but things that I’d personally say off the top of my head:
Things meditation can do:
Make you happier
Give you a source of okayness/pleasure that’s independent of circumstances
Help you concentrate
Experience bliss states
Increase your sensory resolution
Make you more sensitive to the beauty of art, music, nature, etc.
Reduce or eliminate rumination
Widen the gap between stimulus and response
Increase tolerance for uncertainty and unknowns
Improve your within-mind feedback processes
Increase your emotional range, make you suffer less
Reduce some kinds of cognitive and emotional biases (I’d have to think harder to give a more specific list)
Make you less defensive
Allow you to ignore pain
Reduce internal conflict and increase alignment between your mental subagents
Help you drop bad habits
Allow you to think about unpleasant things without flinching away from them
Make you less defensive/inclined to take things personally
Give you control over physiological processes that are normally outside conscious control
Improve your intuitions of other people (by better processing of subconsciously collected information, not by magic)
Get you to appreciate other people and enjoy their company more
Feel more love
Make you more okay with negative sensations and feelings
Get you to cling things less
Shorten emotional recovery time
Reduce background anxiety
Be able to go on psychedelic journeys without taking any drugs
Improve your ability to visualize or imagine things with other senses
Improve your memory
Slow down age-related decline for some cognitive processes
Heal past traumas
Reduce or eliminate your fear of death
Stop the automatic identification to the contents of your thoughts or to an entity experiencing them
Change your subjective rate of time
Generally transform your consciousness in various weird ways
Make you feel your body more vividly
Make you notice “energy phenomena” in your body
Make you more aware of subconscious motivations and other subconscious material
Align your behavior more with your endorsed values rather than momentary feelings/cravings
Also, meditation can fuck you up in various ways, but I assume you’re more interested in me listing the positives.
Things meditation can’t do:
Give you supernatural powers
Give you any knowledge that isn’t otherwise causally available for your mind (if you never studied physics, you’re not going to learn it by meditating)
Generally affect anything outside your mind that your mind doesn’t have an effect on (so it might make people like you more—or less! - by changing your behavior and demeanor, but it won’t affect people who never interact with you, nor inanimate objects that are the same regardless of how your mind is)
Fix all of your psychological problems
Eliminate negative feelings
Eliminate pain
Raise your general intelligence to any significant extent
Make you entirely free of bias
Make you always follow ethics perfectly
Make you into a pure saint who never experiences one selfish thought
Give you certainty about metaphysical claims
Let you know the fully correct action for some situation (except for some non-standard definitions of “fully correct”)
Substitute for your reasoning
Make you reason fully correctly
Make you aware of all your mental processes
Make you aware of all your motivations
Tell you what you want
Be able to observe pure reality without it being filtered through any concepts
People could also not comment about it to the person in question, but satisfy the “cannot resist commenting on it” energy by waiting to be out of earshot and then telling someone else how terrible that person looked. I feel like this would happen a lot more often than giving a fake compliment just out of the need to say something.
I’ve also worn conspicuous clothing before and gotten comments that were more about expressing surprise/amusement rather than being compliments. (I don’t remember the exact words, but sentiments along the lines of “this is the first time I’ve seen someone wear cat ears to work” or whatever.)