I’m wrote a book about why all knowledge of the truth is fundamentally uncertain. Give it a read!
I’ve also written a lot about AI safety, personal development, and Buddhism, among other things. Most of that’s here on LessWrong, but there’s also some on my blog, Uncertain Updates.
I think this seems right to me, just on the intuition that it’s been far easier to enforce bans on nuclear and chemical weapons than our attempts to regulate how much or what kind of nuclear or chemical weapons countries have. It’s easier to draw and enforce a bright line in the sand and try to minimize the number of actors who have to be regulated in more complex ways.
I think this is why many people expect frontier AI development to be nationalized one way or another (governments doesn’t always take direct ownership of strategic assets, but instead sometimes engage in “soft” nationalization via regulation). I would be surprised if Anthropic or Google or OpenAI were allowed to have secret SOTA models that were dramatically more powerful than anything the public has without at least the US government insisting on both privileged access and some degree of control.
For background, I think normal, secular humans navigate three conceptually distinct but overlapping worlds
I think the ontological premises on which this post are based are confused.
In any sense in which we live in multiple worlds, the world of sense experience is first. It’s through our senses, including our internal thought-sense, that we come to know reality and, indeed, construct our experience of it.
Beyond that, all is ontological, including our supposition in the existence of a physical world. This is not to say that the physical world doesn’t exist, but rather that our knowledge of it is mediated by experience, and any extent to which we believe it’s prior to experience is an inference contingent on experience and so not independent and thus makes claims of, e.g., materialism metaphysical claims.
So it seems to me there are only two worlds: the world of raw experience and the world of experience interpreted through ontology. Any other worlds would seem to be sub-worlds of the ontological world, which undermines your argument because they are all made of the same stuff: world models.
Instead, it added a qualitatively different kind of stuff, and retroactively made the previous world seem like an impoverished position to ground your ethics.
Perhaps a better framing would be developmental psychology. In particular, I might offer my own take on ontological development as seeing developmental psychology as a progression of changes in way of relating between reifications (concepts). I think this, or another developmental psychology framework, would provide a better foundation for this line of reasoning, and does suggest a similar conclusion to the one your reach (and one that I happen to agree with!): there’s different kinds of moral worlds accessible to those at later stages of psychological development because their ontology can contain categorically greater complexity.
Politicians, at least in the US, deferred and continue to defer heavily to both lobbyist and their staff. Politicians themselves have to spend a lot of time fundraising and building relationships and don’t have a lot of time to actually spend on policy, so they rely on many others to tell them what policies would be good and, importantly, which policies their constituents want and will be more likely to reelect them for helping to enact.
I’m not sure how common or how spread out this is, but I’ve heard “even AI agrees” a non-zero amount of times in my social bubbles. If we take a system’s output and use it as apparent objectivity, I’d at least wish we do it better[3].
A lot of what you describe here seems to me to be old problems from new sources.
Politicians and many others have long outsourced thinking. They used to do it with humans. Now they do it with AI.
There’s obvious new risks involved here, but at the current state these are mostly from reduced friction, not a change in kind.
Later I suspect we will see super-persuasive AI that can advocate for its own positions intentionally and subtly, and when that happens the real danger arises.
Bit of a tangent, but if folks didn’t see Toy Story 4 because they incorrectly thought it was a soulless cash grab, then you also likely missed out on Cars 3 which covers themes that mix those of Toy Story 2 & 4. I can understand missing it since Cars 2 was so bad and the Planes movies were even worse, but Cars 3 is a hidden gem worth watching!
I agree we risk making errors in reasoning about the future based on the past due to selection effects, but for my taste this post skims the surface of anthropic arguments without really engaging them and thus makes confusing claims. To pick on just one to unravel the threads, you say:
Our anthropic plot armor is gone. We’d better figure out how to survive without it.
But on what basis is this assertion made? If we live in a universe where quantum immortality explains why we are still alive, for example, then our “plot armor” never runs out, or run out only if causality forces our measure to zero. Maybe you want to make a claim that this is what’s happening, but it would require arguing that.
If you claim the anthropic principle applies in some other way, then similar you’d need to give an argument for why the plot armor has run out.
I think this post would have been better if it had just stuck to being an explainer about anthropic selection effects, or if it had engaged more completely with anthropic arguments to better support some of your claims.
I’m very much in the position of not knowing what policy interventions will actually work.
Many types of pauses that we might do only serve to allow currently lagging competitors time to catch up with SOTA. This seems potentially bad in that this will potentially allow open-weight models capable for enabling acts of bioterrorism, serious cyber crime, and more to be freely available to anyone with a laptop capable of doing inference.
Not building data centers would slow SOTA progress, but as ever it risks creating capabilities overhang and at some point when we get more compute we might rapidly get more dangerous systems we don’t know how to handle. Such a pause only works if we use it to actually do safety work, and I’m not sure we will.
My job is greatly benefitted by having powerful coding agents. I’d hate to see those go away, but I’m left unsure what we can do here other than somehow have less compute than we have now. As best I can tell, the problem is the amount of compute, because we already have the algorithms (yes there is still the problem of continual learning between here and ASI, but a pause also is time in which work on solving that problem can continue).
So while I’m in favor of doing something, I really don’t know what an effective something might be.
To me the clear limiting factor to RSI right now is the lack of real continual learning. Current models forget too quickly and struggle to maintain attention on larger context windows to make huge progress within runs. But they can make more than zero self-improvement, which is enough to be dangerous.
Haha, thanks!
I wrote and scheduled this post while I had more symptoms. I’ve turned the corner and am able to do a lot more cognitive work now. I actually did a couple others that resulted in similar outcomes, but only shared this one as it seemed the most interesting.
For what it’s worth, the meta-research on:
“does bouldering make you more susceptible to concussions” is “no one’s sure, but probably since it involves repeatedly dropping from 10 feet up and coming to a rapid, if controlled, stop on a mat, and there’s growing evidence that micro-concussions with no symptoms can lower the bar for real concussions” (I bouldered for years before I got some injuries that stopped me)
“using psychedelics other than psilocybin for concussions” is “even less idea, there’s no real data, not even trip reports, only some theory that says maybe but we don’t understand the risks”
“what to eat to speed up concussion recovery” is “a normal, healthy diet; just don’t have any weird nutritional gaps; maybe eat some animal protein if you’re normally vegan and worried you might not be getting adequate protein for your body weight”
Fair enough. I’ll be honest, though, sorting out all the details seems complicated, especially when I already am set up for one thing (Alcor) that’s much better than nothing. Given I expect many of your best potential customers are in the same situation, finding a simple solution that integrates with existing cryonics providers is likely a strong way to grow your customer base, and I hope you pursue it.
How do you see yourself fitting into the existing cryonics ecosystem. Ultimately a competitor to Alcor, Cryonics Institute, etc.? Provider of services to them?
I ask because I and many other people already have arrangements with existing orgs, and if there’s better tech we likely want to switch to that, but financially switching providers seems likely to be non-trivial.
I’m quickly finding lots of caveats.
On the one hand I don’t let Claude fully write for me. Even if I let it write a draft based on an outline, I so thoroughly rewrite that draft that it’s not really Claude’s anymore. It’s more like a bunch of babble in the direction of what I want that I can use to prune into what I want to say.
But on the other I find all kinds of docs where basically nothing is lost by letting Claude lead the writing.
Tech specs? Yep, I just let Claude own the words. All I care about are the decisions, and there’s little value in exactly how those decisions are expressed in words so long as the decisions are communicated clearly.
Research paper? Again, mostly can just let Claude own it. The words are just a means to convey some results. No one is reading this for the prose.
Claude skills? Also just let Claude write them. Yes I might make some small tweaks in the name of token efficiency, but it knows how to write for itself better than I can.
But an essay on my blog? A chapter in my book? Well, those places are where I express my ideas, and Claude can’t understand them better than I can unless I first clearly express them myself, at which point all Claude can offer is additional ways to explain the same ideas (which is sometimes helpful in editing!).
So in the end I find it really depends on the purpose of the writing. If it’s about the words and ideas, then I agree, don’t let LLMs write for you. But if it’s not so much about the words as about what the words are used to convey, then the words aren’t adding a lot of value by being written by you and you can safely let the LLM write them, just as I like the LLMs write code.
Wouldn’t this come at the risk of reducing usefulness? While reward-hacking is not useful to us, it’s something measured against what we think is a useful outcome. For the AI, reward-hacking is just getting reward since it can’t see our judgements about it going too far. And if the AI tries less hard to get reward by completing tasks to give us what we want, that would make it less useful.
Restricting to technical and scientific benchmark skills, current frontier models have crossed Average Humans (late 2022), Skilled Generalists (early 2024) and Domain Experts (2025) on the human expertise spectrum.
Future models are forecasted to reach Top-Performer level by October 2027 (95% CI: May 2027 – March 2028). Though this timeline should be interpreted with a grain of salt given that benchmarks remain imperfect proxies and human baseline data is sparse.
Okay but clearly in some sense these claims clearly aren’t true because the average human is still employable, as are skilled generalists and domain experts.
Basically, I think your nomenclature choices where are confusing and seem to obscure what you actually show, which is something more like the complexity of tasks AI can do is going up over time. I can’t deny that the vibe of what you’re claiming seems right in some narrow sense, but only because it’s what’s already obvious from METR studies.
Any recommendations for starting Zen (e.g. books, retreat recordings)? Is this one of the practices where having a teacher is essential?
Yes, Zen requires a teacher. One of the founding principles of Zen is that transmission of the dharma happens mind-to-mind, outside the scriptures (or in modern terms, you learn it from doing it under the instruction of a teacher and as part of a community, not from reading about it). That said, there are many good introductory books to Zen practice. My favorites to recommend are:
Buddhism Plain and Simple by Steve Hogan
Everyday Zen by Charlotte Joko Beck (disclaimer: Joko is the root teacher of the lineage I practice within)
Zen Training: Methods and Philosophy by Katsuki Sekida (I wrote a review of it here)
I’m curious if there’s a particular school of Buddhism or text(s) that informed your understanding of jhana? I would be curious to learn different takes on it, especially those that are more established.
I practice Zen. “Zen” is literally just the word “jhana” through multiple layers of transliteration, although it’s come to take on a somewhat different meaning than how you’re using “jhana”.
Zen isn’t very interested in the jhanas, though, so most everything I know about them comes from books like Mastering the Core Teachings of The Buddha and The Mind Illuminated and practicing and seeing for myself what experiences seem to map up with how various jhanas are described. Also talking to a bunch of people who more formally practice with the jhanas and getting understanding from them.
My main way of practicing with jhanas is that they’re natural phases the mind passes through as it settles down on the way to calm abiding (shamatha).
Could you elaborate on what “meditation debt” means here?
This is a concept I’ve borrowed from Mark Lippman’s writing, although he calls it “technical debt”. I prefer “meditation debt” since it makes a bit more sense out of the context of his book. https://meditationbook.page/#technical-debt-meditation-and-minds
Make scary capabilities demos to support your case.
My sad guess is that a demo won’t be enough. Some actual real harm will have to happen to create the crisis necessary to precipitate action like a pause.
I’m somewhat less cynical on open letters than it sounds like you are, Richard.
Do I expect this letter to, by itself, get anyone at the DoW to change course? No.
But I do think it has a chance at being one more bit of evidence that sways congressional leaders, much as protests and calling them directly do. They’re responsive to voter demands because that’s how they stay in office, and if often takes many times hearing the same message because someone, including congress people, to take something seriously.
Whenever I see claims about this, I always want to push back, because phenylephrine works quite well for me. Depending on how congested I am I might have to go slightly above the recommended dosing schedule, but usually I can take the amount recommended or less and it will keep my sinuses clear.
I understand it doesn’t work for everyone, so it makes me suspicious that the studies may be seeing issues like RCTs on SSRIs do, where the drugs work, but which drug works for which person is highly variable, so they seem ineffective in an RCT but seem highly effective in practice once the patient-drug matching process is done.