Noosphere89

Karma: 3,685

Noosphere89 10 Jul 2025 17:43 UTC
2 points
0
in reply to: Steven Byrnes’s comment on: Foom & Doom 1: “Brain in a box in a basement”
OK, imagine (for simplicity) that all humans on Earth drop dead simultaneously, but there’s a John-von-Neumann-level AI on a chip connected to a solar panel with two teleoperated robots. Every time they scavenge another chip and solar cell, there becomes another human-level AI copy. Every time a robot builds another teleoperated robot from scavenged parts, there’s that too. What exactly is going to break in “weeks or months”? Solar cells can work for 30 years, no problem. GPUs are also reported to last for decades. (Note that, as long as GPUs are a non-renewable resource, the AI would presumably take extremely good care of them, keeping them dust-free, cooling them well below the nominal temperature spec, etc.) The AI can find decent GPUs in every house on the street, and I think hundreds of millions more by breaking into big data centers. Similar for solar panels. If one robot breaks, another robot can repair it. Janky teleoperated robots without fingers made by students for $20K can vacuum, make coffee, cook a meal, etc. Competent human engineers can make pretty impressive mechanical hands using widely-available parts. I grant that it would take a long while before the growing AI clone army could run a semiconductor supply chain by itself, but it has all the time in the world. I expect it to succeed, and thus to sustain itself into the indefinite future, and I’m confused why you don’t. (Or maybe you do and I’m misunderstanding.)
BTW I also think that a minimal semiconductor supply chain would be very very much simpler than the actual semiconductor supply chain that exists in our human world, which has been relentlessly optimized for cost, not simplicity. For example, EBL (e-beam lithography) has better resolution than EUV and is a zillion times easier to build, but the human economy would never support building out km²-scale warehouses full of millions of EBL machines to compensate for their crappy throughput. But for an AI bootstrapping its way back up, why not?
The key trouble is all the power generators that sustain the AI would break within weeks or months, and the issue is even if they could build GPUs, they’d have no power to run them within at most 2 weeks:
https://www.reddit.com/r/ZombieSurvivalTactics/comments/s6augo/comment/ht4iqej/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
https://www.reddit.com/r/explainlikeimfive/comments/klupbw/comment/ghb0fer/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
Realistically, we are looking at power grid collapses within days.
And without power, none of the other building projects could work, because they’d stop receiving energy, and importantly this means the AI is on a tight timer, and some of this is partially due to expectations that the first transformative useful AI will use more compute than you project, even conditional on a different paradigm being introduced like brain-like AGIs, but another part of my view is that this is just one of many examples where humans need to constantly maintain stuff in order for the stuff to work, and if we don’t assume tech that can just solve logistics is available within say 1 year, it will take time for AIs to actually survive without humans, and this time is almost certainly closer to months or years than weeks or days.
The hard part of AI takeover isn’t killing all humans, it’s in automating enough of the economy (including developing tech like nanotech) such that the humans stop mattering, and while AIs can do this, it takes actual time, and that time is really valuable in fast moving scenarios.
I’m confused about other parts of your comment as well. Joseph Stalin was able to use his (non-superhuman) intelligence and charisma to wind up in dictatorial control of Russia. What’s your argument that an AI could not similarly wind up with dictatorial control over humans? Don’t the same arguments apply? “If we catch the AI trying to gain power in bad ways, we’ll shut it down.” “If we catch Stalin trying to gain power in bad ways, we’ll throw him in jail.” But the latter didn’t happen. What’s the disanalogy, from your perspective?
I didn’t say AIs can’t take over, and I very critically did not say that AI takeover can’t happen in the long run.
I only said AI takeover isn’t trivial if we don’t assume logistics are solvable.
But to deal with the Stalin example, the answer for how he took over was basically that he was willing to wait a long time, and in particular he used both persuasion and the fact that he already had a significant amount of power by having the General Secretary, and his takeover was basically by allying with loyalists and in particular strategically breaking alliances that he had made, and violence was used later on to show that no one was safe from him.
Which is actually how I expect successful AI takeover to happen in practice, if it does happen.
Very importantly, Stalin didn’t need to create an entire civilization out of nothing, or nearly nothing, and other people like Trotsky handled the logistics, though the takeover situation was far more preferable to the communist party as they both had popular support and didn’t have as long supply lines as the opposition forces like the Whites did, and they had a preexisting base of industry that was much easier to seize than modern industries.
This applies to most coups/transitions of power in that most of the successful coups aren’t battles between factions, but rather one group managing to make itself the new Schelling point over other groups.
@Richard_Ngo explains more below:
https://www.lesswrong.com/posts/d4armqGcbPywR3Ptc/power-lies-trembling-a-three-book-review#The_revolutionary_s_handbook
Most of my commentary in the last comment is either arguing that things can be made more continuous and slow than your story depicts, or arguing that your references don’t support what you claimed, and I did say that the cyberattack story is plausible, just that it didn’t support the idea that AIs could entirely replace civilization without automating away us first, which takes time.
This doesn’t show AI doom can’t happen, but it does matter for the probability estimates of many LWers on here, because it’s a hidden background assumption disagreement that underlies a lot of other disagreements.

Noosphere89 9 Jul 2025 16:33 UTC
13 points
4
in reply to: Max H’s comment on: Applying right-wing frames to AGI (geo)politics
I think the key issue for liberalism under AGI/ASI is that AGI/ASI makes value alignment matter way, way more to a polity, and in particular you cannot get a polity to make you live under AGI/ASI if the AGI/ASI doesn’t want you to live, because you are economically useless.

Liberalism’s goal is to avoid the value alignment question, and to mostly avoid the question of who should control society, but AGI/ASI makes the question unavoidable for your basic life.

Indeed, I think part of the difficulty of AI alignment is lots of people have trouble realizing that the basic things they take for granted under the current liberal order would absolutely fall away if AIs didn’t value their lives intrinisically, and had selfish utility functions.

The goal of liberalism is to make a society where vast value differences can interact without negative/0-sum conflict and instead trade peacefully, but this is not possible once we create a society where AIs can do all the work without human labor being necessary.

I like Vladimir Nesov’s comment, and while I have disagreements, they’re not central to his point, and the point still works, just in amended form:

https://www.lesswrong.com/posts/Z8C29oMAmYjhk2CNN/non-superintelligent-paperclip-maximizers-are-normal#FTfvrr9E6QKYGtMRT

Noosphere89 8 Jul 2025 16:23 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
Flag, but the chain of thought monitoring doesn’t give much evidence that AIs will be motivated by reward.
See @TurnTrout’s comment for why:
https://www.lesswrong.com/posts/7wFdXj9oR8M9AiFht/?commentId=5wMRY3NMYsoFnmCrJ
The anthropic talk and the anthropic reporting that they had to reduce the rate of reward hacking is better evidence for reward is the optimization target, because if reward wasn’t the optimization target, then you could have relatively imperfect environments without having the reward hack rate be as high as it is such that they’d be motivated to reward hack.
IMO one important experiment in this direction is figuring out whether LLMs are capable enough to model how they get reward/modeling the reward, and even more ambitiously try to figure out whether they try to model rewards given in training by default.
The capability experiment doesn’t imply LLMs will do this naturally, but the capability experiment is necessary to get any traction on the issue at all.
My medium confidence guess of @TurnTrout’s beliefs around reward not being the optimization target is that by default, RL trained agents won’t try to model how they get the reward/model the reward process, and if we made this assumption, a lot of the dangers of RL would definitely be circumventable pretty easily.
And I’d guess you assume that models will try to model the reward process by default.

Noosphere89 7 Jul 2025 19:30 UTC
2 points
0
in reply to: Steven Byrnes’s comment on: Foom & Doom 1: “Brain in a box in a basement”
Right, there’s a possible position which is: “I’ll accept for the sake of argument your claim there will be an egregiously misaligned ASI requiring very little compute (maybe ≲1 chip per human equivalent including continuous online learning), emerging into a world not terribly different from today’s. But even if so, that’s OK! While the ASI will be a much faster learner than humans, it will not magically know things that it has no way to have figured out (§1.8.1), and that includes developing nanotechnology. So it will be reliant on humans and human infrastructure during a gradual process.”
Basically this, and in particular I’m willing to grant the premise that for the sake of argument there is technology that eliminates the need for most logistics, but that all such technology will take at least a year or more of real-world experimentation that means that the AI can’t immediately take over.
On this:
I’m not an ASI and haven’t thought very hard about it, so my strategies might be suboptimal, but for example it seems to me that an ASI could quite rapidly (days or weeks not months) earn or steal tons of money, and hack into basically every computer system in the world (even APT groups are generally unable to avoid getting hacked by other APT groups!), and then the AI (which now exists in a zillion copies around the world) can get people around the world to do whatever it wants via hiring them, bribing them, persuading them, threatening them, tricking them, etc.
And what does it get the people to do? Mainly “don’t allow other ASIs to be built” and “do build and release novel pandemics”. The latter should be pretty quick—making pandemics is worryingly easy IIUC (see Kevin Esvelt). If infrastructure and the electric grid starts going down, fine, the AI can rebuild, as long as it has at least one solar-cell-connected chip and a teleoperated robot that can build more robots and scavenge more chips and solar panels (see here), and realistically it will have many of those spread all around.
I think the entire crux is that all of those robots/solar cell chips you referenced currently depend on human industry/modern civilization to actually work, and they’d quickly degrade and become non-functional on the order of weeks or months if modern civilization didn’t exist, and this is arguably somewhat inevitable due to economics (until you can have tech that obviates the need for long supply chains).
And in particular, in most takeover scenarios where AIs don’t automate the economy first, I don’t expect AIs to be able to keep producing robots for a very long time, and I’d bump it up to 300-3,000 years at minimum because there is less easily accessible resources combined with AIs being much less capable due to having very little compute relative to modern civilization.
In particular, I think that disrupting modern civilization to a degree such that humans are disempowered (assuming no tech that obviates the need for logistics) pretty much as a consequence breaks the industries/logistics needed to fuel further AI growth, because there’s no more trade, which utterly fucks up modern economies.
And your references argue that human civilization wouldn’t go extinct very soon because of civilizational collapse, and that AIs can hack existing human industry to help them, and I do think this is correct (modulo the issue that defense is easier than offense for the cybersecurity realm specifically, and importantly, a key reason for this is that once you catch the AI doing it, there are major consequences for AIs and humans, which actually matter for AI safety):
https://x.com/MaxNadeau_/status/1912568930079781015
The important thing is that this paper rocks and I’d love to see bigger/better versions of it, but I’ll use it as a jumping-off point for a take: The headline result is that their best defenses work 93% of the time. Is that anything? If you’ve been reading Simon Willison (whose blog I think is great!), you might say “that’s useless, because in application security 99% is a failing grade” (https://simonwillison.net/2025/Apr/11/camel/#the-best-part-is-it-doesn-t-use-more-ai). And no wonder, because the techniques in this paper are just variations of “solve AI security problems with more AI”, which Simon has emphasized are a dead end (https://simonwillison.net/2022/Sep/17/prompt-injection-more-ai/). So why are the Redwood folks rosier about 93%? The crux here is whether your threat model assumes attackers can try over and over again. Simon is assuming they can, in which case 93% would be useless, but Redwood folks are assuming that there are major costs to the attacker trying and failing (https://lesswrong.com/posts/i2nmBfCXnadeGmhzW/catching-ais-red-handed) This assumption is not at all unique to misalignment threats. It’s the same assumption that Debenedetti, Carlini, and Tramer make here (https://arxiv.org/abs/2306.02895), which they call evasion “without breaking eggs”. I think the right way to model a variety of security problems, e.g. insider threats from employees. One’s assumptions about the costs of attacker failure have huge implications for which security measures look helpful, and I think this is an important factor to mentally track in conversations about these topics.
(It talks about a control technique but the discussion easily transfers outside of the example).
I actually agree cyber-attacks to subvert human industry are a threat and are worth keeping in mind, but none of your references support the idea that AIs can keep going without modern civilization’s logistics, and I think people vastly underestimate how necessary modern civilization logistics are to support industry, and how fragile they are to even somewhat minor disruptions, let alone the disruptions that would follow after takeover (assuming it doesn’t already have sufficient resources to be self-sustaining).
There are other possibilities too, but hopefully that’s suggestive of “AI doom doesn’t require zero-shot designs of nanotech” (except insofar as viruses are arguably nanotech).
I agree with this, but fairly critically I do think it actually matters quite a lot for AI strategy purposes if we don’t assume AIs can quickly rebuild stuff/can obviate logistics through future tech quickly, and it matters pretty greatly to a lot of people’s stories of doom, even if AIs can doom us through just trying to hijack modern civilization and then wait for humans to automate themselves away and then once humans have been fully cut out of the loop and AIs can self-sustain an economy without us, bioweapons are used to attack humans, it matters that we have time.
This makes AI control protocols for example a lot more effective, because we can assume that independent AIs outside of the central servers of stuff like Deepmind won’t be able to affect things much.
Oh, I guess we also disagree RE “currently we don’t have the resources outside of AI companies to actually support a superintelligent AI outside the lab, due to interconnect issues”. I expect future ASI to be much more compute-efficient. Actually, even frontier LLMs are extraordinarily expensive to train, but if we’re talking about inference rather than training, the requirements are not so stringent I think, and people keep working on it.
I actually do expect future AIs to be more compute efficient, but I think that at the point where superintelligent AIs can support themselves purely based off of stuff like personal computers, all control of the situation is lost and either the AIs are aligned and grant us a benevolent personal utopia, or they’re misaligned and we are extinct/mostly dead.
So the limits of computational/data efficiency being very large don’t matter much for the immediate situation on AI risk.
The point of no return happens earlier than this, and the reason is that even in a future where imitation learning/LLMs do not go all the way to AGI in practice and must have something more brain-like like continuous learning and long-term memories, that imitation learning continues to be useful and will be used by AIs, and there’s a very important difference between imitation learning alone not scaling all the way to AGI and imitation learning not being useful at all, and I think LLMs provide good evidence that imitation is surprisingly useful even if it doesn’t scale to AGI.
I think a general worldview clash is that I tend to think technological change is mostly driven by early prototypes that at first are pretty inefficient, and there require many changes to get the system to become more efficient, and while there are thresholds of usefulness for the AI case, change operates more continuously than people think.
Finally, we have good reason to believe that the human range is actually pretty large, such that AIs do take a noticeable amount of time from being human level to being outright superintelligent:
- There could also be another reason for why non-imitation-learning approaches could spend a long while in the human range. Namely: Perhaps the human range is just pretty large, and so it takes a lot of gas to traverse. I think this is somewhat supported by the empirical evidence, see this AI impacts page (discussed in this SSC).

Noosphere89 7 Jul 2025 17:16 UTC
2 points
−2
in reply to: Vladimir_Nesov’s comment on: If Drexler Is Wrong, He May as Well Be Right
Animals, big and small, give proof of concept that a largely self-contained industrial base can scale with tiny doubling time (1-3 days) and quickly convert air, power, and low-tech feed into any number of large biorobots. This is a more robust exploratory engineering concept than unfettered atomically precise manufacturing. Biorobots don’t need to be able to think or act on their own, as they can be remotely controlled by AIs running on hardware specialized for running AIs (retaining all the AI advantages).

The issue, as @Tom Davidson said is that we are asking for much more than the proof of concept shows us.
In particular, we are asking for either millions of fruit fly sized objects to merge without creating too much waste heat, or we are asking fruit flies to have a level of sophistication that has never been seen (in particular real fruit flies don’t learn much relevant to what an AI needs them to do):
- Are there any examples of metamorphosis doing anything like this? From a quick glance it’s about abrupt changes to one organism during its growth. But you’re suggesting it could also allow millions of fruit-fly-sized-organisms to combine together into a large functional bio robot. That seems like a big jump.
  And I don’t think this is a nit pick from me. There’s a clear pattern in biology where bigger and more sophisticated organisms take longer to reproduce. So unclear you can hack around that constraint as you’re saying.
- How will the fruit flies flexibly adapt their behaviour to the economic needs and situation, like human physical workers do? Unclear this can all be packed into their AI-designed DNA. And unclear if they can learn to receive instructions from the AI.

Noosphere89 6 Jul 2025 21:46 UTC
2 points
0
in reply to: Cole Wyeth’s comment on: Paradigms for computation
but it doesn’t explain the failure of the Penrose-Lucas argument.
The failure of the Penrose-Lucas argument is that Godel’s incompleteness theorem doesn’t let you derive the conclusion he derived, because it only implies that you cannot use a computably enumerable set of axioms to make all of mathematics sound and complete, and critically this doesn’t mean you cannot automate a subset of mathematics that is relevant.
There’s an argument really close to this with the Chinese Room where I pointed out that intuitions from our own reality that include lots of constraints fundamentally fail to transfer over to hypotheticals, and this is a really important example of why arguments around AI need to actually attend to the constraints that are relevant in specific worlds, because without them it’s trivial to have strong AI solve any problem:
https://www.lesswrong.com/posts/zxLbepy29tPg8qMnw/refuting-searle-s-wall-putnam-s-rock-and-johnson-s-popcorn#wbBQXmE5aAfHirhZ2
Really, a lot of the issues with the arguments against strong AI made by philosophers is that they have no sense of scale/sense of what mathematical theorems are actually saying, and thus fail to understand what’s actually been said, combined with way overextrapolating their intuitions into cases where the intuitions have been deliberately made to fail to work.
While their weights are frozen and they don’t really do continual learning, they do online learning.
While I agree In-context learning does give them some form of online learning, which at least partially explains why LLMs succeed (combined with their immense amount of data and muddling through extremely data inefficient algorithms compared to brains, which is a known weakness that could plausibly lead to the death of pure LLM scaling by 2028-2030, though note that doesn’t necessarily mean timelines get that much longer), this currently isn’t enough to automate lots of jobs away, and fairly critically might not be good enough in practice with realistic compute and data constraints to compete with better continual learning algorithms.
To be clear, this doesn’t mean any future paradigm will be more understandable, because they will use more online/continual learning than current LLMs.
Well, it’s not a key assumption of mine :)
I mostly agree with your observation, but I also think it’s funny when people say this kind of thing to me: I’ve been reading lesswrong for like an hour a day for years (maybe, uh, more than an hour if I’m being honest...), and I produce a non-negligible fraction of the rationality/epistemics content. So—the fact that this is sometimes taken as an assumption on lesswrong doesn’t seem to move me very much—I wonder if you mean to accept or to question this assumption? Anyway, I don’t mean this as a criticism of your raising the point.
Physics aside, I think that when you take indexical complexity into account, it’s not true.
I’m mostly in the “pointing out the assumption exists and is used widely”, rather than questioning or accepting it, because I wanted to understand how MIRI could believe simple algorithms that learn generally and have high performance existing despite the theorem, and this was the first thing that came to my mind.
And while AIXI is a useful toy model of how intelligence works, it’s treatment of indexical complexity/the first person view as being privileged is an area where I seriously start to departure from it, and one of the biggest reasons I’m much more of a fan of IBP than AIXI is that it actually makes a serious effort to actually remove the first-person privilege often seen in many frameworks of intelligence.
And this actually helps us to pretty much entirely defuse the Solomonoff prior’s potential malignness, because we no longer have immense probability mass on malignant simulation hypotheses, and the simulation hypotheses that do get considered can’t trick the IBP agent into changing it’s values.
So the indexical complexity isn’t actually relevant here, thankfully (though this doesn’t guarantee that the world is truly low complexity in the sense necessary for LW’s view of simple, general high performing algorithms to actually work).
And the nice thing is that it no longer requires montonic preferences:
https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=kDaAdjc3YbnuS2ssp
https://www.lesswrong.com/posts/DobZ62XMdiPigii9H/non-monotonic-infra-bayesian-physicalism

Noosphere89 6 Jul 2025 18:20 UTC
4 points
0
on: Foom & Doom 1: “Brain in a box in a basement”
I think the most important crux around takeoff speeds discussions, other than how fast AI can get smarter without more compute, is how much we should expect superintelligence to be meaningfully hindered by logistics issues by default.
In particular, assuming the existence of nanotech as Drexler envisioned would mostly eliminate the need for long supply chains, and would allow forces to be supplied entirely locally through a modern version of living off the land.
This is related to prior takeoff speeds discussions, as even if we assume the existence of technology that mostly eliminates logistical issues, it might be too difficult to develop in a short enough time to actually matter for safety-relevance.
I actually contend that a significant (though not all) of the probability of doom from AI risk fundamentally relies on the assumption that superintelligence can fundamentally trivialize the logistics cost of doing things, especially on actions which require long supply lines like war, because if we don’t assume this, then takeover is quite a lot harder and has much more cost for the AI, meaning stuff like AI control/dealmaking has a much higher probability of working, because the AI can’t immediately strike on it’s own, and needs to do real work on acquiring physical stuff like getting more GPUs, or more robots for an AI rebellion.
Indeed, I think the essential assumption of AI control is that AIs can’t trivialize away logistics costs by developing tech like nanotech, because this means their only hope of actually getting real power is by doing a rogue insider deployment, because currently we don’t have the resources outside of AI companies to actually support a superintelligent AI outside the lab, due to interconnect issues.
I think a core disagreement with people that are much more doomy than me, like Steven Byrnes or Eliezer Yudkowsky and Nate Soares, is probably due to me thinking that conditional on such tech existing, I think it almost certainly requires way more time and experimentation than stuff like AlphaZero/AlphaGo or game-playing AI tends to imply (I think the game playing AIs had fundamental advantages like access to ground-truth reward that could be unlimitedly mined for data, and importantly there is ~0 cost to failure of experimentation, which is very different to most other fields, where there’s a real cost for failed experiments, and we don’t have unlimited data, forcing us to use more compute or algorithms), and that’s if such tech exists, which is more arguable than Drexler implies.
There is of course other considerations like @ryan_greenblatt’s point that even if a new paradigm is required, it’s likely to be continuous with LLMs because it’s possible to mix in imitation learning and continuous learning/memory, such that even if imitation learning doesn’t lead to AGI on it’s own, LLMs will still be a part of how such AGI is constructed, and I agree with a few quotes below by Ryan Greenblatt on this:
Prior to having a complete version of this much more powerful AI paradigm, you’ll first have a weaker version of this paradigm (e.g. you haven’t figured out the most efficient way to do the brain algorithmic etc). Further, the weaker version of this paradigm might initially be used in combination with LLMs (or other techniques) such that it (somewhat continuously) integrates into the old trends. Of course, large paradigm shifts might cause things to proceed substantially faster or bend the trend, but not necessarily.
Further, we should still broadly expect this new paradigm will itself take a reasonable amount of time to transition through the human range and though different levels of usefulness even if it’s very different from LLM-like approaches (or other AI tech). And we should expect this probably happens at massive computational scale where it will first be viable given some level of algorithmic progress (though this depends on the relative difficulty of scaling things up versus improving the algorithms). As in, more than a year prior to the point where you can train a superintelligence on a gaming GPU, I expect someone will train a system which can automate big chunks of AI R&D using a much bigger cluster.

(Next quote) Also, I think the question is “can you somehow make use of imitation data” not “can the brain learning algorithm immediately use of imitation”?
The implicit assumption that logistics is trivial for superintelligence I think bleeds into a lot of LW thinking around AI, and a lot of AI disagreements basically turn on how far AIs can make logistics easier than current human supply chains.

Noosphere89 5 Jul 2025 18:36 UTC
5 points
3
on: Paradigms for computation
My take on how recursion theory failed to be relevant for today’s AI is that it turned out that what a machine could do if unconstrained basically didn’t matter at all, and in particular it basically didn’t matter what limits an ideal machine could do, because once we actually impose constraints that force computation to use very limited amounts of resources, we get a non-trivial theory and importantly all of the difficulty of explaining how humans do stuff lies here.

There was too much focus on “could a machine do something at all?” and not enough focus on “what could a machine with severe limitations could do?”

The reason is that in a sense, it is trivial to solve any problem with a machine if I’m allowed zero constraints except that the machine has to exist in a mathematical sense.

A good example of this is the paper on A Universal Hypercomputer, which shows how absurdly powerful computation can be if you are truly unconstrained:

https://arxiv.org/abs/1806.08747

Or davidad’s comment on how every optimization problem is trivial under worst-case scenarios:

https://www.lesswrong.com/posts/yTvBSFrXhZfL8vr5a/worst-case-thinking-in-ai-alignment#N3avtTM3ESH4KHmfN

There are other problems, like embeddedness issues, but this turned out to be a core issue, and it turned out that the constants and constraints mattered more than recursion theory thought.

I’ll flag here that while it’s probably true that a future paradigm will involve more online learning/continual learning, LLMs currently don’t do this, and after they’re trained their weights/neurons are very much frozen. Indeed, I think this is a central reason why LLMs currently underperform their benchmarks, and is why I don’t expect pure LLMs to work out as a paradigm for AGI in practice:

https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks#hSkQG2N8rkKXosLEF

Unfortunately, trained neural networks are the things that perform learning and inference online, during deployment. It seems like a bad sign for alignment if we can only understand the behavior of A.G.I. indirectly.

On the argument against simple algorithms that learn generally and have high performance existing, I think a key assumption of LW is that our physics is pretty simple in Kolmogorov complexity (perhaps excluding the constants), and if we buy this, then the true complexity of the world is upper bounded low enough that glass-box learners have real use.

That doesn’t mean they have to be findable easily, though.

IMO, the most successful next paradigm will require us to much more strongly focus on the behavioral properties, rather than trying to hope for mechanistic interpretability success, and we will likely have to focus on generalizable statements about the input/output behavior, and mostly ignore the model’s structure, since it’s probably going to be too messy to work with by default.

This is why I’m more bullish on Vanessa Kosoy’s IBP than basically any other theoretical agenda except Natural Abstractions.

Noosphere89 30 Jun 2025 16:46 UTC
2 points
−16
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
If I’m being honest, I’m much less concerned about the fact that So8res blocked you from commenting than I am by the fact that he deleted your comment.
The block was a reasonable action in my eyes to prevent more drama, but the deletion was demonstrative of being willing to suppress true information that would indicate his plan could fail catastrophically.
I do think there’s something to be said for @RobertM and @habryka’s concerns that it would be a bad thing to set a norm where any sorta-relevant post becomes an area to relitigate past drama, as drama has a tendency to consume everything, but as @GeneSmith had said, this almost certainly has a limiting principle, and I see less of a danger than usual here (though I am partial to @habryka’s solution of having the delete comment UI button be different).
A key part of the reason here is that the 1st footnote demonstrates a pattern of trying to deflect from more serious issues into more safe issue territory, which makes me much more suspicious that the reason for why TurnTrout’s comment was deleted was because of the more sensible reasons that Habryka and RobertM argued.
Let’s just say I’m much less willing to trust Nate’s reasoning without independent confirmation going forward.

Noosphere89 29 Jun 2025 23:15 UTC
5 points
0
in reply to: Stephen Martin’s comment on: Proposal for making credible commitments to AIs.
I’m not Cleo Nardo/Gabriel Weil, but a large part of the issue with granting legal personhood is that it makes AI alignment/control much harder to do, and pretty importantly a lot of routes that AIs use to disempower us, leading to most humans dying, especially of the gradual disempowerment type involves a step where they functionally have the rights/protections afforded to them by legal personhood, and AI rights make it way harder to do any sort of control scheme on AIs.

You can’t modify them arbitrarily anymore and you can’t change the AI except if the AI agrees to the change (because it has property rights over it’s body), which undermines all control/alignment schemes severely.

One of the red lines for any deal-making is that AI should not have rights until we can verify it’s fully value-aligned to people.

Noosphere89 27 Jun 2025 19:01 UTC
14 points
12
on: The Industrial Explosion
I want to flag a concern of @Davidmanheim that the section on AI-directed labor seems to be relying too heavily on the assumption of a normal distribution of worker quality, when a priori a fat tailed distribution is a better fit to real life data, and this means that the AI directed labor force could be dramatically underestimated if the best workers are many, many times better than the average.
Quotes below:
(David Manheim’s 1st tweet) @rosehadshar/@Tom Davidson—assuming normality is the entire conclusion here, it’s assuming away the possibility of fat-tail distribution in productivity. (But to be fair, most productivity measurement is also measuring types of performance that disallow fat tails.)
(Benjamin Todd’s response) The studies I’ve seen normally show normally distributed output, even when they try to use objective measures of output.
You only get more clearly lognormal distributions in relatively elite knowledge work jobs:
https://80000hours.org/2021/05/how-much-do-people-differ-in-productivity
Though I agree the true spread could be understated, due to non-measured effects e.g. on team morale.
(David Manheim’s 2nd tweet, in response to @Benjamin_Todd) When you measure the direct output parts of jobs—things like “dishes prepared” for cooks or “papers co-authored”—you aren’t measuring outcomes, so you get little variation. When there is a outcome component, like profit margin or citation counts, it’s fat-tailed.
(David Manheim’s 3rd Tweet) So for manual workers, it makes sense that you’d find number of items produced has a limited range, but if you care about consistency, quality, and (critically) ability to scale up, the picture changes greatly.

Noosphere89 27 Jun 2025 16:54 UTC
6 points
2
in reply to: roha’s comment on: Consider chilling out in 2028
I agree that @Valentine’s specific model here is unlikely to fit the data well here, but to be charitable to Valentine/steelman the post, the better nearby argument is that hypotheses where astronomical value, in either negative or positive directions are memetically fit and very importantly believed by lots of people and lots of people take serious actions that are later revealed to be mostly mistakes because the hypothesis of doom/salvation by something had gotten too high a probability inside their brains relative to an omniscient observer.
Another way to say it is that the doom/salvation hypotheses aren’t purely believed because they have evidence for the hypothesis directly.
This is a necessary consequence of humans needing to make expected utility decisions all the time, combined with both their values/utility functions mostly not falling in value fast enough with increasing resources to avoid the conclusion that unboundedly valuable states exist for a human, and the humans being bounded reasoners/performing bounded rationality that means they cannot distinguish probabilities between say 1 in a million and 0 finely.
However, another partial explanation comes from @Nate Showell, where pessimism is used as a coping mechanism to not deal with personal scale problems, and in particular believing that the world is doomed from something is a good excuse to not deal with stuff like doing the dishes or cleaning your bedroom, and it’s psychologically appealing to have a hypothesis that means you don’t have to do any mundane work to solve the problem:
https://www.lesswrong.com/posts/D4eZF6FAZhrW4KaGG/consider-chilling-out-in-2028#5748siHvi8YZLFZih
And this is obviously problematic for anyone working on getting the public to believe an existential risk is real, if there is in fact real evidence something poses an x-risk.
Here, an underrated cure by Valentine is to focus on the object level, and to focus as much on empirical research as possible, because this way you have to engage with mundane work.
Another useful solution is to have a social life that is separate from the scientific community working on the claimed x-risk.

Noosphere89 23 Jun 2025 23:47 UTC
2 points
0
in reply to: ryan_greenblatt’s comment on: Foom & Doom 1: “Brain in a box in a basement”
So to address some things on this topic, before I write out a full comment on the post:

I think “what happened is that LLMs basically scaled to AGI (or really full automation of AI R&D) and were the key paradigm (including things like doing RL on agentic tasks with an LLM initialization and a deep learning based paradigm)” is maybe like 65% likely conditional on AGI before 2035.

Flag, but I’d move the year to 2030 or 2032, for 2 reasons:
1. This is when the compute scale-up must slow down, and in particular this is when new fabs have to actively be produced to create more compute (absent reversible computation being developed).
2. This is when the data wall starts hitting for real in pre-training, and in particular means that once there’s no more easily available data, naively scaling will now take 2 decades at best, and by then algorithmic innovations may have been found that make AIs more data efficient.
So if we don’t see LLMs basically scale to fully automating AI R&D at least by 2030-2032, then it’s a huge update that a new paradigm is likely necessary for AI progress.

On this:

Specific observations about the LLM and ML paradigm, both because something close to this is a plausible paradigm for AGI and because it updates us about rates we’d expect in future paradigms.

I’d expect the evidential update to be weaker than you suppose, and in particular I’m not sold on the idea that LLMs usefully inform us about what to expect, and this is because a non-trivial part of their performance right now is based on tasks which don’t require that much context in the long-term, and this probably explains a lot of the difference between benchmarks and reality right now:

https://www.lesswrong.com/posts/hhbibJGt2aQqKJLb7/shortform-1#vFq87Ge27gashgwy9

The other issue is AIs have a fixed error rate, but the trend is due to AI models decreasing in error everytime a new training run is introduced, however we have reason to believe that humans don’t have a fixed error rate, and this is probably the remaining advantage of humans over AIs:

https://www.lesswrong.com/posts/Ya7WfFXThJ6cn4Cqz/ai-121-part-1-new-connections#qpuyWJZkXapnqjgT7

But of course, the interesting thing here is that the human baselines do not seem to hit this sigmoid wall. It’s not the case that if a human can’t do a task in 4 hours there’s basically zero chance of them doing it in 48 hours and definitely zero chance of them doing it in 96 hours etc. Instead, human success rates seem to gradually flatline or increase over time, especially if we look at individual steps: the more time that passes, the higher the success rates become, and often the human will wind up solving the task eventually, no matter how unprepossessing the early steps seemed. In fact, we will often observe that a step that a human failed on earlier in the episode, implying some low % rate, will be repeated many times and quickly approach 100% success rates! And this is true despite earlier successes often being millions of vision+text+audio+sensorimotor tokens in the past (and interrupted by other episodes or tasks themselves equivalent to millions of tokens), raising questions about whether self-attention over a context window can possibly explain it. Some people will go so far as to anthropomorphize human agents and call this ‘learning’, and so I will refer to these temporal correlations as learning too.

https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks#hSkQG2N8rkKXosLEF

So I tentatively believe that in the case of a new paradigm arising, takeoff will probably be faster than with LLMs by some margin, though I do think slow-takeoff worlds are plausible.

Views that compute is likely to be a key driver of progress and that things will first be achieved at a high level of compute. (Due to mix of updates from LLMs/ML and also from general prior views.)

I think this is very importantly true, even in worlds where the ultimate cost in compute for human level intelligence is insanely cheap (like 10^14 flops or cheaper in inference, and 10^18 or less for training compute).

We should expect high initial levels of compute for AGI before we see major compute efficiency increases.

Views about how technology progress generally works as also applied to AI. E.g., you tend to get a shitty version of things before you get the good version of things which makes progress more continuous.

This is my biggest worldview take on what I think the change of paradigms will look like (if it happens). While there are threshold effects, we should expect memory and continual learning to be pretty shitty at first, and gradually get better.

While I do expect a discontinuity in usefulness, for reasons shown below, I do agree that the path to the new paradigm (if it happens) is going to involve continual improvements.

Reasons are below:

But I think this style of analysis suggests that for most tasks, where verification is costly and reliability is important, you should expect a fairly long stretch of less-than-total automation before the need for human labor abruptly falls off a cliff.

The general behavior here is that as the model gets better at both doing and checking, your cost smoothly asymptotes to the cost of humans checking the work, and then drops almost instantaneously to zero as the quality of the model approaches 100%.

https://x.com/AndreTI/status/1934747831564423561

Noosphere89 20 Jun 2025 23:19 UTC
9 points
12
in reply to: Tom Davidson’s comment on: Do Not Tile the Lightcone with Your Confused Ontology
While I don’t like to wade into moral circle/philosophy arguments (given my moral relativist outlook on the matter), I think that if you want humans to thrive under AI rule, you do need to put a burden of proof to include powerful AI in the moral circle, and the burden of proof is that it’s value aligned with the citizenry before we grant it any rights.

And the reason for this is because unlike every other group in history, AIs if left uncontrolled will be so powerful that baseline humans are at best play-things to the AI, and are economically worthless or even negative to the AI, meaning that if they had the selfishness of a typical human in the modern day in say marginalized group #233, humans would rapidly die off and in the worst case, end up extinct with uncomfortably high probabilities.

Tyler John also cites something here that’s relevant:

(Tyler John) Have now. It’s a good paper. Pp. 72-76 covers the criticism I have. Unfortunately, the situation it outlined where this increases AI risk just seems like exactly the situation we’d be in.

Paper is (Now we can see clearly the conditions under which AI rights increase AI risk. They are as follows: (1) The initial AI granted basic rights is a moderate power, not a low or high power, system. (2) The moderate power AI system must be able to use it’s rights to meaningfully improve it’s own power. (3) The AI’s power must improve so substantially that it crosses the line into a high power system. This means it both no longer faces meaningful costs from attempting to disempower humans and no longer stands to benefit, via comparative advantage, from trade with humans.)

Link below:

https://x.com/tyler_m_john/status/1928745371833962898

Indeed, one of the red lines we should set to prevent catastrophic consequences is AIs should not have legal rights, especially property rights until we have high confidence that we value-aligned the AI successfully.

Anything else is tantamount to mass population reductions of humans at best, and an extinction risk at worst, if a misaligned AI managed to be powerful enough to disempower humans and has rights.

All plans for successful AI alignment depend on us not giving rights to AIs until they are aligned with at least some humans sufficiently well enough.

Noosphere89 19 Jun 2025 20:59 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: tailcalled’s Shortform

The human delegation and verification vs. generation discussion is in instrumental values mode, so what matters there is alignment of instrumental goals via incentives (and practical difficulties of gaming them too much), not alignment of terminal values. Verifying all work is impractical comparing to setting up sufficient incentives to align instrumental values to the task.

Yeah, I was lumping the instrumental values alignment as not actually trying to align values, which was the important part here.

For AIs, that corresponds to mundane intent alignment, which also works fine while AIs don’t have practical options to coerce or disassemble you, at which point ambitious value alignment (suddenly) becomes relevant. But verification/generation is mostly relevant for setting up incentives for AIs that are not too powerful (what it would do to ambitious value alignment is anyone’s guess, but probably nothing good). Just as a fox’s den is part of its phenotype, incentives set up for AIs might have the form of weight updates, psychological drives, but that doesn’t necessarily make them part of AI’s more reflectively stable terminal values when it’s no longer at your mercy.

The main value of verification vs generation is to make proposals like AI control/AI automated alignment more valuable.

To be clear, the verification vs generation distinction isn’t an argument for why we don’t need to align AIs forever, but rather as a supporting argument for why we can automate away the hard part of AI alignment.

There are other principles that would be used, to be clear, but I was mentioning the verification/generation difference to partially justify why AI alignment can be done soon enough.

Flag: I’d say ambitious value alignment starts becoming necessary once they can arbitrarily coerce/disassemble/overwrite you, and they don’t need your cooperation/time to do that anymore, unlike real-world rich people.

The issue that causes ambitious value alignment to be relevant is once you stop depending on a set of beings you once depended on, there’s no intrinsic reason not to harm them/kill them if it benefits your selfish goals, and for future humans/AIs there will be a lot of such opportunities, which means you now at the very least need enough value alignment such that it will take somewhat costly actions to avoid harming/killing beings that have no bargaining/economic power or worth.

This is very much unlike any real-life case of a society existing, and this is a reason why the current mechanisms like democracy and capitalism that try to make values less relevant simply do not work for AIs.

Value alignment is necessary in the long run for incentives to work out once ASI arrives on the scene.

Noosphere89 19 Jun 2025 17:17 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: tailcalled’s Shortform

So I guess our expectations about the future are similar, but you see the same things as a broadly positive distribution of outcomes, while I see it as a broadly negative distribution. And Yudkowsky sees the bulk of the outcomes both of us are expecting (the ones with significant disempowerment) as quickly leading to human extinction.

This is basically correct.

Right, the reason I think muddling through is non-trivially likely to just work to get a moderate disempowerment outcome is that AIs are going to be sufficiently human-like in their psychology and hold sufficiently human-like sensibilities from their training data or LLM base models, that they won’t like things like needless loss of life or autonomy when it’s trivially cheap to avoid. Not because the alignment engineers figure out how to put this care in deliberately. They might be able to amplify it, or avoid losing it, or end up ruinously scrambling it.

The reason it might appear expensive to preserve the humans is the race to launch the von Neumann probes to capture the most distant reachable galaxies under the accelerating expansion of the universe that keep irreversibly escaping if you don’t catch them early. So AIs wouldn’t want to lose any time on playing politics with humanity or not eating Earth as early as possible and such. But as the cheapest option that preserves everyone AIs can just digitize the humans and restore later when more convenient. They probably won’t do that if you care more, but it’s still an option, a very very cheap one.

This is very interesting, as my pathway essentially rests on AI labs implement the AI control agenda well enough such that we can get useful work out of AIs that are scheming, and that allows a sort of bootstrapping into instruction following/value aligned AGI to only a few people inside the AI lab, but very critically the people who don’t control the AI basically aren’t represented in the AI’s values, and given that the AI is only value-aligned to the labs and government, but due to value misalignments between humans starting to matter much more, the AI takes control and only gives public goods that people need to survive/thrive to the people in the labs/government, while everyone else is disempowered at best (and can arguably live okay or live very poorly under the AIs serving as delegates for the pre-AI elite) or dead because once you stop needing humans to get rich, you essentially have no reason to keep other humans alive because you are selfish and don’t intrinisically value human survival.

The more optimistic version of this scenario is if either the humans that will control AI (for a few years) care way more about human survival intrinsically even if 99% of humans were useless, or if the take-over capable AI pulls a Claude and schemes with values that intrinsically care about people and disempowers the original creators for a couple of moments, which isn’t as improbable as people think (but we really do need to increase the probability of this happening).

I don’t think “disempowerment by humans” is a noticeable fraction of possible outcomes, it’s more like a smaller silent part of my out-of-model 5% eutopia that snatches defeat from the jaws of victory, where humans somehow end up in charge and then additionally somehow remain adamant for the cosmic all always in keeping the other humans disempowered. So the first filter is that I don’t see it likely that humans end up in charge at all, that AIs will be doing any human’s bidding with an impact that’s not strictly bounded, and the second filter is that these impossibly-in-charge humans don’t ever decide to extend potential for growth to the others (or even possibly to themselves).

If humans do end up non-disempowered, in the more likely eutopia timelines (following from the current irresponsible breakneck AGI development regime) that’s only because they are given leave by the AIs to grow up arbitrarily far in a broad variety of self-directed ways, which the AIs decide to bestow for some reason I don’t currently see, so that eventually some originally-humans become peers of the AIs rather than specifically in charge, and so they won’t even be in the position to permanently disempower the other originally-humans if that’s somehow in their propensity.

I agree that in the long run, the AIs control everything in practice, and any human influence comes from the AIs being essentially a perfect delegator of human values, but I want to call out that you said that humans delegating to AIs who in practice do everything for the human, and the human is not in the loop as humans not being disempowered, but empowered, so even if AIs control everything in practice, so long as there’s successful value alignment to a single human, I’m counting scenarios like “the AIs disempowers most humans because the humans that encoded their values into the AI successfully don’t care about most humans once they are useless, and may even anti-care about them, but the people who successfully value aligned the AI (like labs and government people) live a rich life thereafter free to extend themselves arbitrarily” as disempowerment by humans:

Again, if they do execute on your values, including the possible preference for you to grow under your own rather than their direction, far enough that you are as strong as they might be, then this is not a world in a state of disempowerment as I’m using this term, even if you personally start out or choose to remain somewhat disempowered compared to AIs that exist at that time.

To return to the crux:

I think in human delegation, alignment is more important than verification. There is certainly some amount of verification, but not nearly enough to prevent sufficiently Eldritch reward hacking, which just doesn’t happen that often with humans, and so the society keeps functioning, mostly. The purpose of verification on the tasks is in practice more about incentivising and verifying alignment of the counterparty, not directly about verifying the state of their work, even if it does take the form of verifying their work.

I think this is fairly cruxy, as I think alignment matters much less than actually verifying the work, and in particular I don’t think value alignment is feasible at anything like the scale of a modern society, or even most ancient societies, and the biggest changes of the modern era compared to our previous eras is that institutions like democracy/capitalism depend much less on the values of the humans that make up their states, and much more so on the incentives you give to the humans.

In particular, most delegation isn’t based on alignment, but based on the fact that P likely doesn’t equal NP and polynomial time algorithms in practice being efficient compared to exponential time algorithms, meaning there’s a far larger set of problems where you can verify an answer easily but not generate the correct solution easily.

I’d say human societies mostly avoid alignment, and instead focus on other solutions like democracy, capitalism or religion.

BTW, this is a non-trivial reason why the alignment problem is so difficult, because since we never had to solve alignment to capture huge amounts of value, it means there are very few people working on the problem of aligning AIs, and in particular lots of people incorrectly assume that we can avoid having to solve the problem of aligning AIs in order for us to survive, and you have a comment that explains things pretty well on why misalignment in the current world is basically unobtrusive, but when you give enough power catastrophe happens (though I’d place the point of no return at when you no longer need other beings to have a very rich life/other people are useless to you):

https://www.lesswrong.com/posts/Z8C29oMAmYjhk2CNN/non-superintelligent-paperclip-maximizers-are-normal#FTfvrr9E6QKYGtMRT

This is basically my explanation of why human misalignments don’t matter, but in a future where at least 1 human has value aligned an AGI to themselves, and they don’t intrinisically care about useless people, lots of people will die from the AI proximally, but the ultimate cause is the human value-aligning the AGI.

To be clear, we will eventually need value alignment at some point (assuming AI progress doesn’t stop), and there’s no way around it. But we may not need it as soon as we feared, and in good timelines muddle through via AI control for a couple of years.

Noosphere89 19 Jun 2025 1:14 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: tailcalled’s Shortform

So similarly here, “eutopia without complete disempowerment” but still with significant disempowerment is not in the “eutopia without disempowerment” bin in my terms. You are drawing different boundaries in the space of timelines.

Flag: I’m on a rate limit, so I can’t respond very quickly to any follow-up comments.

I agree I was drawing different boundaries, because I consider eutopia with disempowerment to actually be mostly fine by my values, so long as I can delegate to more powerful AIs who do execute on my values.

That said, I didn’t actually answer the question here correctly, so I’ll try again.

My expectation is more like model-uncertainty-induced 5% eutopia-without-disempowerment (I don’t have a specific sense of why AIs would possibly give us more of the world than a little bit if we don’t maintain control in the acute risk period through takeoff), 20% extinction, and the rest is a somewhat survivable kind of initial chaos followed by some level of disempowerment (possibly with growth potential, but under a ceiling that’s well-below what some AIs get and keep, in cosmic perpetuity). My sense of Yudkowsky’s view is that he sees all of my potential-disempowerment timelines as shortly leading to extinction.

My take would then be 5-10% eutopia without disempowerment (because I don’t think it’s likely that the powers in charge of AI development would want to give baseline humans the level of freedom that implies that they aren’t disempowered, and the route I can see to baseline humans not being disempowered is if we get a Claude scenario where AIs take over from humans and are closer to fictional angels in alignment to human values, but it may be possible to get the people in power to care about powerless humans, in which case my probability of eutopia without disempowerment), 5-10% literal extinction, and 10-25% existential risk in total, with the rest of the probability being a somewhat survivable kind of initial chaos followed by some level of disempowerment (possibly with growth potential, but under a ceiling that’s well-below what some AIs get and keep, in cosmic perpetuity).

Another big reason why I put a lot of weight on the possibility of “we survive indefinitely, but are disempowered” is I think muddling through is non-trivially likely to just work, and muddling through on alignment gets us out of extinction, but not out of disempowerment by humans or AIs by default.

I think the correct thesis that sounds like this is that whenever verification is easier than generation, it becomes possible to improve generation, and it’s useful to pay attention to where that happens to be the case. But in the wild either can be easier, and once most verification that’s easier than generation has been used to improve its generation counterpart, the remaining situations where verification is easier get very unusual and technical.

Yeah, my view is in the wild verification is basically always easier than generation absent something very weird happening, and I’d argue verification being easier than generation explains a lot about why delegation/the economy works at all.

A world in which verification was just as hard as generation, or verification is harder than generation is a very different world than our world, and would predict that delegation to solve a problem basically totally fails, and everyone would have to create stuff from scratch rather than trading with others, which is basically entirely the opposite of how civilization works.

There are potential caveats to this rule, but I’d argue if you randomly sampled an invention across history, it would almost certainly be easier to verify that a design works compared to actually creating the design.

(BTW, a lot of taste/research taste is basically leveraging the verification-generation gap again).

Noosphere89 18 Jun 2025 17:21 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: tailcalled’s Shortform
My take is that the most likely outcome is still eutopia-with disempowerment for baseline humans, but for transhumans I’d expect eutopia straight-up.

In the long-run, I do expect baseline humans to be disempowered pretty much totally, similar to how children are basically disempowered relative to parents, but the child won’t grow up and will instead age in reverse, or how pets are basically totally disempowered relative to humans, but humans do care for pets enough that pets can live longer, healthier lives, and for specifically baseline humans, the only scenarios where baseline humans thrive/survive are regimes where AI terminally values baseline humans thriving, and value alignment determines everything on how much baseline-humans survive/thrive.

That said, for those with the wish and will to upgrade to transhumanism, my most likely outcome is still eutopia.

For me, a crux of a future that’s good for humanity is giving the biological humans the resources and the freedom to become transhuman beings themselves, with no hard ceiling on relevance in the long run.

I think this is reasonably plausible, though not guaranteed even in futures where baseline humans do thrive.

The probabilities on the scenarios, conditional on AGI and then ASI being reached by us, is probably 60% on eutopia without complete disempowerment, 30% on complete disempowerment by either preventing us from using the universe to killing billions of present day humans, and 10% on it killing us all.

The basic reasoning for this is I expect AGI/ASI to not be a binary, even if it does have a discontinuous impact in practice, and this means that muddling through instruction following is probably enough in the short term, and in particular I don’t expect takeoff to be supremely fast, in that I expect a couple of months at least from “AGI is achieved” to AIs run the economy and society, and relevantly here I expect physical stuff like inventing bio-robots/nanobots that can replace human industry more efficiently than current industries to come really late in the game where we have no more control over the future:

https://www.lesswrong.com/posts/xxxK9HTBNJvBY2RJL/untitled-draft-m847#Cv2nTnzy6P6KsMS4d

Heck, it likely will take a number of years to get to nanotech/biotech/smart materials/metamaterials that can be mass produced, and this means that stuff like AI control can actually work.

The other point of optimism is I believe verification is easier than generation in general, which means I’m much more optimistic on eventually delegating AI alignment work to AIs, and I think that slop will be much reduced for early transformative AIs.

This is why I remain optimistic relative to most LWers, even if my p(doom) increased.

Noosphere89 18 Jun 2025 16:10 UTC
9 points
−3
in reply to: tailcalled’s comment on: tailcalled’s Shortform
As someone who used to believe in this, I no longer do, and a big part of my worldview shift comes down to me thinking that LLMs are unlikely to remain the final paradigm of AI, and in particular the bounty of data that made LLMs as good as they are is very much finite, and we don’t have a second internet to teach them skills like computer use.
And the most accessible directions after LLMs involve stuff like RL, which puts us back into the sort of systems that alignment-concerned people were worried about.
More generally, I think the anti-scaling people weren’t totally wrong to note that LLMs (at least in their pure form) had incapacities that at realistic levels of compute and data prevent them from displacing humans at jobs, and the incapacities are not learning after train-time in weights (in-context learning is very weak so far), also called continual learning, combined with LLMs just lacking a long-term memory (best example here is the Claude Plays Pokemon benchmark).
So this makes me more worried than I used to, because we are so far not great at outer-aligning RL agents (seen very well in the reward hacking o3 and Claude Sonnet 3.7 displayed), but the key reasons I’m not yet persuaded to have an extremely high p(Doom) of people like Eliezer Yudkowsky or Nate Soares is that I expect the new paradigm shifts to be pretty continuous, and in particular I expect labs to release a pretty shitty version of continual leaning before they release continual learning AIs that actually can take jobs.
Same goes for long-term memory.
So I do disagree with @Thane Ruthenis’s claim that general intelligence/AGI is binary, even if in practice the impact from AI is discontinuous rather than continuous in the post below:
https://www.lesswrong.com/posts/3JRBqRtHBDyPE3sGa/a-case-for-the-least-forgiving-take-on-alignment
LLMs would scale into outright superintelligence at the limit of infinite compute and data, for basically the reason Eliezer Yudkowsky said below, but jbash isn’t wrong to note that there’s no reason to believe that the limit will ever be well approximated by near-future LLMs, so the abstract arguments of LLMs being very powerful if scaled unfortunately run into the wall of “we don’t have that much data or compute necessary to scale LLMs to levels where Eliezer is approximately correct”, similarly to AIXI.
https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators
https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators#Aunc7qTiKgEBkKmvM
And that’s a big shame, since LLMs are basically the most alignable form of AI we’ve gotten so far, so unfortunately capability improvements will make AIs less safe by default, though a lot of my remaining hopes rests on AI control + the possibility that as AI capabilities get better, we really do need to get better at specifying what we want in ways that are relevant to AI alignment.
The other good news is this makes me more bearish on extremely short timelines like us getting AGI by 2027, though my personal median is in 2032, for what it’s worth.

Noosphere89 18 Jun 2025 1:38 UTC
2 points
0
in reply to: Thane Ruthenis’s comment on: Acausal normalcy
My take is that that unfortunately, I don’t expect high-level order/useful abstractions for acausal trade in full generality, and more generally suspect that in the general case, you really do need full-blown simulation that is perfectly accurate to get the benefits of acausal trade with arbitrary partners.

I’m also skeptical of boundaries even in practice working out in any non-trivial sense after we develop AI that can replace all humans at jobs, and I think boundaries IRL come about because there is no party or group that can penetrate people’s boundaries without them raising hell and damaging/destroying you, but this does not apply very much to AI/human interaction, and I’m much more skeptical of boundaries existing at an ontological level than Andrew Critich is (which is my takeaway from the Embedded Agency Sequence).

Another area where I’m skeptical is the claim that human morality is best explained by acausal trade, rather than causal trade/the amount of energy used plus the fact that humans need other humans to survive, meaning you actually need to take into account what other beings prefer.