Noosphere89

Karma: 4,426

Noosphere89 1 Feb 2026 14:03 UTC
2 points
0
in reply to: Mo Putera’s comment on: Eli’s shortform feed
The nuance was in saying that their framework can’t predict whether or not data or compute scaling made the majority of improvements, nor can they separate out data and compute improvements, but the core finding of algorithmic efficiency being almost all compute-scaling dependent still holds, so if we had a fixed stock of compute now, we would essentially have 0 improvements in AI forever.

Noosphere89 31 Jan 2026 21:43 UTC
5 points
−2
in reply to: Eli Tyre’s comment on: Eli’s shortform feed
Another possibility, in principle, for why automating AI R&D doesn’t lead to an intelligence explosion is because a very large percentage of the progress (at that part of development trajectory) is driven by scaling relative to algorithmic progress.
This is actually happening today, so the real question is why algorithmic progress returns will increase once we attempt to fully automate AI R&D, rather than why we won’t get an intelligence explosion.
More specifically, the algorithmic progress that has happened is basically all downstream of more compute going into AI, and algorithmic efficiency is dependent on compute scale being larger and larger to reap the gains of better algorithms.

Noosphere89 29 Jan 2026 17:39 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: Daniel Kokotajlo’s Shortform
Just LLMs with pretraining, giving superintelligent in-context learning, making AIs able to figure out how to work around their hobblings using just in-context reasoning. This clearly doesn’t work at 2024 levels of compute (which only got to be observed in 2025), and after 2026 levels of compute natural text data starts running out. Maybe there are some relevant sparks at 2029-2031 levels of compute (5 GW), but probably not sufficient on its own.
I want to flag that you estimated that post-2028, we’d slow down due to the MoE data wall, and while MoE is more compute efficient, I don’t think it’s so much more compute efficient that people won’t take the hit of less compute efficiency if data efficiency becomes more relevant post-2028, and we can probably continue mostly pre-training up until the early 2030s based on data limitations alone. That said, I agree this is likely insufficient on its own, so I’m not claiming that pure LLM AGI is a likely path.
Continual learning doesn’t seem directly relevant, but maybe it helps with teaching LLMs to RLVR LLMs. It doesn’t seem directly relevant because it probably doesn’t directly train mildly superhuman capabilities. But since the generalizing glue of adaptation would make everything work better, it might enable method (2) to go forward where it would fail on its own. The relevant benchmarks might be the same as for (2), but only if they don’t already saturate by the time there’s useful continual learning (in some nontrivial sense of the term).
I think continual learning is more likely to end up important in its impact on AI company TAM. If it manages to 10x it, then it will thereby 10x the compute, though it’ll proably take until 2030-2035 to actually build the kind of research/training compute (20-50 GW) that a trillion dollars of revenues buys (even if 2026 already shows that this is on track to happen).
I think this is probably true for the near term (as in 5-10 year timelines), but over a 15-20 year span with Moore’s law, I’d be more confident that continual learning alone could plausibly train in mildly superhuman capabilities, for the same reasons why certain researchers IRL are way more productive than others.
This does depend on human level or better sample efficient learning to work out, though.
That said, I agree with your methodology much more than Daniel Kokotajlo’s methodology on AI prediction (though I do think METR time horizons are more useful and less anchored to specific paradigms than you say it is).

Noosphere89 27 Jan 2026 20:39 UTC
4 points
0
in reply to: Eli Tyre’s comment on: Wei Dai’s Shortform
It’s actually worse, in that almost all such simulators are hypercomputational and have literally infinite compute, memory and data, as well as their programs being infinitely large, so it’s literally useless to think/write with potential simulators as an audience.

Noosphere89 26 Jan 2026 22:05 UTC
6 points
1
in reply to: Vladimir_Nesov’s comment on: Dario Amodei – The Adolescence of Technology
The answer I suspect has to do with a focus on short timelines combined with more bearish views on software-only singularities than most of LW.

Short timelines means there’s a lot less compute resources to run AIs because Moore’s law hasn’t had as much of an effect compared to future dates, combined with less compute being built overall, meaning we can only run millions or at best tens of millions of AIs.

One thing I note also is that he does think that AIs can increase their intelligence, but that this runs into steep diminishing returns early enough that other factors matter more.

Noosphere89 25 Jan 2026 15:07 UTC
2 points
0
in reply to: Jan_Kulveit’s comment on: ricraz’s Shortform
Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do.
The bigger issue, as Jackson Wagner says is that there’s a very real risk that will be coopted by people who want to talk mostly about present-day harms of AI, and thus at best siphoning resources from actually useful work on gradual disempowerment threats/AI x-risk in general, and at worst creating polarization around gradual disempowerment with one party supporting gradual disempowerment of humans and another party opposing gradual disempowerment, while the anti-gradual disempowerment party/people become totally ineffective at dealing with the problem because it’s been taken over by omnicause dynamics.

Noosphere89 18 Jan 2026 16:20 UTC
2 points
0
on: Towards more cooperative AI safety strategies
To be honest, I now think the post has far less value than people thought in in 2024, and this is not because it’s factual statements are wrong, but rather that Richard Ngo and a lot of LWers way underestimated how intractable it was to avoid polarization of AI safety issues in worlds where AI safety is highly salient politically without deep changes to our electoral systems.

(To be clear, this is a US-centric view, but this is justified given that the by far most likely countries to have ASI in their borders initially are the US and China, due to both compute and researcher advantages over the rest of the world, and the US’s laws are more tractable to change for AI safety than China’s laws on AI).

The basic reason for this is that social sorting has fundamentally led to the breakdown of conditions that allowed low polarization without something like proportional voting/score voting, and this is a problem that AI safety people are ill-equipped to solve or even ameliorate significantly, unfortunately.

And this means absent electoral reform, we should assume by default as AI safety becomes higher salience, it will become more polarized, and this means there’s a tradeoff between making AI higher salience to achieving political goals.

I’m giving it a 0 for this reason, because in the end the post was basically irrelevant to how AI safety should act, and how to handle the tradeoff of polarization vs getting enough salience to get laws passed is unfortunately more complicated than the post makes it out to be, and much tougher to solve.

Noosphere89 16 Jan 2026 14:55 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I agree with this, but at that point philosophical dilemmas don’t matter for alignment purposes, because from an alignment perspective you only need to (ideally) have agreement between the AI and human on the answers, and the answers to philosophical dilemmas don’t actually matter (at least for value-laden philosophical dilemmas).
I agree with Eliezer on the hard part of the alignment problem:
Ebenzer Dukais’s comment
So philosophy being hard is not a valid argument for AI alignment being hard.
That said, I do think philosophical dilemmas like this (as well as trying to steer us towards moral-trade futures) is a big reason I’ve come to think of space governance as important, neglected and (for now) much more tractable than basically everything else with the same level of importance or higher.
More generally, one very important priority is to move ourselves away from a regime where whoever makes a unilateral claim and seizes land in space owns it, and we will need to prohibit interstellar settlement for a time (because otherwise due to light-speed limitations, it becomes effectively impossible for the central government to catch up with them).
Thankfully, this is far easier than ending the AI race for now.

Noosphere89 12 Jan 2026 1:28 UTC
7 points
1
on: Preventing model exfiltration with upload limits
This is easily the most underrated AI security post of 2024, mostly because it points out that, unlike most files a defender has, AI weights are very, very large files, and that this means you can slow down exfiltration attempts enough to require physical attacks to be used.
It also inspired follow-up work like Defending Against Model Weight Exfiltration Through Inference Verification to achieve the goal of preventing model exfiltration.
The big question that will need to be answered is whether or not preventing exfiltration of model weights is positive EV, and I currently think it probably is, though there are reasonable arguments that state-proof or even state-resistant security worsens things via making the AI race more safe to engage in, and especially in worlds where long pauses are necessary, security could plausibly be negative, but there’s no way to get robustly positive actions on AI, so you have to use assumptions that could fail.
And at any rate, this is not relevant for the review at hand.
Ryan Greenblatt made a comment that reasoning models have made AI models smaller, but we now know that current RL/test time compute and reasoning was a one-time boost, and we will almost certainly see scaleups of AI models starting in 2026.
This becomes even more impactful if we believe that for the first AGIs, their files will be around the order of magnitude of human brain memory, which is 2.5 petabytes, which makes exfiltration much harder.
+9 for popularizing an idea so good that there’s a realistic chance we could jump from Security Level 2 to Security Level 4 with one idea, where we become resistant to state attacks (and if side effects turn out not to be a viable means of exfiltrating models, we could jump to Security Level 5, which requires defense against top nation states).
What links here?
- Deeper Reviews for the top 15 (of the 2024 Review) by Raemon (14 Jan 2026 23:59 UTC; 45 points)

Noosphere89 10 Jan 2026 0:00 UTC
4 points
1
in reply to: StanislavKrym’s comment on: [Linkpost] On the Origins of Algorithmic Progress in AI
As someone who has written that post, I think the title nowadays over claims, and the only reason I chose the title was because of the fact that it was used originally.

I’d probably argue that it explains a non-trivial amount of progress, but nowadays I’d focus way more on compute being the main driver of AI progress in general.

And yes, new architectures/loss functions could change the situation, but the paper is evidence that we should expect these architectures to rely on a lot of compute to be more efficient.

Noosphere89 8 Jan 2026 20:24 UTC
3 points
0
in reply to: MP’s comment on: AI Futures Timelines and Takeoff Model: Dec 2025 Update

a- Pretraining scaling has slowed down massively since the GPT-4.5 debacle.

This is the one element of the comment that doesn’t really stand up, because new data centers that are much larger are being constructed, and the GPT-4.5 debacle was near entirely because we compared GPT 4.5 to o3 with RL, as well as compute being scaled 10x instead of 100x.

Pre-training is still going strong, it’s just rested a bit due to the crazy RL scaleup, and it will come back in importance (absent continual learning).

This implies that their trend up to 2030 is likely accurate, but post-2030 absent new paradigms will look a lot different than the median scenario in their model.

Noosphere89 7 Jan 2026 18:36 UTC
1 point
0
on: The Technology of Liberalism
My own take on this discussion is that liberalism/democracy is not a normal state, and while liberalism/democracy will be historically important in the post-AGI era, if we seriously take the assumption that labour-replacing AI has a high chance of being developed in the next few decades, then liberalism likely doesn’t survive.
A key crux in my thinking is that to a large extent, liberalism has been as powerful as it was because it turned out that values and power-seeking were much more correlated than the orthogonality thesis suggests could happen in AIs, and it turned out that technological development broadly speaking in the last couple of centuries has reaped way, way more rewards for liberalism than virtually any other value system.
Most people do not care about liberalism and would almost certainly trade away control over politics for convenience (or even the promise of convenience), and this is a takeaway that was strengthened from the 2024 election results in the US.
The compact generator of technological development favoring particular values way more than others, is that pretty much all technology for the last couple of centuries has required ever-increasing logistics + trade to work well, combined with technology amplifying power instead of allowing you to create new power from thin air.
But AGI + robotics + nanotech let you both reduce the logistics needs of states/corporations massively, combined with the ability to no longer depend on the former population for logistics needs, plus the remainder of logistics problems are dealt with superhuman efficiency and in the cases where we do align AI (which looks pretty likely to me), you no longer have to deal with coordination/agency costs.
It also creates power rather than merely amplifying power.
This fundamentally disadvantages liberalism relative to it’s prior position, where it’s now competing on equal ground with other value systems.
And decentralization doesn’t save us, since the technologies that drive autocratization are convergent, combined with most people not caring as much about freedom as much as they care about convenience, so a decentralized world is a world with at best a super-majority of benevolent autocracies.
One area where I’m contrarian on relative to everyone else is that this is probably not going to be nearly as bad as liberals fear, mostly because there are systematic differences between historical autocracies and post-AGI ones that shift the distribution towards more favorable outcomes, but the main positive differences is that people are (slightly) altruistic/care about others, and that this has less diminishing returns compared to their selfish wants, even accounting for The Elephant In The Brain which says we are much less altruistic and more selfish than we pretend ourselves to be, which matters given the abundance of resources we will have in the future, combined with selection effects being much less potent in autocracies that force dictators to choose incompetents because they are loyal, as well as the potential distribution of dictators being much, much better than past distributions (since the fraction of benevolent dictators is 1 in 3-5, which is very, very impressive compared to historical norms).
Especially in worlds where democracy is intractable to preserve and AI pause/slowdowns are difficult to do, we might have to chance benevolent dictatorship.
One final point: I have a pretty large disagreement with the following as it relates to AI:
1. There is a lot of path dependence in the development of technologies, and effects like Wright’s law mean that things can snowball: if you manage to produce a bit, it gets cheaper and it’s better to produce a larger amount, and then an enormous amount. This is how solar panels went from expensive things stuck on satellites to on track to dominate the global energy supply. Tech, once it exists, is also often sticky and hard to roll back. A trickle you start can become a stream. (And whether or not you can change the final destination, you can definitely change the order, and that’s often what you need to avert risks. All rivers end in the ocean, but they take different paths, and the path matters a lot for whether people get flooded.)
More specifically, I’m disagreeing with the view that Wright’s Law lets us shape technology for AI very well, and the reason for this is that the primary drivers of progress were compute and data (arguably misleadingly called algorithmic progress), and we incorrectly assumed that powerful AI would (initially) be able to be run with small scales of compute/data available to academics.
A key turning point that, in retrospect, foreshadowed the Deep Learning revolution was the creation of (large at the time) tens of millions of parameters with several GPUs, as it showed that the scaling hypothesis could be reasonable.
Nowadays, Richard Sutton’s scaling hypothesis has massively more evidence behind it, and while there are certainly algorithmic/loss function challenges, as Adam Marblestone states, it’s now pretty clear that Hans Moravec was much closer to correct when he focused on the compute being necessary than any of his contemporaries.

Noosphere89 5 Jan 2026 16:17 UTC
0 points
−3
in reply to: Thomas Kwa’s comment on: AI Futures Timelines and Takeoff Model: Dec 2025 Update
It seems to me like a time horizon of 3 years or 125 years is complete overkill for automation of enough coding for the bottleneck to shift to experiments and research taste.
My small comment on this is that this is mostly fine if you take the worldview that tacit knowledge matters and the inflated time horizon is to make sure that tacit knowledge needs are adequately modeled here.
Steve Newman has a good discussion on this here.

Noosphere89 2 Jan 2026 17:26 UTC
3 points
1
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
Basically agree with this in the near term, though I do think in the longer term, especially in the 2030s, continual learning will bring the dangers of AGI, and probably will lead to faster takeoffs than purely LLM-based takeoff worlds.
But yes, for at least the next 5 years, continual learning will differentially wake up the world to AGI without bringing the dangers of AGI, but unlike many on here, I don’t expect it to lead to policy that lets us reduce x-risk from AI much for the reasons Anton Leicht states here, but in short form, even if accelerationist power declines, this doesn’t necessarily mean AI existential safety can take advantage of it, and AI safety money will decline as a percentage compared to money for various job protection lobbies, and while accelerationists won’t be able to defeat entire anti-AI bills, it will still remain easy for them to neuter AI safety bills enough to make the EV of politics for reducing existential risk either much less than technical AI safety, or even outright worthless/negative depending on the politics of AI.
There’s a Dwarkesh quote on continual learning that I really want to emphasize here:
“Solving” continual learning won’t be a singular one-and-done achievement. Instead, it will feel like solving in context learning. GPT-3 demonstrated that in context learning could be very powerful (its ICL capabilities were so remarkable that the title of the GPT-3 paper is ‘Language Models are Few-Shot Learners’). But of course, we didn’t “solve” in-context learning when GPT-3 came out—and indeed there’s plenty of progress still to be made, from comprehension to context length. I expect a similar progression with continual learning. Labs will probably release something next year which they call continual learning, and which will in fact count as progress towards continual learning. But human level continual learning may take another 5 to 10 years of further progress.

Noosphere89 2 Jan 2026 3:31 UTC
3 points
0
on: Open Thread Winter 2025/26
Now that it is the New Year, I made a massive thread on twitter concerning a lot of my own opinionated takes on AI, which to summarize are my lengthening timelines, which correlates to my view that new paradigms for AI are likelier than they used to be and more necessary, which reduces AI safety from our vantage point in expectation, AI will be a bigger political issue than I used to think and depending on how robotics ends up, it might be the case that by 2030 LLMs are just good enough to control robots even if their time horizon for physical tasks is pretty terrible, because you don’t need much long term planning, which would make AI concern/salience go way up, though contra the hopes of a lot of people in AI safety, this almost certainly doesn’t let us reduce x-risk by much, for reasons Anton Leicht talks about here, and many more takes in the full thread above.
But to talk about some takes that didn’t make it into the main Twitter thread, here are some to enjoy:
1. My current prediction is LLMs top out at somewhat above the capability of a superhuman coder/automated coder as defined by AI 2027/AI Futures Model absent other forces, as this is when constraints on data begin to bind more strongly than they have, slowing down scaling by potentially a lot more absent new paradigms, though it might take until 2032 for the entire data stock to run out.
2. Most, if not all, futures that are good from a human value perspective and where AIs take over inevitably involve value lock-in/massively slowing down the rate of evolution, because otherwise it’s likely that very bad outcomes for humanity arise. This is relatively easy to do, but quite important to do for human survival, especially in worlds where theory-backed unbounded alignment doesn’t happen in time.
3. After AGI, power concentration leading to 1980s-2010s China-type government/economics but mixed in with feudalistic elements across the globe, is likely to happen absent massive changes in politics, mainly because AGI plus nanotech lets values and technology/economic systems decouple way more than they currently are. Liberal democracy is dependent on the fact that growth/national power is dependent on giving political freedom to the masses. China only managed to decouple democracy from capitalism for 3 decades, meaning that the orthogonality thesis is false from a large-scale perspective. However, AGI + nanotech pretty much allows you to not give freedom to almost every citizen while still being able to grow economically. That said, as I stated in the Twitter thread, I don’t think this is likely to be a bad thing, because most of the reasons historical dictatorships suck are due to a combination of selection effects for who gets to be powerful, combined with the fact that even a nominal dictator doesn’t rule alone means they have to select people, and the incentives in dictatorship are to select loyal people instead of competent people, but conditional on alignment being solved, this just goes away, and the fact that human returns diminish sufficiently much that selfish preferences are massively easy to satisfy with galaxies at your disposal, but not current economies meaning that it’s very easy for even mild altruistic preferences to dominate and let the citizens have quality of life unheard of in democracies like the US.
4. I currently think space governance is unusually underrated in terms of funding/talent relative to it’s attention, and in particular reducing the incentives to rush to grab all the resources in space after AGI is really important, but contra people like Jordan Stone, I believe that even a race to the stars because of AGI almost certainly doesn’t cause an existential risk with probability of more than 0.0000001% percent, and the reason I am this confident is that pretty much all of the sources of x-risk either are too slow-acting to overwhelm defense systems, or rely on physics getting overturned in a way that loosens constraints, which has not been the norm for science progress, and importantly a lot of threat models for how physics alterations to our universe could cause existential risk require huge resources like (stars/galaxies) even for technologically mature civilizations, and superintelligence makes it very easy to coordinate to prevent existential risks that arise from somehow altering physics (since AI evolution is likely massively slowed down and there are likely going to be at most 3-5 AI factions off of our current economic incentives).
More generally, once you are able to go into space and create enough ships such that you can colonize solar systems/galaxies, your civilization is immune to existential threats that rely solely on our known physics, which is basically everything that isn’t using stellar/galactic resources, and this vastly simplifies the coordination problems compared to coordination problems here on Earth.
I instead want space governance to be prioritized more for 2 reasons:
1. To make moral trade-based futures on our universe more likely, relative to other outcomes. My current view is that given the likelihood of moral relativism (where there’s an infinite number of correct moral views) being rather high in my view relative to moral objectivism, combined with the variability of human values being also large even in the infinitely far future, this means that moral trade becomes much more important in my view to get good outcomes for most humans with an AI-dominated world, relative to a free-for-all regime that locks in wealthy people’s values in power.
2. To figure out whether or not a theory of multi-agency exists and is coherent, as an instrumental goal to figure out whether acausal trade exists, which if feasible would be quite good for our civilization to do, and this is true even with my low probability of x-risk from space travel that leads to free for all grabbing of reasources, because coordination on a galactic/universal scale would fundamentally allow for certain mega-projects to be done, especially mega-projects that change physics, and it also lets us simplify coordination problems drastically, making x-risk negligible even over the lifetime of the universe.
These are my takes for New Years today.
What links here?
- Noosphere89's comment on Noosphere89′s Shortform by Noosphere89 (30 Dec 2024 1:39 UTC; 6 points)

Noosphere89 27 Dec 2025 17:04 UTC
13 points
3
on: Moving Goalposts: Modern Transformer Based Agents Have Been Weak ASI For A Bit Now
A couple of things happened that made terms like AGI/ASI less useful:
1. We didn’t realize how correlated benchmark progress and general AI progress was, and benchmark tasks systematically sample from disproportionately easy to automate areas of work:
Not a novel point, but: one reason that progress against benchmarks feels disconnected from real-world deployment is that good benchmarks and AI progress are correlated endeavors. Both benchmarks and ML fundamentally require verifiable outcomes. So, within any task distribution, the benchmarks we create are systematically sampling from the automatable end.
Importantly, there’s no reason to believe this will stop. So we should expect benchmarks will continue to feel rosey compared to real-world deployment.
Thane Ruthenis also has an explanation about why benchmarks tend to overestimate progress here.
2. We assumed intelligence/IQ was much more lower-dimensional than it really was. Now I don’t totally blame people for thinking that there was a chance that AI capabilities were lower dimensional than they were, but way too many expected an IQ analogue for AI to work. This is in part an artifact of current architectures being more limited on some key dimensions like continual learning/long-term memory, but I wouldn’t put anywhere close to all of the jaggedness on AI deficits, and instead LWers forgot that reality is surprisingly detailed.
Remember, even in humans IQ only explains 30-40% of human performance, which while being more than a lot of people want to admit, nerd communities like LessWrong have the opposite failure mode of believing that intelligence/IQ/capabilities is very low dimensional with a single number dominating how performant you are.
To be frank, this is a real-life version of an ontological crisis, where certain assumptions about AI, especially on LW turned out to be entirely wrong, meaning that certain goal-posts/risks have turned out to at best require conceptual fragmentation, and at worst turned out to be incoherent.

Noosphere89 25 Dec 2025 17:06 UTC
6 points
0
on: Near-mode thinking on AI
This is still a quite good post on how to think about AI in the near-term, and the lessons generalize broadly beyond even the specific examples.

The main lessons I take away from this post are these:
1. Take AI jaggedness seriously, and don’t expect AI complete problems (yes, continual/online learning will reduce the jaggedness, but I do not expect AI to eliminate jagged capabilities).
Summed up as “often the first system to do X will not be the first systems to do Y”.
1. Be careful of relying too much on abstraction, and remember that reality is surprisingly detailed (which explains why AI capabilities are in part as jagged as they are).
Summed up by this quote:

But knowing only that superintelligences are really intelligent doesn’t help with designing the scheming-focused capability evaluations we should do on GPT-5, and abstracting over the specific prerequisite skills makes it harder to track when we should expect scheming to be a problem (relative to other capabilities of models).[1] And this is the viewpoint I was previously missing.
1. Being better than a human at something, or more generally being better than someone at something is way more mundane than fictional portrayals of superheroes/AIs are (this is usually a consequence of the human tendency to assume discontinuities everywhere combined with rules in fiction often being way more ad-hoc than in our reality.)
And this matters because this should update us towards believing that we can delegate/automate quite a lot of alignment research in the critical period, meaning that our chance at surviving AI is higher than often assumed.

I will say that there is a caveat in that I now believe that persuasion/epistemics is one of the few areas where I expect much more discontinuous changes, but that’s going to be discussed below when I discuss my use of reacts on the post.
1. We should take AI safety automation more seriously. This is admittedly a thing LWers have gotten a lot better on, but it was somewhat needed at the time.
I do admit that this is quite continuous with the 3rd lesson, so I’m not going to dwell on details here.

Just know that AI automation of AI safety shouldn’t be dismissed casually.

Now that I’m done listing off lessons, I want to talk about my use of reacts.

I used 5 reacts on the post, and I missed the mark on 2 reacts, that being the hit the mark react on persuasion and using the important react on a study.

The main reason I missed the mark here is that I now believe that the study only shows that it’s easy for AIs to persuade people for a short time when the topic isn’t salient, and unfortunately most of the high-value epistemic applications will make topics salient, meaning that persuasion is way harder than people believe it is.

Persuasion/Epistemics is one of the few domains where I expect strongly discontinuous progress, but due to the hardness of persuasion I now think that AI persuasion is much less of a threat than I used to think (in the regime where our choices actually matter for AI risk), and this makes me more optimistic than I used to be around trusting AI outputs even in domains where tasks are hard to verify, as humans are very, very hard to persuade.

(A good book on this is Not Born Yesterday: The Science of Who We Trust and What We Believe by Hugo Mercier.)

I’d give this a +4 vote. It’s good and important for the near-term, and while not the most important for AI, it’s still a pretty good collection of ideas (though persuasion capability will increase far more discontinuously than the author claims).

Noosphere89 21 Dec 2025 15:30 UTC
2 points
0
in reply to: Oliver Daniels’s comment on: 2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target
The weaker version of pure reward optimization in humans is basically just the obesity issue, and the biggest reason humans became more obese on a population level in the 20th and 21st century is because we have figured out how to goodhart human reward models in the domain of food such that sugary, fatty and high-calorie foods (with the high-calorie part being most important for obesity) are very, very highly rewarding to at least part of our brains.

Essentially everything has gotten more palatable and rewarding to eat than pre-20th century food.

And as you say, part of the issue is that drugs are very crippling for capabilities, whereas reward functions for AIs that are optimized will have much less of this issue, and optimizing the reward function likely makes AIs more capable.

Noosphere89 14 Dec 2025 20:25 UTC
5 points
0
in reply to: interstice’s comment on: 6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa
An underrated answer is that humans are very, very dependent on other people to survive, and we have easily the longest childhood where we are vulnerable of any mammal, and even once we do become an adult, we are still really, really bad at surviving on our own compared to other animals, and since we are K-selected, every dead child mattters a lot in evolution, so it’s very, very difficult for sociopathy to be selected for.

Noosphere89 12 Dec 2025 21:52 UTC
2 points
−2
on: New 80k problem profile: extreme power concentration
(Crossposted from EAF)
Nice write-up on the issue.
One thing I will say is that I’m maybe unusually optimistic on power concentration compared to a lot of EAs/LWers, and the main divergence I have is that I basically treat this counter-argument as decisive enough to make me think the risk of power-concentration doesn’t go through, even in scenarios where humanity is basically as careless as possible.
This is due to evidence on human utility functions showing that most people have diminishing returns on utility on exclusive goods to use personally that are fast enough that altruism matters much more than their selfish desires on stellar/galaxy wide scales, combined with me being a relatively big believer in quite a few risks like suffering risks being very cheap to solve via moral trade where most humans are apathetic on.
More generally, I’ve become mostly convinced of the idea that a crucial positive consideration on any post-AGI/ASI future is that it’s really, really easy to prevent most of the worst things that can happen in those futures under a broad array of values, even if moral objectivism/moral realism is false and there isn’t much convergence on values amongst the broad population.
Edit: I edited in a link.