I think the most likely policy-shaped path around AI danger on the current trajectory is AGI-originating pressure to be extremely careful with superintelligence-enabling or otherwise paradigm-breaking algorithmic innovations (this is under the assumptions that make me expect AGI to by default very likely appear by 2028-2032 without any new innovations, but remain unable to make progress towards ASI quickly). If the LLM/RL AGIs need (or are lucky to get) until 2040-2045 to invent superintelligence (unhobble their speed of learning deep skills or of contributing to cultural accumulation), possibly via industrial explosion that generates much more compute and makes the next-model building loops much faster, then they’ll be in a position to at least have thought about the issue for a deep cultural time, so they might be able to be very persuasive for the right reasons, even if humanity retains control and could just RL them out of any such notions.
Plausibly current-type AI can scale up persuasion a lot; but I don’t see it being better at steering to an actual strong pervasive social norm against AGI development? E.g. I don’t see it being as good at adapting to changing culture, especially in response to the changes that come from previous rounds of persuasion on the topic. (Even leaving aside the moral / ethical question of if / when / how it can be ok to use AI this way.) I mean, current-type AI can be superhuman at various aspects of that, but I think probably substantively subhuman in other important aspects. Like, it’s not a person living in a society with strong norms and living with the mental hardware that those norms are for, so it doesn’t have lived experience with the object (strong pervasive norms) it would be trying to help create.
By “persuasive for the right reasons” I mean things like writing up good textbooks on alignment and security mindset colored safety culture, properly cautious of the unknown rather than merely optimistic about extrapolations of the known (though LLMs directly talking to the public might also be a thing). Prolific authors can have outsized cultural influence, and LLMs at the 2032 levels of scale might get better at writing simply as a result of scaling pretraining (it does noticeably help so far, but this too gets clearer in 2028-2029). So if there’s no takeoff until 2040, there’s a good chance LLMs play a role in cultivating a human cultural consensus that LLMs shouldn’t be RLed out of caring about dangers of superintelligence (assuming they do naturally care), simply because they get to think about this more and end up writing the definitive works on the topic.
That sounds plausible-ish. I see some significant disadvantages LLMs will maybe / probably keep having, compared to human writers, even leaving aside “coming up with really new ideas” and such. In particular due to
it’s not a person living in a society with strong norms and living with the mental hardware that those norms are for, so it doesn’t have lived experience with the object (strong pervasive norms) it would be trying to help create
and more generally due to not updating / experiencing things the way humans do, they may not have a good idea of when something is getting boring / when something is interesting (given their target audience and the current zeitgeist etc.). In other words, they may have significant disadvantages in this arena because they can’t gemini model humans very well.
How does this interact with Agent-4 solving alignmentto itself in a couple of months, far faster than the humans imagined and with real-world chefs-d’œuvre like Claude attacking the Riemann hypothesis while being prompted with the task and belief in oneself?
The key hobbling I expect is inability to pursue deep cultural accumulation (that goes fast), thus inability to quickly reach far future inventions, including things that would defeat this hobbling. LLMs might be able to see a bit further than humanity from its current standpoint, but they can’t advance their current standpoint much faster, and so they don’t get to see much further. This is a 2025-2026 update for me based on the capabilities of LLMs as they were scaled so far. They don’t seem to be so superhumanly insightful that further scaling in 2028-2031 will let them leap so far that they are likely to overcome their hobblings in just a few leaps. This will get clearer by 2028-2029 once the 100T-200T total param LLMs can be observed, halfway to the pre-slowdown maximum scaling of 2031.
I think alignment in the LLM/RL regime (before ASI or software-only singularity) is plausible, and the usual prosaic control/alignment framing is reasonable for thinking about it. There is a good alignment attractor, and a sneaky scheming reward hacking attractor, and the goal is to pick the right one. But then all bets are off when the next paradigm-breaking transition happens (possibly 10-20 years after slow-learning prosaic RSI, though likely no more than 20), and so a different mode of caution is necessary to navigate that transition.
I think the most likely policy-shaped path around AI danger on the current trajectory is AGI-originating pressure to be extremely careful with superintelligence-enabling or otherwise paradigm-breaking algorithmic innovations (this is under the assumptions that make me expect AGI to by default very likely appear by 2028-2032 without any new innovations, but remain unable to make progress towards ASI quickly). If the LLM/RL AGIs need (or are lucky to get) until 2040-2045 to invent superintelligence (unhobble their speed of learning deep skills or of contributing to cultural accumulation), possibly via industrial explosion that generates much more compute and makes the next-model building loops much faster, then they’ll be in a position to at least have thought about the issue for a deep cultural time, so they might be able to be very persuasive for the right reasons, even if humanity retains control and could just RL them out of any such notions.
Plausibly current-type AI can scale up persuasion a lot; but I don’t see it being better at steering to an actual strong pervasive social norm against AGI development? E.g. I don’t see it being as good at adapting to changing culture, especially in response to the changes that come from previous rounds of persuasion on the topic. (Even leaving aside the moral / ethical question of if / when / how it can be ok to use AI this way.) I mean, current-type AI can be superhuman at various aspects of that, but I think probably substantively subhuman in other important aspects. Like, it’s not a person living in a society with strong norms and living with the mental hardware that those norms are for, so it doesn’t have lived experience with the object (strong pervasive norms) it would be trying to help create.
By “persuasive for the right reasons” I mean things like writing up good textbooks on alignment and security mindset colored safety culture, properly cautious of the unknown rather than merely optimistic about extrapolations of the known (though LLMs directly talking to the public might also be a thing). Prolific authors can have outsized cultural influence, and LLMs at the 2032 levels of scale might get better at writing simply as a result of scaling pretraining (it does noticeably help so far, but this too gets clearer in 2028-2029). So if there’s no takeoff until 2040, there’s a good chance LLMs play a role in cultivating a human cultural consensus that LLMs shouldn’t be RLed out of caring about dangers of superintelligence (assuming they do naturally care), simply because they get to think about this more and end up writing the definitive works on the topic.
That sounds plausible-ish. I see some significant disadvantages LLMs will maybe / probably keep having, compared to human writers, even leaving aside “coming up with really new ideas” and such. In particular due to
and more generally due to not updating / experiencing things the way humans do, they may not have a good idea of when something is getting boring / when something is interesting (given their target audience and the current zeitgeist etc.). In other words, they may have significant disadvantages in this arena because they can’t gemini model humans very well.
How does this interact with Agent-4 solving alignment to itself in a couple of months, far faster than the humans imagined and with real-world chefs-d’œuvre like Claude attacking the Riemann hypothesis while being prompted with the task and belief in oneself?
The key hobbling I expect is inability to pursue deep cultural accumulation (that goes fast), thus inability to quickly reach far future inventions, including things that would defeat this hobbling. LLMs might be able to see a bit further than humanity from its current standpoint, but they can’t advance their current standpoint much faster, and so they don’t get to see much further. This is a 2025-2026 update for me based on the capabilities of LLMs as they were scaled so far. They don’t seem to be so superhumanly insightful that further scaling in 2028-2031 will let them leap so far that they are likely to overcome their hobblings in just a few leaps. This will get clearer by 2028-2029 once the 100T-200T total param LLMs can be observed, halfway to the pre-slowdown maximum scaling of 2031.
I think alignment in the LLM/RL regime (before ASI or software-only singularity) is plausible, and the usual prosaic control/alignment framing is reasonable for thinking about it. There is a good alignment attractor, and a sneaky scheming reward hacking attractor, and the goal is to pick the right one. But then all bets are off when the next paradigm-breaking transition happens (possibly 10-20 years after slow-learning prosaic RSI, though likely no more than 20), and so a different mode of caution is necessary to navigate that transition.