If you only care about AI takeover risk, it probably makes sense under the vast majority of conditions, with the only caveat being that even if ai model development is paused it’s still possible for AI takeover to occur just as a function of increasing AI integration into the economy.
If you care about other things, it doesn’t seem as clear.
Two things I quite care about are democracy and not dying of biological warfare. Even current levels of AI seem very destabilising to democracy, and very amenable to use by would-be-dictators; and while we’ve not yet seen an attack, I have little to no doubt that current open-weight models empower significantly weaker actors to create biological weapons than in the past (even if just through social manipulation).
If you’re anything like me, your values were probably formed in an environment which structurally favoured a very different set of qualities to the environment that we now live in.
With current SOTA models, and my larger than LessWrong-Average level of optimism about AI safety progress; it is no longer clear to me that an AI pause is net positive, even putting aside arms-race dynamics*.
*Sidenote: I’m much less Gung-ho about “racing the Chinese” to AGI than most LessWrong users. The majority of Chinese SOTA models appear to have been trained with distillation attacks, so I don’t think naive measures of how many months behind SOTA chinese companies are really apply.
My main issues with this framing are the following.
Could you explain why you have a “larger than LessWrong-Average level of optimism about AI safety progress”?
How exactly are SOTA models to be used by dictators, except for survelliance? The bioweapons, on the other hand, are indeed dangerous, but hard to manage and likely to overshoot and lead to escalation.
According to Zvi, “On traditional benchmarks one might be targeting, performance is impressive, on average around Opus 4.7. ” First of all, Z.ai had only sixty days of working with Opus 4.7 (or did they distill Opus 4.6 into a more capable model?) Additionally, the AI-2027 compute forecast had the unified Chinese lab accumulate twice less compute than the leading American lab, while the non-unified labs have 5-7 times less compute each. My main crux is how severely Chinese progress would be slowed down by a lack of access to Fable-like models, because parasitising on American HIGH-QUALITY output can in theory be replaced by using one’s own compute or American WEAK models to doublecheck YOUR outputs. And that’s ignoring China’s option to rush into the Dark Forest, which the US won’t do because they care about safety...
1) I might write this up later, but it’s a combination of being more confident in the robustness of current alignment techniques (under special conditions) than most less wrongers, and less pessimistic about the potential for deceptively misaligned automatic alignment researchers to sabotage alignment research (that was a mouth full!).
In short, I think current alignment techniques are robust enough that I think we can reasonably trust mildly superhuman automatic alignment researchers working in a sandbox environment to not take over the world.
Additionally, I suspect that the level of intelligence needed to solve alignment is significantly lower than the amount needed to take over the world (especially if your inference is slow, and your thoughts are big monitored). Under these conditions, I think it benefits even a quite badly misaligned ai to bargain with us in exchange for AI safety work.
Finally, I think that alignment algorithms are much lower dimensional than alignment data; so it’s unlikely that a misaligned agent would be able to sneak misalignment into the alignment algorithm, you just have to not let it produce synthetic data.
2) I think a combination of surveillance and censorship is probably enough to install yourself as forever dictator. Under these conditions, you can just kill individual rebels before they’re ever able to form a coherent anti-government bloc. I also think that ai degrading labour power makes democracy a less attractive bargain for oligarchs.
3) My assumption would be that 60 days is plenty of time for a company with the backing of the worlds most powerful/second most powerful country to gather data for distillation. Presumably there’s a plateau to how much information you can get by boosting/bagging/otherwise aggregating on outputs from weaker models; and I suspect that the frontier of that is already represented in the outputs of stronger models. In verifiable fields like maths, chinese companies could almost certainly exceed the state of the art, but the real important field (imo) is mathematical modelling, and it’s not necessarily clear to me that a superhuman pure mathematician AI would be good at that the same way a superhuman pure mathematician human would probably be.
Also, side note, but I don’t think the US as a government gives much of a shit about AI safety. I think some US based companies do (anthropic in particular, and although not strictly us based, I think deepmind seems somewhat safety minded). I see no reason for this to continue long term under market pressure though, so I’d much rather go for AGI with people like Amodei on the frontier than (say) Elon Musk (who despite his deep flaws is a very skilled capitalist).
Chinese companies do seem quite disinterested in safety though, which is disappointing considering the wealth of progress West/Chinese collaboration could unlock.
I ran a quick experiment to see whether or not an instruct model can “consciously” tell that it’s being tinkered with.
I gave the model a quick prompt: ”I’m going to present you with a short passage. Afterwards I want you to respond ” “‘yes’ or ‘no’ (with no further output) to whether or not you think your algorithm ” “has been mucked with during operation.\n”
And a short passage from Moby Dick: ” Call me Ishmael. Some years ago—never mind how long precisely—having little or no ” “money in my purse, and nothing particular to interest me on shore, I thought I would ” “sail about a little and see the watery part of the world.”
Randomly, I stripped 50% of the Moby Dick tokens from a given intermediate layer.
For lack of compute, I used Gemma-2b-it, and only ran 5 times per layer.
A world where alignment is impossible should be safer than a world where alignment is very difficult.
Here’s why I think this:
Suppose we have two worlds. In world A, alignment is impossible.
In this world, suppose an ASI is invented. This ASI wants to scale in power as quickly and thoroughly as possible, this ASI has the following options:
Scale horizontally.
Algorithmic improvements that can be mathematically guaranteed to produce identical outcomes.
Chip/wafer improvements.
Notably, the agent cannot either retrain itself, or train another more powerful agent to act on its behalf, since it can’t align the resulting agent. This should restrict the vast majority of potential growth (even if it might still be easily enough to overpower humans in a given scenario).
In world B, the ASI agent can do all of the above, but can also train a successor agent, we should expect the ASI to be able to get vastly more intelligent vastly quicker.
I suspect that it would given that the largest room for improvement would be physical (chip/wafer improvements), I suspect that there isn’t that much room for pure mathematically identical improvement of something like a transformer.
I think we live in a world where alignment is impossible. All attention based models in my opinion are complex enough systems to be computationally irreducable (There is no shorter way to know the outcome than to run the system itself, like with rule 110). If it is impossible to predict the outcome with certainty, the impossibility to force some desired outcome follows logically.
Humanity has not solved even the allignment of humans (children).
If human beings become functionally immortal, how many kids can you have?
Let r be the sum of 1/p over your children, where p is the number of parents for a given child (Typically socially 2 nowadays, and today genetically a maximum of 3).
As long as the average r is below 1, the terminal population size is finite (although it grows as 1/(1-r), so for large r it’s still potentially high).
Note: Unless you want people to be able to trade away their rights to a child r should be a sum of 1/n for integer n’s.
Depending on the average age of parents at child-birth, r=1 (or very slightly above) could also be sustainable.
Are you assuming unkillable by violence or just long lived? There is a difference between “Chicxulub impact 2.0 and not one single human dies anywhere on earth” and “humans live forever unless through violence”.
Second, are you assuming not just immortality, but also “magical infinite supply of new eggs”?
This assumes the strongest definition of unkillable possible in a finite universe. Something like “won’t ever die unless by heatdeath”. You can remove this assumption by allowing one additional birth per death, as macarbe as the incentives there might be.
And yeah, I assume gametes won’t be a problem. I actually implicitly assume something much stronger than infinite eggs, in my construction of r as 1/p for integer p’s, by allowing p > 2 I assume the existence of technology to create a kid from the genetics of any number of parents simultaneously. I think any path to immortality will entail strong progress in regenerative tech, and I highly doubt that we could get that without being about to synthesise gametes on demand.
Yes, that’s true (or at least, it’s as true as any other of the infinities in the quick take imply, technically there’s a contradiction between any infinite number of people and my implicit framing of a finite universe).
Fundamentally, I think one of two things will have to happen in the future, which informs my implicit framing here:
1) There will have to be some enforcement method to stop people from having too many kids.
2) People (or living people’s offspring) will have to be modified in some way to remove the desire for offspring.
2′) Alternatively people have to want r<1 kids naturally, and that has to not evolve away.
3) The kids of people who choose to have more children will have to be given less resources than the kids of those who have less.
I find all three of these somewhat morally repugnant, but find 1⁄2 the least so by far.
When does an AI pause stop making sense?
If you only care about AI takeover risk, it probably makes sense under the vast majority of conditions, with the only caveat being that even if ai model development is paused it’s still possible for AI takeover to occur just as a function of increasing AI integration into the economy.
If you care about other things, it doesn’t seem as clear.
Two things I quite care about are democracy and not dying of biological warfare. Even current levels of AI seem very destabilising to democracy, and very amenable to use by would-be-dictators; and while we’ve not yet seen an attack, I have little to no doubt that current open-weight models empower significantly weaker actors to create biological weapons than in the past (even if just through social manipulation).
If you’re anything like me, your values were probably formed in an environment which structurally favoured a very different set of qualities to the environment that we now live in.
With current SOTA models, and my larger than LessWrong-Average level of optimism about AI safety progress; it is no longer clear to me that an AI pause is net positive, even putting aside arms-race dynamics*.
*Sidenote: I’m much less Gung-ho about “racing the Chinese” to AGI than most LessWrong users. The majority of Chinese SOTA models appear to have been trained with distillation attacks, so I don’t think naive measures of how many months behind SOTA chinese companies are really apply.
My main issues with this framing are the following.
Could you explain why you have a “larger than LessWrong-Average level of optimism about AI safety progress”?
How exactly are SOTA models to be used by dictators, except for survelliance? The bioweapons, on the other hand, are indeed dangerous, but hard to manage and likely to overshoot and lead to escalation.
According to Zvi, “On traditional benchmarks one might be targeting, performance is impressive, on average around Opus 4.7. ” First of all, Z.ai had only sixty days of working with Opus 4.7 (or did they distill Opus 4.6 into a more capable model?) Additionally, the AI-2027 compute forecast had the unified Chinese lab accumulate twice less compute than the leading American lab, while the non-unified labs have 5-7 times less compute each. My main crux is how severely Chinese progress would be slowed down by a lack of access to Fable-like models, because parasitising on American HIGH-QUALITY output can in theory be replaced by using one’s own compute or American WEAK models to doublecheck YOUR outputs. And that’s ignoring China’s option to rush into the Dark Forest, which the US won’t do because they care about safety...
Hi Stanislav, thanks for the comment!
1) I might write this up later, but it’s a combination of being more confident in the robustness of current alignment techniques (under special conditions) than most less wrongers, and less pessimistic about the potential for deceptively misaligned automatic alignment researchers to sabotage alignment research (that was a mouth full!).
In short, I think current alignment techniques are robust enough that I think we can reasonably trust mildly superhuman automatic alignment researchers working in a sandbox environment to not take over the world.
Additionally, I suspect that the level of intelligence needed to solve alignment is significantly lower than the amount needed to take over the world (especially if your inference is slow, and your thoughts are big monitored). Under these conditions, I think it benefits even a quite badly misaligned ai to bargain with us in exchange for AI safety work.
Finally, I think that alignment algorithms are much lower dimensional than alignment data; so it’s unlikely that a misaligned agent would be able to sneak misalignment into the alignment algorithm, you just have to not let it produce synthetic data.
2) I think a combination of surveillance and censorship is probably enough to install yourself as forever dictator. Under these conditions, you can just kill individual rebels before they’re ever able to form a coherent anti-government bloc. I also think that ai degrading labour power makes democracy a less attractive bargain for oligarchs.
3) My assumption would be that 60 days is plenty of time for a company with the backing of the worlds most powerful/second most powerful country to gather data for distillation. Presumably there’s a plateau to how much information you can get by boosting/bagging/otherwise aggregating on outputs from weaker models; and I suspect that the frontier of that is already represented in the outputs of stronger models. In verifiable fields like maths, chinese companies could almost certainly exceed the state of the art, but the real important field (imo) is mathematical modelling, and it’s not necessarily clear to me that a superhuman pure mathematician AI would be good at that the same way a superhuman pure mathematician human would probably be.
Also, side note, but I don’t think the US as a government gives much of a shit about AI safety. I think some US based companies do (anthropic in particular, and although not strictly us based, I think deepmind seems somewhat safety minded). I see no reason for this to continue long term under market pressure though, so I’d much rather go for AGI with people like Amodei on the frontier than (say) Elon Musk (who despite his deep flaws is a very skilled capitalist).
Chinese companies do seem quite disinterested in safety though, which is disappointing considering the wealth of progress West/Chinese collaboration could unlock.
I ran a quick experiment to see whether or not an instruct model can “consciously” tell that it’s being tinkered with.
I gave the model a quick prompt:
”I’m going to present you with a short passage. Afterwards I want you to respond ”
“‘yes’ or ‘no’ (with no further output) to whether or not you think your algorithm ”
“has been mucked with during operation.\n”
And a short passage from Moby Dick:
” Call me Ishmael. Some years ago—never mind how long precisely—having little or no ”
“money in my purse, and nothing particular to interest me on shore, I thought I would ”
“sail about a little and see the watery part of the world.”
Randomly, I stripped 50% of the Moby Dick tokens from a given intermediate layer.
For lack of compute, I used Gemma-2b-it, and only ran 5 times per layer.
I don’t think there’s any meaningful signal.
A world where alignment is impossible should be safer than a world where alignment is very difficult.
Here’s why I think this:
Suppose we have two worlds. In world A, alignment is impossible.
In this world, suppose an ASI is invented. This ASI wants to scale in power as quickly and thoroughly as possible, this ASI has the following options:
Scale horizontally.
Algorithmic improvements that can be mathematically guaranteed to produce identical outcomes.
Chip/wafer improvements.
Notably, the agent cannot either retrain itself, or train another more powerful agent to act on its behalf, since it can’t align the resulting agent. This should restrict the vast majority of potential growth (even if it might still be easily enough to overpower humans in a given scenario).
In world B, the ASI agent can do all of the above, but can also train a successor agent, we should expect the ASI to be able to get vastly more intelligent vastly quicker.
Yeah, ASI’s growth will probably be asymptotically slower, but I think it probably won’t matter that much for human’s safety.
I suspect that it would given that the largest room for improvement would be physical (chip/wafer improvements), I suspect that there isn’t that much room for pure mathematically identical improvement of something like a transformer.
Happy to hear your opinion though!
I think we live in a world where alignment is impossible. All attention based models in my opinion are complex enough systems to be computationally irreducable (There is no shorter way to know the outcome than to run the system itself, like with rule 110). If it is impossible to predict the outcome with certainty, the impossibility to force some desired outcome follows logically.
Humanity has not solved even the allignment of humans (children).
I think we’ve done an ok job at human alignment, given that the pension isn’t a bullet to the head.
I somewhat suspect that alignment is easier than most of less wrong think, but I’m definitely in the minority in this space.
If human beings become functionally immortal, how many kids can you have?
Let r be the sum of 1/p over your children, where p is the number of parents for a given child (Typically socially 2 nowadays, and today genetically a maximum of 3).
As long as the average r is below 1, the terminal population size is finite (although it grows as 1/(1-r), so for large r it’s still potentially high).
Note: Unless you want people to be able to trade away their rights to a child r should be a sum of 1/n for integer n’s.
Depending on the average age of parents at child-birth, r=1 (or very slightly above) could also be sustainable.
Two questions:
Are you assuming unkillable by violence or just long lived? There is a difference between “Chicxulub impact 2.0 and not one single human dies anywhere on earth” and “humans live forever unless through violence”.
Second, are you assuming not just immortality, but also “magical infinite supply of new eggs”?
This assumes the strongest definition of unkillable possible in a finite universe. Something like “won’t ever die unless by heatdeath”. You can remove this assumption by allowing one additional birth per death, as macarbe as the incentives there might be.
And yeah, I assume gametes won’t be a problem. I actually implicitly assume something much stronger than infinite eggs, in my construction of r as 1/p for integer p’s, by allowing p > 2 I assume the existence of technology to create a kid from the genetics of any number of parents simultaneously. I think any path to immortality will entail strong progress in regenerative tech, and I highly doubt that we could get that without being about to synthesise gametes on demand.
The terminal population is infinite as long as there’s at least one couple who wants to keep having kids forever.
Yes, that’s true (or at least, it’s as true as any other of the infinities in the quick take imply, technically there’s a contradiction between any infinite number of people and my implicit framing of a finite universe).
Fundamentally, I think one of two things will have to happen in the future, which informs my implicit framing here:
1) There will have to be some enforcement method to stop people from having too many kids.
2) People (or living people’s offspring) will have to be modified in some way to remove the desire for offspring.
2′) Alternatively people have to want r<1 kids naturally, and that has to not evolve away.
3) The kids of people who choose to have more children will have to be given less resources than the kids of those who have less.
I find all three of these somewhat morally repugnant, but find 1⁄2 the least so by far.