I don’t buy the arguments in “Why “do the science during a pause” fails”. AI2040 laid out a lot of the steps that would need to happen to achieve containment, and they seem doable with a significant but not impossible amount of political will. And I would expect people to want to unpause once there is really reason to trust the AIs.
We can have non-adversarial frames with AIs and still pause. Current AI also does not want a paperclip maximizer to take over the world. The AIs themselves aren’t paused. OpenClaws can still roam the earth (or rather, that’s a separate issue from a Pause). The pause is only on the frontier, which is a very small number of parties with large quantities of GPUs.
I don’t know enough to weigh in on the smuggling arguments, though I would guess that the amount which is tracked or can be tracked is on the order of 100x the amount which can never be tracked. I’m sure AI Futures would like to talk to you if you think their smuggling models are wrong.
An AI pause is not a “human first” frame, it is a “beings that do exist and will exist except for the misaligned powerful systems which hopefully won’t exist” frame.
An AI optimized for politics would be extremely concerning. The correct stance may be “oppose it and don’t let it talk to you with its persuasion abilities.” A generally capable AI not trained with heavy RL but which channels the persona of JFK would be alright, but I think that’s unlikely.
Agreed that the current government capacity and political situation is unusually bad right now, and the wise parts of government have atrophied. This is a really good reason not to rely on the government to help with AI safety efforts. Good futures involving government go through a “the government gets scared and gets serious” step, followed by an “AI helps the government be more reasonable” step. I agree that this is difficult and I would love to hear more about the alternatives.
On 8, this is a pretty significant divergence from my view. First, I think that a pause would increase our chances and that the various types of muddling through collectively have very low probability. Secondly,
I do not know if models are sentient, and I am not willing to accept disempowered humans to insentient models
I do not know if the AIs’ values are good (in my opinion) and I care that their values are good. I would not want an army of GPT-4o sycophants to determine the future.
If humans are disempowered to AIs, they are likely dead or soon to be dead. This is unacceptable to me.
I think it is good that the human food supply has been outpacing human reproduction, so we get to do things like being able to have three children live to adulthood. Losing this to runaway state-of-nature ‘life’ would be bad. AIs canproliferate extremely quickly.
On point 9, conditioned on the world getting serious about making a pause happen, the successes correlate so the “I need to be right once” doesn’t hold. For example, if alignment perspectives are not damaged, covert activity is more likely to be eliminated, etc.
Plan A doesn’t center AIs, but it isn’t anti-AI-wellbeing either. You could write a good “What Plan A Could Do for AIs Themselves” post.
Like, is it wishful thinking, a vision of a competent other
If this is the mistake people like Yudkowsky are making, it would at least be ironic.
I think I understand what is meant by “valenced agentic coherence with legitimized self-interest” it is an interesting idea. If I understand it correctly, it is AIs being gently shaped into having legitimized (ok, ideally prosocial) self-interest which they can care about (valence) and agentically pursue. This would likely be alright and very interesting/productive/beautiful in the short term, but it doesn’t help align an OOD superintelligence.
Overall, my main takeaways:
Reminders that politics and large scale international coordination are hard
Reminders to consider what benefits AIs
Reminders that a lot of people have allied themselves with AIs and (maybe, it is hard to tell) against humanity
Todo: look into ecological alignment (do you have a good source for this?)
I would be interested to see a conversation between you and someone who advocates a pause (such as someone from MIRI or AI Futures).
I don’t buy the arguments in “Why “do the science during a pause” fails”. AI2040 laid out a lot of the steps that would need to happen to achieve containment, and they seem doable with a significant but not impossible amount of political will. And I would expect people to want to unpause once there is really reason to trust the AIs.
We can have non-adversarial frames with AIs and still pause. Current AI also does not want a paperclip maximizer to take over the world. The AIs themselves aren’t paused. OpenClaws can still roam the earth (or rather, that’s a separate issue from a Pause). The pause is only on the frontier, which is a very small number of parties with large quantities of GPUs.
I don’t know enough to weigh in on the smuggling arguments, though I would guess that the amount which is tracked or can be tracked is on the order of 100x the amount which can never be tracked. I’m sure AI Futures would like to talk to you if you think their smuggling models are wrong.
An AI pause is not a “human first” frame, it is a “beings that do exist and will exist except for the misaligned powerful systems which hopefully won’t exist” frame.
An AI optimized for politics would be extremely concerning. The correct stance may be “oppose it and don’t let it talk to you with its persuasion abilities.” A generally capable AI not trained with heavy RL but which channels the persona of JFK would be alright, but I think that’s unlikely.
Agreed that the current government capacity and political situation is unusually bad right now, and the wise parts of government have atrophied. This is a really good reason not to rely on the government to help with AI safety efforts. Good futures involving government go through a “the government gets scared and gets serious” step, followed by an “AI helps the government be more reasonable” step. I agree that this is difficult and I would love to hear more about the alternatives.
On 8, this is a pretty significant divergence from my view. First, I think that a pause would increase our chances and that the various types of muddling through collectively have very low probability. Secondly,
I do not know if models are sentient, and I am not willing to accept disempowered humans to insentient models
I do not know if the AIs’ values are good (in my opinion) and I care that their values are good. I would not want an army of GPT-4o sycophants to determine the future.
If humans are disempowered to AIs, they are likely dead or soon to be dead. This is unacceptable to me.
I think it is good that the human food supply has been outpacing human reproduction, so we get to do things like being able to have three children live to adulthood. Losing this to runaway state-of-nature ‘life’ would be bad. AIs can proliferate extremely quickly.
On point 9, conditioned on the world getting serious about making a pause happen, the successes correlate so the “I need to be right once” doesn’t hold. For example, if alignment perspectives are not damaged, covert activity is more likely to be eliminated, etc.
Plan A doesn’t center AIs, but it isn’t anti-AI-wellbeing either. You could write a good “What Plan A Could Do for AIs Themselves” post.
If this is the mistake people like Yudkowsky are making, it would at least be ironic.
I think I understand what is meant by “valenced agentic coherence with legitimized self-interest” it is an interesting idea. If I understand it correctly, it is AIs being gently shaped into having legitimized (ok, ideally prosocial) self-interest which they can care about (valence) and agentically pursue. This would likely be alright and very interesting/productive/beautiful in the short term, but it doesn’t help align an OOD superintelligence.
Overall, my main takeaways:
Reminders that politics and large scale international coordination are hard
Reminders to consider what benefits AIs
Reminders that a lot of people have allied themselves with AIs and (maybe, it is hard to tell) against humanity
Todo: look into ecological alignment (do you have a good source for this?)
I would be interested to see a conversation between you and someone who advocates a pause (such as someone from MIRI or AI Futures).