If you believe that spending more on safety leads to acceleration instead, you should try to refute my argument for why it is a net positive.
I’m honestly very curious how my opponents will reply to my “net positive” arguments, so I promise I’ll appreciate a reply and upvote you.
I pasted it in this comment so you don’t have to look for it:
Why I feel almost certain this open letter is a net positive
Delaying AI capabilities alone isn’t enough. If you wished for AI capabilities to be delayed by 1000 years, then one way to fulfill your wish is if the Earth had formed 1000 years later, which delays all of history by the same 1000 years.
Clearly, that’s not very useful. AI capabilities have to be delayed relative to something else.
That something else is either:
Progress in alignment (according to optimists like me)
or
Progress towards governments freaking out about AGI and going nuclear to stop it (according to LessWrong’s pessimist community)
Either way, the AI Belief-Consistency Letter speeds up that progress by many times more than it speeds up capabilities. Let me explain.
Case 1:
Case 1 assumes we have a race between alignment and capabilities. From first principles, the relative funding of alignment and capabilities matters in this case.
Increasing alignment funding by 2x ought to have a similar effect to decreasing capability funding by 2x.
Various factors may make the relationship inexact, e.g. one might argue that increasing alignment by 4x might be equivalent to decreasing capabilities by 2x, if one believes that capabilities is more dependent on funding.
But so long as one doesn’t assume insane differences, the AI Belief-Consistency Letter is a net positive in Case 1.
If the AI Belief-Consistency Letter increases both by $1 billion, that’s a 5x to 10x alignment increase and only a 1.002x to 1.005x capabilities increase. That would clearly be a net positive.
Case 2:
Even if the wildest dreams of the AI pause movement succeed, and the US, China, and EU all agree to halt all capabilities above a certain threshold, the rest of the world still exists, so it only reduces capabilities funding by 10x effectively.
That would be very good, but we’ll still have a race between capabilities and alignment, and Case 1 still applies. The AI Belief-Consistency Letter still increases alignment funding by far more than capabilities funding.
The only case where we should not worry about increasing alignment funding, is if capabilities funding is reduced to zero, and there’s no longer a race between capabilities and alignment.
The only way to achieve that worldwide, is to “solve diplomacy,” which is not going to happen, or to “go nuclear,” like Eliezer Yudkowsky suggests.
If your endgame is to “go nuclear” and make severe threats to other countries despite the risk, you surely can’t oppose the AI Belief-Consistency Letter on the grounds that “it speeds up capabilities because it makes governments freak out about AGI,” since you actually need governments to freak out about AGI.
Conclusion
Make sure you don’t oppose this idea based on short term heuristics like “the slower capabilities grow, the better,” without reflecting on why you believe so. Think about what your endgame is. Is it slowing down capabilities to make time for alignment? Or is it slowing down capabilities to make time for governments to freak out and halt AI worldwide?
You make a very good point about political goals, and I have to agree that this letter probably won’t convince politicians whose political motivations prevent them from supporting AI alignment spending.
Yes, military spending indeed rewards constituents, and some companies go out of their way to hire people in multiple states etc.
PS: I actually mentioned the marginal change in a footnote, but I disabled the sidebar so maybe you missed it. I’ll add the sidebar footnotes back.
If you believe that spending more on safety leads to acceleration instead, you should try to refute my argument for why it is a net positive.
I’m honestly very curious how my opponents will reply to my “net positive” arguments, so I promise I’ll appreciate a reply and upvote you.
I pasted it in this comment so you don’t have to look for it:
Why I feel almost certain this open letter is a net positive
Delaying AI capabilities alone isn’t enough. If you wished for AI capabilities to be delayed by 1000 years, then one way to fulfill your wish is if the Earth had formed 1000 years later, which delays all of history by the same 1000 years.
Clearly, that’s not very useful. AI capabilities have to be delayed relative to something else.
That something else is either:
Progress in alignment (according to optimists like me)
or
Progress towards governments freaking out about AGI and going nuclear to stop it (according to LessWrong’s pessimist community)
Either way, the AI Belief-Consistency Letter speeds up that progress by many times more than it speeds up capabilities. Let me explain.
Case 1:
Case 1 assumes we have a race between alignment and capabilities. From first principles, the relative funding of alignment and capabilities matters in this case.
Increasing alignment funding by 2x ought to have a similar effect to decreasing capability funding by 2x.
Various factors may make the relationship inexact, e.g. one might argue that increasing alignment by 4x might be equivalent to decreasing capabilities by 2x, if one believes that capabilities is more dependent on funding.
But so long as one doesn’t assume insane differences, the AI Belief-Consistency Letter is a net positive in Case 1.
This is because alignment funding is only at $0.1 to $0.2 billion, while capabilities funding is at $200+ billion to $600+ billion.
If the AI Belief-Consistency Letter increases both by $1 billion, that’s a 5x to 10x alignment increase and only a 1.002x to 1.005x capabilities increase. That would clearly be a net positive.
Case 2:
Even if the wildest dreams of the AI pause movement succeed, and the US, China, and EU all agree to halt all capabilities above a certain threshold, the rest of the world still exists, so it only reduces capabilities funding by 10x effectively.
That would be very good, but we’ll still have a race between capabilities and alignment, and Case 1 still applies. The AI Belief-Consistency Letter still increases alignment funding by far more than capabilities funding.
The only case where we should not worry about increasing alignment funding, is if capabilities funding is reduced to zero, and there’s no longer a race between capabilities and alignment.
The only way to achieve that worldwide, is to “solve diplomacy,” which is not going to happen, or to “go nuclear,” like Eliezer Yudkowsky suggests.
If your endgame is to “go nuclear” and make severe threats to other countries despite the risk, you surely can’t oppose the AI Belief-Consistency Letter on the grounds that “it speeds up capabilities because it makes governments freak out about AGI,” since you actually need governments to freak out about AGI.
Conclusion
Make sure you don’t oppose this idea based on short term heuristics like “the slower capabilities grow, the better,” without reflecting on why you believe so. Think about what your endgame is. Is it slowing down capabilities to make time for alignment? Or is it slowing down capabilities to make time for governments to freak out and halt AI worldwide?
You make a very good point about political goals, and I have to agree that this letter probably won’t convince politicians whose political motivations prevent them from supporting AI alignment spending.
Yes, military spending indeed rewards constituents, and some companies go out of their way to hire people in multiple states etc.
PS: I actually mentioned the marginal change in a footnote, but I disabled the sidebar so maybe you missed it. I’ll add the sidebar footnotes back.
Thanks!