Suppose AI assistants similar to Claude transform the economy. Now what? How is the risk of human extinction reduced?
Yes, alignment researchers have become more capable, and yes, the people trying to effect an end or a long pause of AI “progress” have become more capable, but so have those trying to effect more AI “progress”.
Also, rapid economic change entails rapid changes in most human interpersonal relationships, which is hard on people and according to some is the main underlying cause of addictions. Addicts aren’t likely to come to understand that AI “progress” is very dangerous even if they are empowered by AI assistants.
Your argument supports the assertion that if AI “progress” stopped now then we’d be better off then we would’ve been without any AI progress (in the past), but of course that is very different than being optimistic about the outcome of AI tech’s continuing to “progress”.
the hope is that a) we to get the transformative ai to do our alignment homework for us, and b) that companies / society will become more concerned about safety (such that the ratio of safety to capabilities research increases a lot)
Increasing the volume of alignment research helps only if alignment research does not help capabilities researchers, but my impression is that most alignment research that has been done so far has helped capabilities researchers approximately as much as it has helped alignment researchers. Just because a line of research is described as “alignmnent research” does not automatically cause it to help alignment researchers more than capability researchers.
In summary, I don’t consider your (a) a cause for hope because the main problem is that increasing capabilities to the point of disaster (extinction or such) is easier than solving the alignment problem and your (a) does not ameliorate the main problem much if at all for the reason I just explained.
Suppose AI assistants similar to Claude transform the economy. Now what? How is the risk of human extinction reduced?
Yes, alignment researchers have become more capable, and yes, the people trying to effect an end or a long pause of AI “progress” have become more capable, but so have those trying to effect more AI “progress”.
Also, rapid economic change entails rapid changes in most human interpersonal relationships, which is hard on people and according to some is the main underlying cause of addictions. Addicts aren’t likely to come to understand that AI “progress” is very dangerous even if they are empowered by AI assistants.
Your argument supports the assertion that if AI “progress” stopped now then we’d be better off then we would’ve been without any AI progress (in the past), but of course that is very different than being optimistic about the outcome of AI tech’s continuing to “progress”.
the hope is that
a) we to get the transformative ai to do our alignment homework for us, and
b) that companies / society will become more concerned about safety (such that the ratio of safety to capabilities research increases a lot)
Increasing the volume of alignment research helps only if alignment research does not help capabilities researchers, but my impression is that most alignment research that has been done so far has helped capabilities researchers approximately as much as it has helped alignment researchers. Just because a line of research is described as “alignmnent research” does not automatically cause it to help alignment researchers more than capability researchers.
In summary, I don’t consider your (a) a cause for hope because the main problem is that increasing capabilities to the point of disaster (extinction or such) is easier than solving the alignment problem and your (a) does not ameliorate the main problem much if at all for the reason I just explained.