my view of how elites will act when informed about AI x-risk is based on actual examples that happened
If you are referring to the founding OpenAI and Anthropic, you should look at how they happened. Sam Altman is a career entrepreneur, whose job is to create businesses. Dario Amodei was an AI researcher, whose job was to research AI. Inertia, my default predictor for individuals, is a very good predictor here, good enough I don’t feel the need to perform any update on other heuristics.
In democratic institutions, things tend to happen quite differently. From the post:
Normally, parts of a country’s executive branch are responsible for international negotiations around urgent issues concerning national and global security. … Branches of the government are generally not in the business of independently taking bold positions and then pursuing those positions to their logical ends. Instead, their stances and actions are mostly shaped by prevailing social currents.
Basically decisions are much more diffused, and this should favor caution about x-risks if people are properly informed about x-risks.
Correct! Elon Musk is an entrepreneur, Sam Altman is an entrepreneur, Dario Amodei is a researcher. Informing them led to accelerating the race. Now you want to make a bunch of politicians more informed about AI. The job of politicians is to get power. How do you think it will go? A race to develop AI for military and social control, that’s how. It’s worth being as cynical as possible about this, because reality will look more cynical still.
I agree with your focus on democratic institutions. But less on “institutions” and more on “democratic”. The people at the top will always face competitive incentives to use AI to get more power, even if they know about the risk. That’s why I keep saying that an anti-AI movement needs to be able to overrule most people at the top, and its power base needs to be at the bottom.
To me it seems well worth a shot. Politics has worked reasonably well for limiting atomic weapons and curtailing ozone hole damage. AI has some important disanalogies to those things, but still. I have talked to a couple of state-level politicians about AI. One of them was very easy to talk to and seemed quite sane. The other one (who I talked to much less, so, didn’t get too much of a chance to reason with) was worried about the USA winning the AI race. On the whole, I felt there was more sanity than I expected from politicians. EY expressed a similar sentiment recently in Only Law Can Prevent Extinction:
(I am still feeling amazed, awed, and a little humbled, about the part where my words plausibly had any effect whatsoever. Politicians are a lot more sensible, in some real-life cases, than angry libertarian literature had led me to believe a few decades earlier.)
Politics has worked reasonably well for limiting atomic weapons
Politics also worked very well for creating atomic weapons.
“Worth a shot” is the type of conclusion that is best applied to things that have positive-skewed outcomes, but seems to be missing a mood when applied to things that could cause big positive or negative effects.
On the whole, I felt there was more sanity than I expected from politicians.
Conditional on observing that the system as a whole operates at a given level of insanity, if there’s more sanity than you expected in conversations with individual politicians then there’s likely less sanity than you expected in the process by which conversations with individual politicians end up as policy outcomes.
For example, politicians might be better than you expect at saying reassuring things while totally compartmentalizing those statements from their actions (or, indeed, then saying just-as-reassuring things to other people whose beliefs are the polar opposite of yours).
“Worth a shot” is the type of conclusion that is best applied to things that have positive-skewed outcomes, but seems to be missing a mood when applied to things that could cause big positive or negative effects.
In my opinion, it is extremely overdetermined that we all die without a global ASI ban. I don’t have time to flesh out my full view rn, but to summarize, there are at least these 3 layers to it:
We have no idea how to build safe ASI; TAIS projects are not nearly on track to enable us to build safe ASI by the time we build ASI.
Even if you are more optimistic wrt to the feasibility of TAIS, there will be extreme pressure to cut corners on safety in a world without a global ASI ban (+ what Connor Leahy says in his post You can only build safe ASI if ASI is globally banned)
Powerful actors in the world are not going to wait around as you build your safe ASI and perform a pivotal act; they will take control of your ASI project, or declare war on you preemptively before you can get a DSA, or something else in that flavor.
While you could try to forward chain from the present and predict whether spreading awareness has specific negative or positive consequences, I think that backtracking from the desired future state (a global ASI ban) makes the decision much more obvious.
My reasoning goes: I want a robust global ASI ban. The only plan that i’ve ever seen about how to achieve a robust ASI ban involves mass awareness of x-risk from ASI.
The decision to pursue mass awareness is not about calculating the probs of various specific events and seeing if it results in a net positive expectation, it’s a matter of fulfilling an absolute requirement to achieving a global ASI ban, which is the only state the world can be where I’m sure ASI won’t kill everyone.
I think you have a point, but I still think the Control AI strategy is probably now the best investment out there in terms of reducing p(doom). The backfire effects are limited (given many powerful people are already aware of AI), and the upside is very high in expectation.
its power base needs to be at the bottom.
This might be true, but the fact remains that we need a well enforced international treaty. And only the people at the top can make that happen. The masses have a big role too though: enacting a global taboo on AGI/ASI research, similar to the ones that already exists for other things like human cloning, which will make enforcing the treaty so much easier.
If you are referring to the founding OpenAI and Anthropic, you should look at how they happened. Sam Altman is a career entrepreneur, whose job is to create businesses. Dario Amodei was an AI researcher, whose job was to research AI. Inertia, my default predictor for individuals, is a very good predictor here, good enough I don’t feel the need to perform any update on other heuristics.
In democratic institutions, things tend to happen quite differently. From the post:
Basically decisions are much more diffused, and this should favor caution about x-risks if people are properly informed about x-risks.
Correct! Elon Musk is an entrepreneur, Sam Altman is an entrepreneur, Dario Amodei is a researcher. Informing them led to accelerating the race. Now you want to make a bunch of politicians more informed about AI. The job of politicians is to get power. How do you think it will go? A race to develop AI for military and social control, that’s how. It’s worth being as cynical as possible about this, because reality will look more cynical still.
I agree with your focus on democratic institutions. But less on “institutions” and more on “democratic”. The people at the top will always face competitive incentives to use AI to get more power, even if they know about the risk. That’s why I keep saying that an anti-AI movement needs to be able to overrule most people at the top, and its power base needs to be at the bottom.
To me it seems well worth a shot. Politics has worked reasonably well for limiting atomic weapons and curtailing ozone hole damage. AI has some important disanalogies to those things, but still. I have talked to a couple of state-level politicians about AI. One of them was very easy to talk to and seemed quite sane. The other one (who I talked to much less, so, didn’t get too much of a chance to reason with) was worried about the USA winning the AI race. On the whole, I felt there was more sanity than I expected from politicians. EY expressed a similar sentiment recently in Only Law Can Prevent Extinction:
Politics also worked very well for creating atomic weapons.
“Worth a shot” is the type of conclusion that is best applied to things that have positive-skewed outcomes, but seems to be missing a mood when applied to things that could cause big positive or negative effects.
Conditional on observing that the system as a whole operates at a given level of insanity, if there’s more sanity than you expected in conversations with individual politicians then there’s likely less sanity than you expected in the process by which conversations with individual politicians end up as policy outcomes.
For example, politicians might be better than you expect at saying reassuring things while totally compartmentalizing those statements from their actions (or, indeed, then saying just-as-reassuring things to other people whose beliefs are the polar opposite of yours).
In my opinion, it is extremely overdetermined that we all die without a global ASI ban. I don’t have time to flesh out my full view rn, but to summarize, there are at least these 3 layers to it:
We have no idea how to build safe ASI; TAIS projects are not nearly on track to enable us to build safe ASI by the time we build ASI.
Even if you are more optimistic wrt to the feasibility of TAIS, there will be extreme pressure to cut corners on safety in a world without a global ASI ban (+ what Connor Leahy says in his post You can only build safe ASI if ASI is globally banned)
Powerful actors in the world are not going to wait around as you build your safe ASI and perform a pivotal act; they will take control of your ASI project, or declare war on you preemptively before you can get a DSA, or something else in that flavor.
While you could try to forward chain from the present and predict whether spreading awareness has specific negative or positive consequences, I think that backtracking from the desired future state (a global ASI ban) makes the decision much more obvious.
My reasoning goes: I want a robust global ASI ban. The only plan that i’ve ever seen about how to achieve a robust ASI ban involves mass awareness of x-risk from ASI.
The decision to pursue mass awareness is not about calculating the probs of various specific events and seeing if it results in a net positive expectation, it’s a matter of fulfilling an absolute requirement to achieving a global ASI ban, which is the only state the world can be where I’m sure ASI won’t kill everyone.
I think you have a point, but I still think the Control AI strategy is probably now the best investment out there in terms of reducing p(doom). The backfire effects are limited (given many powerful people are already aware of AI), and the upside is very high in expectation.
This might be true, but the fact remains that we need a well enforced international treaty. And only the people at the top can make that happen. The masses have a big role too though: enacting a global taboo on AGI/ASI research, similar to the ones that already exists for other things like human cloning, which will make enforcing the treaty so much easier.