I think “What’s your P(AI takeover)” is a totally reasonable question and don’t understand Eliezer’s problem with it (or his problem with asking about timelines). Especially given that he is actually very (and IMO overconfidently) opinionated on this topic! I think it would be crazy to never give bottom-line estimates for P(AI takeover) when talking about AI takeover risk.
(I think talking about timelines is more important because it’s more directly decision relevant (whereas P(AI takeover) is downstream of the decision-relevant variables).)
In practice the numbers I think are valuable to talk about are mostly P(some bad outcome|some setting of the latent facts about misalignment, and some setting of what risk mitigation strategy is employed). Like, I’m constantly thinking about whether to try to intervene on worlds that are more like Plan A worlds or more like Plan D worlds, and to do this you obviously need to think about the effect on P(AI takeover) of your actions in those worlds.
I think “What’s your P(AI takeover)” is a totally reasonable question and don’t understand Eliezer’s problem with it (or his problem with asking about timelines). Especially given that he is actually very (and IMO overconfidently) opinionated on this topic! I think it would be crazy to never give bottom-line estimates for P(AI takeover) when talking about AI takeover risk.
(I think talking about timelines is more important because it’s more directly decision relevant (whereas P(AI takeover) is downstream of the decision-relevant variables).)
In practice the numbers I think are valuable to talk about are mostly P(some bad outcome|some setting of the latent facts about misalignment, and some setting of what risk mitigation strategy is employed). Like, I’m constantly thinking about whether to try to intervene on worlds that are more like Plan A worlds or more like Plan D worlds, and to do this you obviously need to think about the effect on P(AI takeover) of your actions in those worlds.