I almost included a somewhat more general disclaimer of “also, this question is much more opinionated on framing”, and then didn’t end up including in the original post. But, I just edited that in.
Broadly, do you think there should be any question that’s filling the niche that p(Doom) is filling right now? (“no” seems super reasonable, but wondering if there is a question you actually think is useful for some kind of barometer-of-consensus that carves reality at the joints?)
I think “What’s your P(AI takeover)” is a totally reasonable question and don’t understand Eliezer’s problem with it (or his problem with asking about timelines). Especially given that he is actually very (and IMO overconfidently) opinionated on this topic! I think it would be crazy to never give bottom-line estimates for P(AI takeover) when talking about AI takeover risk.
(I think talking about timelines is more important because it’s more directly decision relevant (whereas P(AI takeover) is downstream of the decision-relevant variables).)
In practice the numbers I think are valuable to talk about are mostly P(some bad outcome|some setting of the latent facts about misalignment, and some setting of what risk mitigation strategy is employed). Like, I’m constantly thinking about whether to try to intervene on worlds that are more like Plan A worlds or more like Plan D worlds, and to do this you obviously need to think about the effect on P(AI takeover) of your actions in those worlds.
I almost included a somewhat more general disclaimer of “also, this question is much more opinionated on framing”, and then didn’t end up including in the original post. But, I just edited that in.
Broadly, do you think there should be any question that’s filling the niche that p(Doom) is filling right now? (“no” seems super reasonable, but wondering if there is a question you actually think is useful for some kind of barometer-of-consensus that carves reality at the joints?)
I think “What’s your P(AI takeover)” is a totally reasonable question and don’t understand Eliezer’s problem with it (or his problem with asking about timelines). Especially given that he is actually very (and IMO overconfidently) opinionated on this topic! I think it would be crazy to never give bottom-line estimates for P(AI takeover) when talking about AI takeover risk.
(I think talking about timelines is more important because it’s more directly decision relevant (whereas P(AI takeover) is downstream of the decision-relevant variables).)
In practice the numbers I think are valuable to talk about are mostly P(some bad outcome|some setting of the latent facts about misalignment, and some setting of what risk mitigation strategy is employed). Like, I’m constantly thinking about whether to try to intervene on worlds that are more like Plan A worlds or more like Plan D worlds, and to do this you obviously need to think about the effect on P(AI takeover) of your actions in those worlds.