Thanks for this response, I’m enjoying this debate.
You say “Despite this, he is more extreme in his confidence that things will be ok than the average expert”
From the perspective of an outsider like me, this statement doesn’t seem right. In the only big survey I could find with thousands of AI experts in 2024, the median p doom (which equates with the average expert) was 5% - pretty close to BB’s. In addition Expert forecasters (who are usually better than domain experts at predicting the future) put risk below 1 %. Sure many higher profile experts have more extreme positions , but these aren’t the average and there are some like Yann Lacunn, Hassibis and Andreeson who are below 2.6% . Even Ord is at 10% which isn’t that much higher than BB—who IMO to his credit tried to use statistics to get to his.
My second issue here (maybe just personal preference) is that I don’t love the way both you and @Bentham’s Bulldog talk about “confidence” Statistically when we talk about how confident we are in our predictions, this relates to how sure (confident) we are that our prediction is correct, not about whether our percentage (in this case pdoom) is high or low. I understand that both meanings can be correct, but for precision and to avoid confusion I prefer the statistical “confidence” definition. It might seem like a nitpick, but I even prefer “how sure are you” ASI will kill us all or even just “I think there’s a high probability that...”
By my definition of confidence then, Bentham’s Bulldog is far less confident than you in his prediction of 2.6%. He doesn’t quote his error bars but expresses that he is very uncertain, and wide error bars are implicit in his probability tree method as well. YS on the otherhand seem to have very narrow error bars around their claim “if anyone builds ASI with modern methods, everyone will die.”
Thanks for this comment! (I also saw you commented on the EA forum. I’m just going to respond here because I’m a LW guy and want to keep things simple.)
As you said, the median expert gives something like a 5% chance of doom. BB’s estimate is about a factor of two more confident than that that things will be okay. That factor of two difference is what I’m referencing. I am trying to say that I think it would be wiser for BB to be a factor of two less confident, like Ord. I’m not sure what doesn’t seem right about what I wrote.
I agree that superforecasters are even more confident than BB. I also agree that many domain experts are more confident.
I think that BB and I are using Bayesian language where “confidence” means “degree of certainty,” rather than degree of calibration or degree of meta-certainty or size of some hypothetical confidence interval or whatever. I agree that Y&S think the title thesis is “an easy call” and that they have not changed their mind on it even after talking to a lot of people. I buy that BB’s beliefs here are less stable/fixed.
I think there is a disconnect here related to different usages of the word “confidence”. You say in the OP:
he is more extreme in his confidence that things will be ok
Which I would interpret as being 1- P(not-okay), in other words 1 − 0.026 = 97.4% for BB, very confident.
On the other hand, I think many people probably believe that extinction from misaligned AI is very unlikely apriori, and so might use “confidence” in a sense that is relative to priors. To understand why people might do this, lets imagine that I said “there is a 55% chance that irrelevant AI blogger The Floating Droid will win the 2028 US presidential election”. Now imagine someone said “Wow! That’s insanely overconfident!”. I think people would be a bit suspicious if I responded that it was actually pretty unconfident because it quotes a probability near 50%.
I think this different usages of “confidence” is also relevant to the OP since it is reviewing BBs statements. For example, your statement:
Reads to me as a suggestion of hypocrisy or contadiction. BB accuse YS of being overconfident when really he is the one who is being extremely confident! But for this to be the case, we would need to evaluate the statement using the notion of “confidence” that was intended by BB. Its not clear to me that your post is actually using the same notion of “confidence”.
Glad the comment was helpful. I will register my prediction that BB most likely meant the “relative to priors” meaning rather than the one that you use in the OP. I also think among people who aren’t steeped in the background of AI risk, this would be the significantly more common interpretation upon reading what BB wrote.
Thanks for this response, I’m enjoying this debate.
You say “Despite this, he is more extreme in his confidence that things will be ok than the average expert”
From the perspective of an outsider like me, this statement doesn’t seem right. In the only big survey I could find with thousands of AI experts in 2024, the median p doom (which equates with the average expert) was 5% - pretty close to BB’s. In addition Expert forecasters (who are usually better than domain experts at predicting the future) put risk below 1 %. Sure many higher profile experts have more extreme positions , but these aren’t the average and there are some like Yann Lacunn, Hassibis and Andreeson who are below 2.6% . Even Ord is at 10% which isn’t that much higher than BB—who IMO to his credit tried to use statistics to get to his.
My second issue here (maybe just personal preference) is that I don’t love the way both you and @Bentham’s Bulldog talk about “confidence” Statistically when we talk about how confident we are in our predictions, this relates to how sure (confident) we are that our prediction is correct, not about whether our percentage (in this case pdoom) is high or low. I understand that both meanings can be correct, but for precision and to avoid confusion I prefer the statistical “confidence” definition. It might seem like a nitpick, but I even prefer “how sure are you” ASI will kill us all or even just “I think there’s a high probability that...”
By my definition of confidence then, Bentham’s Bulldog is far less confident than you in his prediction of 2.6%. He doesn’t quote his error bars but expresses that he is very uncertain, and wide error bars are implicit in his probability tree method as well. YS on the otherhand seem to have very narrow error bars around their claim “if anyone builds ASI with modern methods, everyone will die.”
Thanks for this comment! (I also saw you commented on the EA forum. I’m just going to respond here because I’m a LW guy and want to keep things simple.)
As you said, the median expert gives something like a 5% chance of doom. BB’s estimate is about a factor of two more confident than that that things will be okay. That factor of two difference is what I’m referencing. I am trying to say that I think it would be wiser for BB to be a factor of two less confident, like Ord. I’m not sure what doesn’t seem right about what I wrote.
I agree that superforecasters are even more confident than BB. I also agree that many domain experts are more confident.
I think that BB and I are using Bayesian language where “confidence” means “degree of certainty,” rather than degree of calibration or degree of meta-certainty or size of some hypothetical confidence interval or whatever. I agree that Y&S think the title thesis is “an easy call” and that they have not changed their mind on it even after talking to a lot of people. I buy that BB’s beliefs here are less stable/fixed.
I think there is a disconnect here related to different usages of the word “confidence”. You say in the OP:
Which I would interpret as being 1- P(not-okay), in other words 1 − 0.026 = 97.4% for BB, very confident.
On the other hand, I think many people probably believe that extinction from misaligned AI is very unlikely apriori, and so might use “confidence” in a sense that is relative to priors. To understand why people might do this, lets imagine that I said “there is a 55% chance that irrelevant AI blogger The Floating Droid will win the 2028 US presidential election”. Now imagine someone said “Wow! That’s insanely overconfident!”. I think people would be a bit suspicious if I responded that it was actually pretty unconfident because it quotes a probability near 50%.
I think this different usages of “confidence” is also relevant to the OP since it is reviewing BBs statements. For example, your statement:
Reads to me as a suggestion of hypocrisy or contadiction. BB accuse YS of being overconfident when really he is the one who is being extremely confident! But for this to be the case, we would need to evaluate the statement using the notion of “confidence” that was intended by BB. Its not clear to me that your post is actually using the same notion of “confidence”.
Hmmm… Good point. I’ll reach out to Bentham’s Bulldog and ask him what he even means by “confidence.” Thanks.
Glad the comment was helpful. I will register my prediction that BB most likely meant the “relative to priors” meaning rather than the one that you use in the OP. I also think among people who aren’t steeped in the background of AI risk, this would be the significantly more common interpretation upon reading what BB wrote.