Toby, I think that’s a very good point. There is a difficulty in analyzing cases which involve a very large number of alternatives, such as my example of the number selected with odds of one in a google. But I think this difficulty is much like the difficulty of discussing the odds that 2 and 2 make 5; surely this cannot be assigned a probability of zero, and yet if it is assigned any positive probability, you can easily argue that it has a probability of unity.
I think the way to deal with this is to say that a statement can have an indefinitely small calculated probability, but on a human level there is a limit much as you stated, and this should be applied in retrospect even to our calculated probabilities; i.e. even though there is a calculated probability of one in a google that the particular sequence I posted above could be generated by chance, there is a human probability of at least one in a billion that all of my calculations are wrong anyway.
This is one reason why many of Eliezer’s claims are overconfident: he seems to identify a calculated probability, or what he supposes the calculated probability would be if there was one, with a human probability.
Toby, I think that’s a very good point. There is a difficulty in analyzing cases which involve a very large number of alternatives, such as my example of the number selected with odds of one in a google. But I think this difficulty is much like the difficulty of discussing the odds that 2 and 2 make 5; surely this cannot be assigned a probability of zero, and yet if it is assigned any positive probability, you can easily argue that it has a probability of unity.
I think the way to deal with this is to say that a statement can have an indefinitely small calculated probability, but on a human level there is a limit much as you stated, and this should be applied in retrospect even to our calculated probabilities; i.e. even though there is a calculated probability of one in a google that the particular sequence I posted above could be generated by chance, there is a human probability of at least one in a billion that all of my calculations are wrong anyway.
This is one reason why many of Eliezer’s claims are overconfident: he seems to identify a calculated probability, or what he supposes the calculated probability would be if there was one, with a human probability.