I believe the extinction year question was asking for a median, not an expected value. In one place in the paper it is paraphrased as asking “by what year humanity is 50% likely to go extinct”.
If extinction caused by AI or value drift is somewhat unlikely, then extinction only happens once there is no more compute in the universe, which might take a very long time. So “the year humanity is 50% likely to go extinct” could be 1044 or something.
I believe the extinction year question was asking for a median, not an expected value. In one place in the paper it is paraphrased as asking “by what year humanity is 50% likely to go extinct”.
If extinction caused by AI or value drift is somewhat unlikely, then extinction only happens once there is no more compute in the universe, which might take a very long time. So “the year humanity is 50% likely to go extinct” could be 1044 or something.