Hello everyone, Like many people, I come to this site via an interest in transhumanism, although it seems unlikely to me that FAI implementing CEV can actually be designed before the singularity
Many here would agree with you. (And, for instance, consider a ~10% chance of success better than near certain extinction.)
I agree that 10% chance of success is better than near zero, and furthermore I agree that expected utility maximization means that putting in a great deal of effort to achieve a positive outcome is wiser than saying “oh well, we’re doomed anyway, might as well party hard and make the most of the time we have left”. However, the question is whether, if FAI has a low probability of success, are other possibilities, e.g. tool AI a better option to pursue?
The last survey reported that Less Wrongers on average believe that humanity has about a 68% chance of surviving the century without a disaster killing >90% of the species. (Median 80%, though, which might be a better measure of the community feeling than the mean in this case.) That’s a lower bar than actual extinction, but also a shorter timescale, so I expect the answer to your question would be in the same ballpark.
Many here would agree with you. (And, for instance, consider a ~10% chance of success better than near certain extinction.)
I agree that 10% chance of success is better than near zero, and furthermore I agree that expected utility maximization means that putting in a great deal of effort to achieve a positive outcome is wiser than saying “oh well, we’re doomed anyway, might as well party hard and make the most of the time we have left”. However, the question is whether, if FAI has a low probability of success, are other possibilities, e.g. tool AI a better option to pursue?
Would you say that many people here (and yourself?) believe that the probable end of our species is within the next century or two?
The last survey reported that Less Wrongers on average believe that humanity has about a 68% chance of surviving the century without a disaster killing >90% of the species. (Median 80%, though, which might be a better measure of the community feeling than the mean in this case.) That’s a lower bar than actual extinction, but also a shorter timescale, so I expect the answer to your question would be in the same ballpark.
For myself: Yes! p(extinct within 200 years) > 0.5