What I meant is that if I have a choice either to do something that has a very tiny chance of having a very large good effect (e.g., working on friendly AI and possibly preventing a hostile takeover of the world by nasty AI) or to do something with a high chance of having a small good effect (e.g., teaching math to university students), I may take the latter option where others may take the former. Neither need be irrational.
Problem is that the expected utility of an outcome often grows faster than its probability shrinks. If the utility you assign to a galactic civilization does not outweigh the low probability of success you can just take into the account all beings that could be alive until the end of time in the case of a positive Singularity. Whatever you care about now, there will be so much more of it after a positive Singularity that it does always outweigh the tiny probability of it happening.
Hmm, I wonder if there is a bias in human cognition, which makes it easier for us to think of ever larger utilities/disutilities than of ever tinier probabilities. My intuition says the former, which is why I tend to be skeptical of such large impact small probability events.
You raise this issue a lot, so now I’m curious how you cash it out in actual numbers:
For the sake of concreteness, call Va the value to you of 10% of your total assets at this moment. (In other words, going from your current net worth to 90% of your net worth involves a loss of Va.)
What’s your estimate of the value of a positive Singularity in terms of Va?
What’s your estimate of the probability (P%) of a positive Singularity at this moment?
What’s your estimate of how much P% would increment by if you invested an additional 10% of your total assets at this moment in the most efficient available mechanism for incrementing P%?
Problem is that the expected utility of an outcome often grows faster than its probability shrinks. If the utility you assign to a galactic civilization does not outweigh the low probability of success you can just take into the account all beings that could be alive until the end of time in the case of a positive Singularity. Whatever you care about now, there will be so much more of it after a positive Singularity that it does always outweigh the tiny probability of it happening.
Hmm, I wonder if there is a bias in human cognition, which makes it easier for us to think of ever larger utilities/disutilities than of ever tinier probabilities. My intuition says the former, which is why I tend to be skeptical of such large impact small probability events.
You raise this issue a lot, so now I’m curious how you cash it out in actual numbers:
For the sake of concreteness, call Va the value to you of 10% of your total assets at this moment. (In other words, going from your current net worth to 90% of your net worth involves a loss of Va.)
What’s your estimate of the value of a positive Singularity in terms of Va?
What’s your estimate of the probability (P%) of a positive Singularity at this moment?
What’s your estimate of how much P% would increment by if you invested an additional 10% of your total assets at this moment in the most efficient available mechanism for incrementing P%?