Even if your model says there’s a high probability of X, it doesn’t mean X is very likely. You also need to take into account the probability that the model itself is right. See: When the uncertainty about the model is higher than the uncertainty in the model For example, you could ask yourself: what’s the probability that I could read something that changed my mind about the validity of this model?
Beliefs vs impressions
Even if you have the impression that X is true, it might still be prudent to believe that maybe ~X if:
a lot of people you (otherwise) trust epistemologically disagree
a lot of our thinking on this seems still confused
it seems like we’re still making progress on the topic
it seems likely that there’s a lot of unknown unknowns
this type of question has a poor track record at being tackled accurately
you have been wrong with similar beliefs in the past
Option value is generally useful; it’s a convergent instrumental goal. Even if you are confident about some model of the world or moral framework, it might still remain a priority to keep your options open just in case your wrong. See Hard-to-reverse decisions destroy option value.
Group rationality
Promoting a norm of taking actions even when they are based on a model of the world few people share seems bad. See Unilateralist’s curse
taking actions even when they are based on a model of the world few people share seems bad. See Unilateralist’s curse
This depends a lot on the actions (and the world). If few people think cryonics will work out, but you do it anyway...oh no! You might live longer!
Some actions don’t seem dangerous.
If few people think the spread of disease can be reduced by washing hands, but you’re in charge of the hospital, and you make people do it, and see that the rate of patients dying (or getting sick) drops enormously, then why not continue? Why not roll it out to the rest of the hospital?*
Empiricism may march ahead of consensus, especially when there’s a lot of very low hanging fruit.*
*Historically this once included ‘wash your hands after conducting an autopsy, and especially before doing anything to help out with a pregnancy’.
Ah, yes absolutely! I should have specified my original claim further to something like “when it affects the whole world” and “a lot of people you’ve identified as rational disagree”.
Model uncertainty
Even if your model says there’s a high probability of X, it doesn’t mean X is very likely. You also need to take into account the probability that the model itself is right. See: When the uncertainty about the model is higher than the uncertainty in the model For example, you could ask yourself: what’s the probability that I could read something that changed my mind about the validity of this model?
Beliefs vs impressions
Even if you have the impression that X is true, it might still be prudent to believe that maybe ~X if:
a lot of people you (otherwise) trust epistemologically disagree
a lot of our thinking on this seems still confused
it seems like we’re still making progress on the topic
it seems likely that there’s a lot of unknown unknowns
this type of question has a poor track record at being tackled accurately
you have been wrong with similar beliefs in the past
etc.
See: Beliefs vs impressions
Option value
Option value is generally useful; it’s a convergent instrumental goal. Even if you are confident about some model of the world or moral framework, it might still remain a priority to keep your options open just in case your wrong. See Hard-to-reverse decisions destroy option value.
Group rationality
Promoting a norm of taking actions even when they are based on a model of the world few people share seems bad. See Unilateralist’s curse
This depends a lot on the actions (and the world). If few people think cryonics will work out, but you do it anyway...oh no! You might live longer!
Some actions don’t seem dangerous.
If few people think the spread of disease can be reduced by washing hands, but you’re in charge of the hospital, and you make people do it, and see that the rate of patients dying (or getting sick) drops enormously, then why not continue? Why not roll it out to the rest of the hospital?*
Empiricism may march ahead of consensus, especially when there’s a lot of very low hanging fruit.*
*Historically this once included ‘wash your hands after conducting an autopsy, and especially before doing anything to help out with a pregnancy’.
Ah, yes absolutely! I should have specified my original claim further to something like “when it affects the whole world” and “a lot of people you’ve identified as rational disagree”.