I agree that carefully landing the plane is better than maintaining the course if catastrophic outcomes suddenly seem more plausible than before.
Yeah, but my point is not about catastrophic risk—it’s about the risk/reward trade-off in general. You can have risk>reward in scenarios that aren’t catastrophic. Catastrophic risk is just a good general example of where things don’t add up to normality (catastrophic risks by nature correspond to not-normal scenarios and also coincide with high risk). Don’t promise yourself to steer the plane mostly as normal, promise yourself to pursue the path that reduces risk over all outcomes you’re uncertain about.
I don’t think it applies very strongly in your example about animal welfare, unless the protagonist has unusually high leverage on a big decision about to be made. The cost of continuing to stay in the old job for a few weeks while thinking things over (especially if leaving and then coming back would be infeasible) is plausibly worth the value of information thus gained.
Good point, it really depends on the details of the example but this is just because of the different risk-reward trade-offs, not because you ought to always treat things as adding up to normality. I’ll counter that while you shouldn’t leave the job (high risk, hard to reverse), you should see if you could use your PTO as soon as possible so you can figure things out without potentially causing further negative impact. It all depends on the risk-reward trade-off:
If stopping activism corresponds to something like leaving a job, which is hard to reverse, doing so involves taking on a lot of risk if you’re uncertain and waiting for a bit can reduce that risk.
If stopping activism corresponds to something like shifting your organizations priorities, and your organization’s path can be reversed, then stopping work (after satisfying all existing contracts of course) is pretty low risk and you should stop
If stopping activism corresponds to donating large amounts of money (in earning-to-give contexts), your strategy can easily be reversed and you should stop now.
This is true even if you only have “small” amounts of impact.
Caveat:
People engage in policies for many reasons at once. So if you think the goal of your policy is X, but it’s actually X, Y and Z, then dramatic actions justified on uncertainty about X alone will probably be harmful due to Y and Z effects even if its the appropriate decision with respect to X. Because it’s easy to notice when why a thing might go wrong (like X) and hard to notice why they’re going right (like Y and Z), adding-up-to-normality serves as a way to generally protect Y and Z.
Don’t know if you saw, but I updated the post yesterday because of your (and khafra’s) points.
Also, your caveat is a good reframe of the main mechanism behind the post.
I do still disagree with you somewhat, because I think that people going through a crisis of faith are prone to flailing around and taking naive actions that they would have reconsidered after a week or month of actually thinking through the implications of their new belief. Trying to maximize utility while making a major update is safe for ideal Bayesian reasoners, but it fails badly for actual humans.
In the absence of an external crisis, taking relatively safe actions (and few irreversible actions) is correct in the short term, and the status quo is going to be reasonably safe for most people if you’ve been living it for years. If you can back off from newly-suspected-wrong activities for the time being without doing so irreversibly, then yes that’s better.
I do still disagree with you somewhat, because I think that people going through a crisis of faith are prone to flailing around and taking naive actions that they would have reconsidered after a week or month of actually thinking through the implications of their new belief. Trying to maximize utility while making a major update is safe for ideal Bayesian reasoners, but it fails badly for actual humans.
Ah, yeah I agree with this observation—and it could be good to just assume things add up to normality as a general defense against people rapidly taking naive actions. Scarcity bias is a thing after all and if you get into a mindset where now is the time to act, it’s really hard to prevent yourself from acting irrationally.
I agree with this, but a counterpoint is that it’s very hard for people to change longstanding habits and behaviors at all, and sometimes a major internal update is a good moment to make significant behavior changes because that’s the only time most people can manage major behavioral changes at all.
Yeah, but my point is not about catastrophic risk—it’s about the risk/reward trade-off in general. You can have risk>reward in scenarios that aren’t catastrophic. Catastrophic risk is just a good general example of where things don’t add up to normality (catastrophic risks by nature correspond to not-normal scenarios and also coincide with high risk). Don’t promise yourself to steer the plane mostly as normal, promise yourself to pursue the path that reduces risk over all outcomes you’re uncertain about.
Good point, it really depends on the details of the example but this is just because of the different risk-reward trade-offs, not because you ought to always treat things as adding up to normality. I’ll counter that while you shouldn’t leave the job (high risk, hard to reverse), you should see if you could use your PTO as soon as possible so you can figure things out without potentially causing further negative impact. It all depends on the risk-reward trade-off:
If stopping activism corresponds to something like leaving a job, which is hard to reverse, doing so involves taking on a lot of risk if you’re uncertain and waiting for a bit can reduce that risk.
If stopping activism corresponds to something like shifting your organizations priorities, and your organization’s path can be reversed, then stopping work (after satisfying all existing contracts of course) is pretty low risk and you should stop
If stopping activism corresponds to donating large amounts of money (in earning-to-give contexts), your strategy can easily be reversed and you should stop now.
This is true even if you only have “small” amounts of impact.
Caveat:
People engage in policies for many reasons at once. So if you think the goal of your policy is X, but it’s actually X, Y and Z, then dramatic actions justified on uncertainty about X alone will probably be harmful due to Y and Z effects even if its the appropriate decision with respect to X. Because it’s easy to notice when why a thing might go wrong (like X) and hard to notice why they’re going right (like Y and Z), adding-up-to-normality serves as a way to generally protect Y and Z.
Don’t know if you saw, but I updated the post yesterday because of your (and khafra’s) points.
Also, your caveat is a good reframe of the main mechanism behind the post.
I do still disagree with you somewhat, because I think that people going through a crisis of faith are prone to flailing around and taking naive actions that they would have reconsidered after a week or month of actually thinking through the implications of their new belief. Trying to maximize utility while making a major update is safe for ideal Bayesian reasoners, but it fails badly for actual humans.
In the absence of an external crisis, taking relatively safe actions (and few irreversible actions) is correct in the short term, and the status quo is going to be reasonably safe for most people if you’ve been living it for years. If you can back off from newly-suspected-wrong activities for the time being without doing so irreversibly, then yes that’s better.
Ah, yeah I agree with this observation—and it could be good to just assume things add up to normality as a general defense against people rapidly taking naive actions. Scarcity bias is a thing after all and if you get into a mindset where now is the time to act, it’s really hard to prevent yourself from acting irrationally.
Huzzah, convergence! I appreciate the points you’ve made.
I agree with this, but a counterpoint is that it’s very hard for people to change longstanding habits and behaviors at all, and sometimes a major internal update is a good moment to make significant behavior changes because that’s the only time most people can manage major behavioral changes at all.