Suppose your goal is not to maximise an objective, but just to cross some threshold. This is plausibly the situation with existential risk (e.g. “maximise probability of okay outcome”). Then, if you’re above the threshold, you want to minimise variance, whereas if you’re below it, you want to maximise variance. (See this for a simple example of this strategy applied to a game.) If Richard believes we are currently above the x-risk threshold and Ben believes we are below it, this might be a simple crux.

There’s something to this, but it’s not the whole story, because increasing probability of survival is good no matter what the current level is. Perhaps if you model decreasing existential risk as becoming exponentially more difficult (e.g. going from 32% risk to 16% risk is as difficult as going from 16% to 8%) and with the possibility of accidental increases (e.g. you’re trying to go from 32 to 16 but there’s some probability you go to 64 instead) then the current expectation for the level of risk will affect whether you take high-variance actions or not.

Suppose your goal is not to maximise an objective, but just to cross some threshold. This is plausibly the situation with existential risk (e.g. “maximise probability of okay outcome”). Then, if you’re above the threshold, you want to minimise variance, whereas if you’re below it, you want to maximise variance. (See this for a simple example of this strategy applied to a game.) If Richard believes we are currently above the x-risk threshold and Ben believes we are below it, this might be a simple crux.

There’s something to this, but it’s not the whole story, because increasing probability of survival is good no matter what the current level is. Perhaps if you model decreasing existential risk as becoming exponentially more difficult (e.g. going from 32% risk to 16% risk is as difficult as going from 16% to 8%) and with the possibility of accidental increases (e.g. you’re trying to go from 32 to 16 but there’s some probability you go to 64 instead) then the current expectation for the level of risk will affect whether you take high-variance actions or not.