I really want to see the post on multidimensional pessimism.
As for why, I’d argue 1 is happening.
For examples of 1, a good example of this is FOOM probabilities. I think MIRI hasn’t updated on the evidence that FOOM is likely impossible for classical computers, and this ought to lower their probabilities to the chance that quantum/reversible computers appear.
Another good example is the emphasis on pivotal acts like “burn all GPUs.” I think MIRI has too much probability mass on it being necessary, primarily because I think that they are biased by fiction, where problems must be solved by heroic acts, while in the real world more boring things are necessary. In other words, it’s too exciting, which should be suspicious.
However that doesn’t mean alignment is much easier. We can still fail, there’s no rule that we make it through. It’s that MIRI is systematically irrational here regarding doom probabilities or alignment.
I really want to see the post on multidimensional pessimism.
As for why, I’d argue 1 is happening.
For examples of 1, a good example of this is FOOM probabilities. I think MIRI hasn’t updated on the evidence that FOOM is likely impossible for classical computers, and this ought to lower their probabilities to the chance that quantum/reversible computers appear.
Another good example is the emphasis on pivotal acts like “burn all GPUs.” I think MIRI has too much probability mass on it being necessary, primarily because I think that they are biased by fiction, where problems must be solved by heroic acts, while in the real world more boring things are necessary. In other words, it’s too exciting, which should be suspicious.
However that doesn’t mean alignment is much easier. We can still fail, there’s no rule that we make it through. It’s that MIRI is systematically irrational here regarding doom probabilities or alignment.