I think the bolded text is about Yudkowsky himself being wrong.
That is also how I interpreted it.
If you have a bunch of specific arguments and sources of evidence that you think all point towards a particular conclusion X, then discovering that you’re wrong about something should, in expectation, reduce your confidence in X.
I think Yudkowsky is making a different statement. I agree it would be bizarre for him to be saying “if I were wrong, it would only mean I should have been more confident!”
Yudkowsky is not the aerospace engineer building the rocket who’s saying “the rocket will work because of reasons A, B, C, etc”.
I think he is (inside of the example). He’s saying “suppose an engineer is wrong about how their design works. Is it more likely that the true design performs better along multiple important criteria than expectation, or that the design performs worse (or fails to function at all)?”
Note that ‘expectation’ is referring to the confidence level inside an argument, but arguments aren’t Bayesians; it’s the outside agent that shouldn’t be expected to predictably update. Another way to put this: does the engineer expect to be disappointed, excited, or neutral if the design doesn’t work as planned? Typically, disappointed, implying the plan is overly optimistic compared to reality.
If this weren’t true—if engineers were calibrated or pessimistic—then I think Yudkowsky would be wrong here (and also probably have a different argument to begin with).
That is also how I interpreted it.
I think Yudkowsky is making a different statement. I agree it would be bizarre for him to be saying “if I were wrong, it would only mean I should have been more confident!”
I think he is (inside of the example). He’s saying “suppose an engineer is wrong about how their design works. Is it more likely that the true design performs better along multiple important criteria than expectation, or that the design performs worse (or fails to function at all)?”
Note that ‘expectation’ is referring to the confidence level inside an argument, but arguments aren’t Bayesians; it’s the outside agent that shouldn’t be expected to predictably update. Another way to put this: does the engineer expect to be disappointed, excited, or neutral if the design doesn’t work as planned? Typically, disappointed, implying the plan is overly optimistic compared to reality.
If this weren’t true—if engineers were calibrated or pessimistic—then I think Yudkowsky would be wrong here (and also probably have a different argument to begin with).