I discussed this a bit with Oliver and I think I understand better what the objection is.
I think the substantive disagreement is whether the actual empirical iteration you get on the current trajectory is a big enough deal that you believe that alignment is difficult (say, p(doom on current trajectory) > 80%) just due to oneshotness.
However many of the quotes above are instead saying something to the effect of “Eliezer thinks that empirical iteration is unimportant / provides no alignment-relevant info”. This is in fact a different thing; one can consistently believe both (1) alignment is very difficult and won’t be solved given the amount and quality of empirical iteration we get by default, and (2) empirical iteration is incredibly valuable.
And of course Eliezer believes both (1) and (2); (1) is just a statement of his most prominently known view, and for (2) the value of empirical iteration is blatantly obvious and it would be shocking if Eliezer disagreed with it.
I do pretty strongly disagree with the psychologizing that Eliezer does in the post if that is supposed to apply to the authors of the quotes above (as opposed to e.g. randos on Twitter), e.g.
The opposing faction is forced to that effort of hurried misinterpretation, because the actual impact of the concept of “we only get one shot at ASI” is so devastating to their position, in the presence of even a slight understanding of why there might be any noteworthy engineering difficulties whatsoever.
There is in fact a substantive disagreement and it’s not the case that Eliezer’s position is so self-evident that of course everyone else must be forced to do a hurried misinterpretation. (I initially hadn’t even realized that this post was maybe supposed to be responding to the people that Oliver quotes above, because according to me the majority of them obviously understand the idea of oneshotness.)
I suggest that another underlying source of disagreement could be about the general factor of: what is this function, approximately:
[the similarity of all previous tries put together, to try X]
--> [how much should we expect to fail on try X on the grounds that it’s first-try-ish, i.e. how much is it Murphy’s Cursed]
If you think that even a fair amount of similarity still doesn’t get you to success on one-shot problems, then you’d talk about oneshotness as being a strong argument against AGI alignment attempts working out well. This kinda sounds like what Yudkowsky is arguing in this post. Someone else could disagree with that.
I discussed this a bit with Oliver and I think I understand better what the objection is.
I think the substantive disagreement is whether the actual empirical iteration you get on the current trajectory is a big enough deal that you believe that alignment is difficult (say, p(doom on current trajectory) > 80%) just due to oneshotness.
However many of the quotes above are instead saying something to the effect of “Eliezer thinks that empirical iteration is unimportant / provides no alignment-relevant info”. This is in fact a different thing; one can consistently believe both (1) alignment is very difficult and won’t be solved given the amount and quality of empirical iteration we get by default, and (2) empirical iteration is incredibly valuable.
And of course Eliezer believes both (1) and (2); (1) is just a statement of his most prominently known view, and for (2) the value of empirical iteration is blatantly obvious and it would be shocking if Eliezer disagreed with it.
I do pretty strongly disagree with the psychologizing that Eliezer does in the post if that is supposed to apply to the authors of the quotes above (as opposed to e.g. randos on Twitter), e.g.
There is in fact a substantive disagreement and it’s not the case that Eliezer’s position is so self-evident that of course everyone else must be forced to do a hurried misinterpretation. (I initially hadn’t even realized that this post was maybe supposed to be responding to the people that Oliver quotes above, because according to me the majority of them obviously understand the idea of oneshotness.)
I suggest that another underlying source of disagreement could be about the general factor of: what is this function, approximately:
If you think that even a fair amount of similarity still doesn’t get you to success on one-shot problems, then you’d talk about oneshotness as being a strong argument against AGI alignment attempts working out well. This kinda sounds like what Yudkowsky is arguing in this post. Someone else could disagree with that.