Without digging in too much, I’ll say that this exchange and the OP is pretty confusing to me. It sounds like MB is like “MIRI doesn’t say it’s hard to get an AI that has a value function” and then also says “GPT has the value function, so MIRI should update”. This seems almost contradictory.
A guess: MB is saying “MIRI doesn’t say the AI won’t have the function somewhere, but does say it’s hard to have an externally usable, explicit human value function”. And then saying “and GPT gives us that”, and therefore MIRI should update.
And EY is blobbing those two things together, and saying neither of them is the really hard part. Even having the externally usable explicit human value function doesn’t mean the AI cares about it. And it’s still a lot of bits, even if you have the bits. So it’s still true that the part about getting the AI to care has to go precisely right.
If there’s a substantive disagreement about the facts here (rather than about the discourse history or whatever), maybe it’s like:
Straw-EY: Complexity of value means you can’t just get the make-AI-care part to happen by chance; it’s a small target.
Straw-MB: Ok but now we have a very short message pointing to roughly human values: just have a piece of code that says “and now call GPT and ask it what’s good”. So now it’s a very small number of bits.
This seems really implausible. I’d like to see a debate about this. E.g. why can’t I improve on heat by having super-cooled fluid pumped throughout my artificial brain; doesn’t having no skull-size limit help a lot; doesn’t metal help; doesn’t it help to not have to worry about immune system stuff; doesn’t it help to be able to maintain full neuroplasticity; etc.