I feel like you’re overreacting to this. Surely the most likely explanation is that talking to LLMs is some evidence that LLMs will be aligned in the limit of power and intelligence, but (a) Davidad is overconfident for non-AI-psychosis reasons, (b) current quasi-alignment is due to the hard work of alignment researchers, and/or (c) precautionary principle, and so alignment researchers shouldn’t go home just yet?
Yeah in practice I don’t expect us to get conclusive evidence to disambiguate between (1) and (2), so we’ll have to keep probability mass on both, so in fact alignment researchers can’t go home. It’s still very surprising to me that this is where we ended up.
I feel like you’re overreacting to this. Surely the most likely explanation is that talking to LLMs is some evidence that LLMs will be aligned in the limit of power and intelligence, but (a) Davidad is overconfident for non-AI-psychosis reasons, (b) current quasi-alignment is due to the hard work of alignment researchers, and/or (c) precautionary principle, and so alignment researchers shouldn’t go home just yet?
Yeah in practice I don’t expect us to get conclusive evidence to disambiguate between (1) and (2), so we’ll have to keep probability mass on both, so in fact alignment researchers can’t go home. It’s still very surprising to me that this is where we ended up.