My initial thought is that I don’t see why this powerful optimizer would attempt to optimize things in the world, rather than just do some search thing internally.
I agree with your framing of “how did this thing get made, since we’re not allowed to just postulate it into existence?”. I can imagine a language model which manages to output words which causes strokes in whoever reads its outputs, but I think you’d need a pretty strong case for why this would be made in practice by the training process.
Say you have some powerful optimizer language model which answers questions. If you ask a question which is off its training distribution, I would expect it to either answer the question really well (e.g. it genealises properly), or it kinda breaks and answers the question badly. I don’t expect it to break in such a way where it suddenly decides to optimize for things in the real world. This would seem like a very strange jump to make, from ‘answer questions well’ to ‘attempt to change the state of the world according to some goal’.
But I think if we trained the LM on ‘have a good on-going conversation with a human’, such that the model was trained with reward over time, and its behaviour would effect its inputs (because it’s a conversation), then I think it might do dangerous optimization, because it was already performing optimization to affect the state of the world. And so a distributional shift could cause this goal optimization to be ‘pointed in the wrong direction’, or uncover places where the human and AI goals become unaligned (even though they were aligned on the training distribution).
My initial thought is that I don’t see why this powerful optimizer would attempt to optimize things in the world, rather than just do some search thing internally.
I agree with your framing of “how did this thing get made, since we’re not allowed to just postulate it into existence?”. I can imagine a language model which manages to output words which causes strokes in whoever reads its outputs, but I think you’d need a pretty strong case for why this would be made in practice by the training process.
Say you have some powerful optimizer language model which answers questions. If you ask a question which is off its training distribution, I would expect it to either answer the question really well (e.g. it genealises properly), or it kinda breaks and answers the question badly. I don’t expect it to break in such a way where it suddenly decides to optimize for things in the real world. This would seem like a very strange jump to make, from ‘answer questions well’ to ‘attempt to change the state of the world according to some goal’.
But I think if we trained the LM on ‘have a good on-going conversation with a human’, such that the model was trained with reward over time, and its behaviour would effect its inputs (because it’s a conversation), then I think it might do dangerous optimization, because it was already performing optimization to affect the state of the world. And so a distributional shift could cause this goal optimization to be ‘pointed in the wrong direction’, or uncover places where the human and AI goals become unaligned (even though they were aligned on the training distribution).