I disagree with your implication and agree with OP: Inner alignment is not yet solved, therefore we don’t know how to make an AGI that is “trying” to do something in particular, and by extension, we don’t know how to make an AGI with a particular utility function. I was actually going to comment that OP is not only correct but uncontroversial, at least among experts. (That doesn’t mean it’s a pointless post, good pedagogy is always welcome.) So I’m surprised and confused by your comment.
I am aware of Reinforcement Learning (I am actually sitting right next to Sutton’s book on the field, which I have fully read), but I think you are right that my point is not very clear.
The way I see it RL goals are really only the goals of the base optimizer. The agents themselves either are not intelligent (follow simple procedural ‘policies’) or are mesa-optimizers that may learn to follow something else entirely (proxies, etc). I updated the text, let me know if it makes more sense now.
You should look up the machine learning subfield of reinforcement learning.
I disagree with your implication and agree with OP: Inner alignment is not yet solved, therefore we don’t know how to make an AGI that is “trying” to do something in particular, and by extension, we don’t know how to make an AGI with a particular utility function. I was actually going to comment that OP is not only correct but uncontroversial, at least among experts. (That doesn’t mean it’s a pointless post, good pedagogy is always welcome.) So I’m surprised and confused by your comment.
That is fair enough, I was just confused and thought OP had not heard about it because there wasn’t a hint of it in the post.
I am aware of Reinforcement Learning (I am actually sitting right next to Sutton’s book on the field, which I have fully read), but I think you are right that my point is not very clear.
The way I see it RL goals are really only the goals of the base optimizer. The agents themselves either are not intelligent (follow simple procedural ‘policies’) or are mesa-optimizers that may learn to follow something else entirely (proxies, etc). I updated the text, let me know if it makes more sense now.