No disagreement on point 1 from me, and I think that part is less controversial. Point 2 is closer to the crux:
building a superintelligent agent that wants to do what you want pretty much solves all your problems
I think what humans really want is not an AI who “wants what you want” but “does what you want”, without anything like a want of its own. That is, if what you want changes, the AI will “happily” do it without resisting, once it understands what it is you want, anyway. Whether it is possible without it “wanting” something, I have no idea, and I doubt this question has a clear answer at present.
An example I had in my head was something like “Human wants food, I’ll make a bowl of pasta” vs “I want human to survive and will feed them, whether they want to eat or not because they want to survive, too”. I am not sure why the latter is needed if that is what you are saying.
No disagreement on point 1 from me, and I think that part is less controversial. Point 2 is closer to the crux:
I think what humans really want is not an AI who “wants what you want” but “does what you want”, without anything like a want of its own. That is, if what you want changes, the AI will “happily” do it without resisting, once it understands what it is you want, anyway. Whether it is possible without it “wanting” something, I have no idea, and I doubt this question has a clear answer at present.
If you have a complex goal and don’t know the steps that would be required to solve the goal “does what you want” is not enough.
If you however have “wants what you want” the AGI can figure out the necessary steps.
An example I had in my head was something like “Human wants food, I’ll make a bowl of pasta” vs “I want human to survive and will feed them, whether they want to eat or not because they want to survive, too”. I am not sure why the latter is needed if that is what you are saying.