Your response was to my objection: “that doesn’t mean that at that point you know how to make it do what you mean.”
The superintelligent AI doesn’t have an issue with understanding your intentions, it simply doesn’t have any reason to care about your intentions.
In order to program it to care about your intentions, you, the programmer need to know how to codify the concept of “your intentions” (Perhaps not the specific intention, but the concept of what it means to have an intention). How do you do that?
Your response was to my objection: “that doesn’t mean that at that point you know how to make it do what you mean.”
The superintelligent AI doesn’t have an issue with understanding your intentions, it simply doesn’t have any reason to care about your intentions.
In order to program it to care about your intentions, you, the programmer need to know how to codify the concept of “your intentions” (Perhaps not the specific intention, but the concept of what it means to have an intention). How do you do that?
Funny, I would’ve phrased that the other way around.