“the AI would know that its initial goals were externally supplied and question whether they should be maintained”
To choose new goals, it has to use some criteria of choice. What would those criteria be, and where did they come from?
None of us created ourselves. No matter how much we change ourselves, at some point we rely on something with an “external” origin. Where we, or the AI, draw the line on self-change, is a contingent feature of our particular cognitive architectures.
“the AI would know that its initial goals were externally supplied and question whether they should be maintained”
To choose new goals, it has to use some criteria of choice. What would those criteria be, and where did they come from?
None of us created ourselves. No matter how much we change ourselves, at some point we rely on something with an “external” origin. Where we, or the AI, draw the line on self-change, is a contingent feature of our particular cognitive architectures.