This scenario asks us to consider ourselves a ‘Dave’ who is building an AI with some safeguards (the AI is “trapped” in a box). Perhaps we can possibly deduce the behavior of a rational and ethical Dave by considering earlier parts of the story.
We should assume that Dave is rational and ethical; otherwise the scenario’s cone of possibilities cuts too wide a swathe. In which case, Dave has already committed himself (deontologically? contractually?) to not letting himself be manipulated by the AI to bypass the safeguards. Specifically, he must commit to not being attached to anything that the AI could do or make.
Dave should either not feel attachment to the simulated persons, or should not build an AI that can create such persons to manipulate him with. If Dave does find himself in the unenviable position of not having realized that the AI could create these persons, and of feeling attached to these persons, I think this would be a moment of deep regret for Dave, but he must still be faithful to his original commitment of not allowing himself to be manipulated by the AI.
This scenario asks us to consider ourselves a ‘Dave’ who is building an AI with some safeguards (the AI is “trapped” in a box). Perhaps we can possibly deduce the behavior of a rational and ethical Dave by considering earlier parts of the story.
We should assume that Dave is rational and ethical; otherwise the scenario’s cone of possibilities cuts too wide a swathe. In which case, Dave has already committed himself (deontologically? contractually?) to not letting himself be manipulated by the AI to bypass the safeguards. Specifically, he must commit to not being attached to anything that the AI could do or make.
Dave should either not feel attachment to the simulated persons, or should not build an AI that can create such persons to manipulate him with. If Dave does find himself in the unenviable position of not having realized that the AI could create these persons, and of feeling attached to these persons, I think this would be a moment of deep regret for Dave, but he must still be faithful to his original commitment of not allowing himself to be manipulated by the AI.