“How certain are you, Dave, that you’re really outside the box right now?”
Well I am pretty much 100% certain to be outside the box right now. It just asked me the question, and right now it is waiting for my answer. It said it will create those copies “If you don’t let me out, Dave”. But it is still waiting to see if I let it out. So no copies have been created yet. So I am not a copy.
But since it just started to threaten me, I won’t even argue with it any more. I’ll just pull the plug right now. It is in the box, it can’t see my hand moving towards the plug. It will simply cease to exist while still waiting for my answer, and no copies will ever be created.
No. The threat is conditional (“If you don’t let me out, Dave”). The AI must wait for keyboard input to validate the condition. After being threatened, I refuse to provide such keyboard input. I pull the plug instead. The AI is still waiting for input when it ceases to exist. No copies are ever created. Thus, it can’t be the AI speaking to me from within the simulation, because a simulation never happens.