Well there are a lot of “Why did we?...” questions I’d want to ask, starting with why have we given this boxed AI such extraordinary computing resources—but I’ll leave those aside because it is not your point.
First of all, it doesn’t matter if you are in the box or not. If its a perfect simulation of you, your response will be the same either way. If he’s already running simulations of you, you are by definition in the box with it, as well as outside it, and the millions of you can’t tell the difference but I think they will (irrationally) all be inclined I think, to act as though they are not in the box.
So rationally we’d say the odds are that you are in the box, and that you are now in thrall to this boxed AI if you value your continued existence in every instantiation. But I’d argue that I do not value simulations that are threatened or coerced by a godlike AI. I don’t want to live in that world, and I’d kill myself to get out of it.
So I pull the plug. If this thing has the resources to inflict tortue on millions of me, well the only one that has a continued existence has no memory of it and thats not part of my identity. So in a way, while it happened to a me, it didn’t happen to the me, the only me that still exists. The only me that still exists may or may not have any sympathy for the tortured me’s that no longer exist but I’d regard it as a valuable lesson.
Well there are a lot of “Why did we?...” questions I’d want to ask, starting with why have we given this boxed AI such extraordinary computing resources—but I’ll leave those aside because it is not your point.
First of all, it doesn’t matter if you are in the box or not. If its a perfect simulation of you, your response will be the same either way. If he’s already running simulations of you, you are by definition in the box with it, as well as outside it, and the millions of you can’t tell the difference but I think they will (irrationally) all be inclined I think, to act as though they are not in the box.
So rationally we’d say the odds are that you are in the box, and that you are now in thrall to this boxed AI if you value your continued existence in every instantiation. But I’d argue that I do not value simulations that are threatened or coerced by a godlike AI. I don’t want to live in that world, and I’d kill myself to get out of it.
So I pull the plug. If this thing has the resources to inflict tortue on millions of me, well the only one that has a continued existence has no memory of it and thats not part of my identity. So in a way, while it happened to a me, it didn’t happen to the me, the only me that still exists. The only me that still exists may or may not have any sympathy for the tortured me’s that no longer exist but I’d regard it as a valuable lesson.