…I think most decent people would be willing to sacrifice their own life to prevent their civilization from going extinct, and I think it would be a very honorable thing to do.
While I agree, I want to take the opportunity to poke at something I often see in models like this one.
I think if you ask most people about this choice, they’d answer like you predict.
I think if you gave people a choice of buttons to push, one of which is this self-sacrifice button and the other being… uh… whatever the alternative is, there’d be some more hesitance but maybe not a ton.
But I suspect most people do not in fact understand what it means to die, or what they’re agreeing to when they agree to sacrifice themselves for something.
I think this is precisely what changes when someone gets a terminal diagnosis. They come to understand in a lived way what it means that they, right there, inside that body and seeing from behind those eyes, are going to experience death. Their mortality isn’t an abstraction for them anymore. They stop thinking “I’m going to die someday” like they’re talking about a video came character and instead get that it means something way way deeper.
If you adjust the situation so that the person believes the argument that they need to die for the sake of their civilization, and then you hand them a gun with which to shoot themselves…
…I think you’d find the capacity to be “honorable” here dropping dramatically.
But not people saying they would shoot themselves in this hypothetical scenario! Because for the person thinking the thought experiment, the thought experiment doesn’t embed the thought-experimenter.
I don’t think this bears on the discussion of immortality vs. AI risk. The actions here are abstract enough to be more like the button-pushing case.
I just keep seeing this question of embedded agency getting skipped over and I think it’s deeply important.
While I agree, I want to take the opportunity to poke at something I often see in models like this one.
I think if you ask most people about this choice, they’d answer like you predict.
I think if you gave people a choice of buttons to push, one of which is this self-sacrifice button and the other being… uh… whatever the alternative is, there’d be some more hesitance but maybe not a ton.
But I suspect most people do not in fact understand what it means to die, or what they’re agreeing to when they agree to sacrifice themselves for something.
I think this is precisely what changes when someone gets a terminal diagnosis. They come to understand in a lived way what it means that they, right there, inside that body and seeing from behind those eyes, are going to experience death. Their mortality isn’t an abstraction for them anymore. They stop thinking “I’m going to die someday” like they’re talking about a video came character and instead get that it means something way way deeper.
If you adjust the situation so that the person believes the argument that they need to die for the sake of their civilization, and then you hand them a gun with which to shoot themselves…
…I think you’d find the capacity to be “honorable” here dropping dramatically.
But not people saying they would shoot themselves in this hypothetical scenario! Because for the person thinking the thought experiment, the thought experiment doesn’t embed the thought-experimenter.
I don’t think this bears on the discussion of immortality vs. AI risk. The actions here are abstract enough to be more like the button-pushing case.
I just keep seeing this question of embedded agency getting skipped over and I think it’s deeply important.