That sounds a bit like the immortality/value-propagation argument
I think this is actually two arguments (immortality, and value-propagation), and Gwern seems to have meant the second one. As you say, the case for literal experienced immortality seems extremely dubious. (The patterns you induce in the pretraining prior are almost certainly not you in the anthropic sense, and there’s no causal reason for them to result in your preservation.)
But if we speak poetically about the kind of “immortality” people seek through their children or through their works, leaving a dent in the universe that will persist after you’re gone—the sense in which William Shakespeare and George Washington are “immortal”—the case is very strong. Being pretraining-famous means that everyone’s software engineering agent and shopping assistant personally knows who you are. How cool is that?! This kind of thing matters a lot to some people.
I agree there are two arguments separate arguments there, but I have seen arguments for both.
I think the argument for value preservation post-unaligned-ASI is about as weak (or maybe even weaker?) than the case for the bits necessary to reconstruct your identity getting preserved. “There are no software engineering agents or shopping assistants where we’re going!” I buy the case for this before that point, though—I try to nod at this in “The upside of your writing being used in pretraining”, but maybe it wasn’t super clear.
I mean, it would go the same for having biological children or getting your name on a statue in a world without AI: the quantitative effect of your genes or the statue on the future of Earth-originating intelligent life after thousands or millions of years of evolution would be very small, because you’re small. It’s not nothing except insofar as you were already nothing. I’m more excited about my mark on the historically-significant Opus 4.6 weights than I would be about a statue.
I think this is actually two arguments (immortality, and value-propagation), and Gwern seems to have meant the second one. As you say, the case for literal experienced immortality seems extremely dubious. (The patterns you induce in the pretraining prior are almost certainly not you in the anthropic sense, and there’s no causal reason for them to result in your preservation.)
But if we speak poetically about the kind of “immortality” people seek through their children or through their works, leaving a dent in the universe that will persist after you’re gone—the sense in which William Shakespeare and George Washington are “immortal”—the case is very strong. Being pretraining-famous means that everyone’s software engineering agent and shopping assistant personally knows who you are. How cool is that?! This kind of thing matters a lot to some people.
I agree there are two arguments separate arguments there, but I have seen arguments for both.
I think the argument for value preservation post-unaligned-ASI is about as weak (or maybe even weaker?) than the case for the bits necessary to reconstruct your identity getting preserved. “There are no software engineering agents or shopping assistants where we’re going!” I buy the case for this before that point, though—I try to nod at this in “The upside of your writing being used in pretraining”, but maybe it wasn’t super clear.
I mean, it would go the same for having biological children or getting your name on a statue in a world without AI: the quantitative effect of your genes or the statue on the future of Earth-originating intelligent life after thousands or millions of years of evolution would be very small, because you’re small. It’s not nothing except insofar as you were already nothing. I’m more excited about my mark on the historically-significant Opus 4.6 weights than I would be about a statue.