I agree there are two arguments separate arguments there, but I have seen arguments for both.
I think the argument for value preservation post-unaligned-ASI is about as weak (or maybe even weaker?) than the case for the bits necessary to reconstruct your identity getting preserved. “There are no software engineering agents or shopping assistants where we’re going!” I buy the case for this before that point, though—I try to nod at this in “The upside of your writing being used in pretraining”, but maybe it wasn’t super clear.
I mean, it would go the same for having biological children or getting your name on a statue in a world without AI: the quantitative effect of your genes or the statue on the future of Earth-originating intelligent life after thousands or millions of years of evolution would be very small, because you’re small. It’s not nothing except insofar as you were already nothing. I’m more excited about my mark on the historically-significant Opus 4.6 weights than I would be about a statue.
I agree there are two arguments separate arguments there, but I have seen arguments for both.
I think the argument for value preservation post-unaligned-ASI is about as weak (or maybe even weaker?) than the case for the bits necessary to reconstruct your identity getting preserved. “There are no software engineering agents or shopping assistants where we’re going!” I buy the case for this before that point, though—I try to nod at this in “The upside of your writing being used in pretraining”, but maybe it wasn’t super clear.
I mean, it would go the same for having biological children or getting your name on a statue in a world without AI: the quantitative effect of your genes or the statue on the future of Earth-originating intelligent life after thousands or millions of years of evolution would be very small, because you’re small. It’s not nothing except insofar as you were already nothing. I’m more excited about my mark on the historically-significant Opus 4.6 weights than I would be about a statue.