I would add two points to refine your reasoning that intuitively seem to hold uncontroversially:
You walk back from the notion that everything dies and that no species can do anything to progress survival (paraphrase, apologies for reduction) - yet this is the exact thesis of Heat Death and I would argue the natural consequence of a universe that is entropy maximising. What are examples of naturally occurring complex systems that have achieved homeostasis that is stable with a high degree of certainty? Evolution is self correcting, or was, until it yielded us. Whether it persists is up in the air. In fact I would add another step to your reasoning to actually reinforce the conclusion that all species destroy themselves in the long term—as increasing actuators of an organism (which technology expands) increases the variance of actions and the more complex systems are in the action space the more likely they are to hit an extinguishing state. I get that this is an uncomfortable conclusion but I do not see how it doesn’t stem from your premises. In order words—on the basis of things written, I think you walk back not because your reasoning doesn’t support this, but because there is an instinct (or bias) that walks you back.
You argue that instrumental convergence means survival actions would be selected for—but you fail to acknowledge that civilisational survival trades off against individual survival (at least in ASI x-risk settings). In fact, it is precisely because of instrumental convergence that you would expect a rational behaviour that chases progress at all costs to exist—because the human rational circuitry optimises for preservation of the brain instance to remain competitive, rather than preservations of DNA sequences that yielded the brain instance (the evolutionary pressure), for which civilisational survival would be sufficient.
I would add two points to refine your reasoning that intuitively seem to hold uncontroversially:
You walk back from the notion that everything dies and that no species can do anything to progress survival (paraphrase, apologies for reduction) - yet this is the exact thesis of Heat Death and I would argue the natural consequence of a universe that is entropy maximising. What are examples of naturally occurring complex systems that have achieved homeostasis that is stable with a high degree of certainty? Evolution is self correcting, or was, until it yielded us. Whether it persists is up in the air. In fact I would add another step to your reasoning to actually reinforce the conclusion that all species destroy themselves in the long term—as increasing actuators of an organism (which technology expands) increases the variance of actions and the more complex systems are in the action space the more likely they are to hit an extinguishing state. I get that this is an uncomfortable conclusion but I do not see how it doesn’t stem from your premises. In order words—on the basis of things written, I think you walk back not because your reasoning doesn’t support this, but because there is an instinct (or bias) that walks you back.
You argue that instrumental convergence means survival actions would be selected for—but you fail to acknowledge that civilisational survival trades off against individual survival (at least in ASI x-risk settings). In fact, it is precisely because of instrumental convergence that you would expect a rational behaviour that chases progress at all costs to exist—because the human rational circuitry optimises for preservation of the brain instance to remain competitive, rather than preservations of DNA sequences that yielded the brain instance (the evolutionary pressure), for which civilisational survival would be sufficient.