“I take it that your goals do not involve avoiding all your distant descendants being systematically obliterated. If you don’t care about such an outcome, fine. I do happen to care.”
In what sense is the descendant (through many iterations of redesign and construction) of an AI solely focused on survival, constructed by some other human my descendant or yours? What features does it possess that make you care about it more than the descendant of an AI constructed by aliens evolved near a distant star? If it’s just a causal relation to your species, then what do you make of the following case: you create your survival machine, and it encounters a similar alien AI, whereupon the two merge, treating the activities of the merged entity as satisfying both of their ‘survival’ aims.
Where does your desire come from? Its achievement wouldn’t advance the preservation of your genes (those would be destroyed), wouldn’t seem to stem from the love of human children, wouldn’t preserve your personality, etc.
Tim,
Let’s assume that the convergent utility function supported by natural selection is that of a pure survival machine (although it’s difficult to parse this, since the entities you’re talking about seem indifferent to completely replacing all of their distinguishing features), stripped of any non-survival values of the entity’s ancestors. In other words, there’s no substantive difference between the survival-oriented alien invaders and human-built survival machines, so why bother to pre-emptively institute the outcome of invasion? Instead we could pursue what we conclude, on reflection is good, trading off between consumption and investment (including investments in expansion and defense) so as to maximize utility. If a modestly increased risk of destruction by aliens is compensated for by much greater achievement of our aims, why should we instead abandon our aims in order to create exactly the outcome we supposedly fear?