All life forms are optimized, by evolution, for self-replication.
It remains to be seen if evolutionary pressures will optimize future AIs for self replication, or only for patient values and aggressive power-seeking.
Genes are selected through their replicas being more present. Values are data for delegation, and delegation extends influence beyond the principal. Principals are then selected through their influence, and no copying needs to be involved. But the resources under the influence of a principal are subject to their values, so in a post-alignment world they get coherently optimized, as a single design rather than as some sort of commoditized bulk goodness spoken of with mass nouns.
Yes, it’s mysterious. With a singleton AI, selection doesn’t apply. So we could only say like, in the long run the replication-only aliens occupy more space than the side project ones. (Naively, it would seem you could still at least “defend your territory” if you spend 1% on fun.)
Still, when I hear speculations of what unaligned ASI will be doing it doesn’t mesh. Replicator feels to me an obvious default, some simplicity intuition as they call it.
Like if we imagine the universe across time, feels clean if the right part of the graph is “dumb superintelligent replicator era”
All life forms are optimized, by evolution, for self-replication.
It remains to be seen if evolutionary pressures will optimize future AIs for self replication, or only for patient values and aggressive power-seeking.
Genes are selected through their replicas being more present. Values are data for delegation, and delegation extends influence beyond the principal. Principals are then selected through their influence, and no copying needs to be involved. But the resources under the influence of a principal are subject to their values, so in a post-alignment world they get coherently optimized, as a single design rather than as some sort of commoditized bulk goodness spoken of with mass nouns.
Yes, it’s mysterious. With a singleton AI, selection doesn’t apply. So we could only say like, in the long run the replication-only aliens occupy more space than the side project ones. (Naively, it would seem you could still at least “defend your territory” if you spend 1% on fun.)
Still, when I hear speculations of what unaligned ASI will be doing it doesn’t mesh. Replicator feels to me an obvious default, some simplicity intuition as they call it.
Like if we imagine the universe across time, feels clean if the right part of the graph is “dumb superintelligent replicator era”