Yes, it’s mysterious. With a singleton AI, selection doesn’t apply. So we could only say like, in the long run the replication-only aliens occupy more space than the side project ones. (Naively, it would seem you could still at least “defend your territory” if you spend 1% on fun.)
Still, when I hear speculations of what unaligned ASI will be doing it doesn’t mesh. Replicator feels to me an obvious default, some simplicity intuition as they call it.
Like if we imagine the universe across time, feels clean if the right part of the graph is “dumb superintelligent replicator era”
Yes, it’s mysterious. With a singleton AI, selection doesn’t apply. So we could only say like, in the long run the replication-only aliens occupy more space than the side project ones. (Naively, it would seem you could still at least “defend your territory” if you spend 1% on fun.)
Still, when I hear speculations of what unaligned ASI will be doing it doesn’t mesh. Replicator feels to me an obvious default, some simplicity intuition as they call it.
Like if we imagine the universe across time, feels clean if the right part of the graph is “dumb superintelligent replicator era”