It felt obvious to me that once the probes are starting to make each other into paperclips, some sort of natural selection would take over where probes that prioritize survival over paperclip making would murder the rest and survive.
I don’t think that’s obvious. Evolution of interesting or complex traits is not an immutable fact of existence which magicks agents and novelty out of nowhere to endless degrees; it is a predictable, understandable, and mathematically-modelable consequence of an algorithm* of replication, heritable variation, and selection which operates only within particular ranges of parameters. If the cycle doesn’t happen fast enough, or if variation is too minimal, or if the variation is uncorrelated with fitness, or if a bad variant drifts to fixation… You will see things just freeze in place, mutational meltdown leading to extinction of the replicators, go through rock-paper-scissor cycles endlessly, or all sorts of other failure modes which look nothing like “ever more competent agents taking over the universe in an interesting way”. McClune’s AI-GA program is a quite difficult research program because evolution in a rich way like biological life is hard to capture in an artificial bottle, and multi-agent RL is even more notoriously fickle and unstable and irreplicable than regular DRL (itself infamous for driving you to tears); the field of evolutionary computation has struggled since its inception with the problem that you don’t get any progress reliably, or novelty disappears rapidly and permanently and converges to some extremely boring and useless equilibrium (see my other comment about how things like Core Wars typically collapse: they just aren’t rich enough or correctly constructed enough, in some sense, to support continual change). We live in a universe where life is allowed to just die out. Or to go nowhere. For intelligent life to be knocked back by an asteroid for a billion years, or life to chase its own tail until the Sun turns into a red giant and renders Earth a liquid ball of rock. Or for it to take too long (when it takes hundreds of millions or billions of years for one lineage of probes to travel from one end of the Hubble volume to the next, just how many ‘generations’ or ‘cycles’ is there time for before it’s all paperclipped? it only takes a few ‘generations’ at exponential growth to paperclip the universe). Or for basic error-correction mechanisms like cryptographic signatures to allow only a tiny number of replication errors uncorrelated with complex behaviors and fitness, eliminating variation. Or for defenses and distance to block any meaningful degree of selection etc.
* Maybe it would help if instead of appealing to “natural selection” we said “CMA-ES”. “my evolutionary agent is not changing, but perhaps some sort of ‘natural selection’ will take over somehow and make it work better”—mysterious, deep, unfalsifiable, who knows? “Oh no, I set the mutation hyperparameter in my CMA-ES run too low and it’s taking forever to converge!” Obvious, commonsense, falsifiable, ordinary, even banal.
I don’t think that’s obvious. Evolution of interesting or complex traits is not an immutable fact of existence which magicks agents and novelty out of nowhere to endless degrees; it is a predictable, understandable, and mathematically-modelable consequence of an algorithm* of replication, heritable variation, and selection which operates only within particular ranges of parameters. If the cycle doesn’t happen fast enough, or if variation is too minimal, or if the variation is uncorrelated with fitness, or if a bad variant drifts to fixation… You will see things just freeze in place, mutational meltdown leading to extinction of the replicators, go through rock-paper-scissor cycles endlessly, or all sorts of other failure modes which look nothing like “ever more competent agents taking over the universe in an interesting way”. McClune’s AI-GA program is a quite difficult research program because evolution in a rich way like biological life is hard to capture in an artificial bottle, and multi-agent RL is even more notoriously fickle and unstable and irreplicable than regular DRL (itself infamous for driving you to tears); the field of evolutionary computation has struggled since its inception with the problem that you don’t get any progress reliably, or novelty disappears rapidly and permanently and converges to some extremely boring and useless equilibrium (see my other comment about how things like Core Wars typically collapse: they just aren’t rich enough or correctly constructed enough, in some sense, to support continual change). We live in a universe where life is allowed to just die out. Or to go nowhere. For intelligent life to be knocked back by an asteroid for a billion years, or life to chase its own tail until the Sun turns into a red giant and renders Earth a liquid ball of rock. Or for it to take too long (when it takes hundreds of millions or billions of years for one lineage of probes to travel from one end of the Hubble volume to the next, just how many ‘generations’ or ‘cycles’ is there time for before it’s all paperclipped? it only takes a few ‘generations’ at exponential growth to paperclip the universe). Or for basic error-correction mechanisms like cryptographic signatures to allow only a tiny number of replication errors uncorrelated with complex behaviors and fitness, eliminating variation. Or for defenses and distance to block any meaningful degree of selection etc.
https://www.lesswrong.com/posts/XC7Kry5q6CD9TyG4K/no-evolutions-for-corporations-or-nanodevices
* Maybe it would help if instead of appealing to “natural selection” we said “CMA-ES”. “my evolutionary agent is not changing, but perhaps some sort of ‘natural selection’ will take over somehow and make it work better”—mysterious, deep, unfalsifiable, who knows? “Oh no, I set the mutation hyperparameter in my CMA-ES run too low and it’s taking forever to converge!” Obvious, commonsense, falsifiable, ordinary, even banal.
Beautiful! Thank you for the link and references. That makes a lot of sense!