The obvious next step is a Cambrian explosion of emergent behaviors. Coordination among groups of agents, competition among others, shared infrastructure, roles, rules, and many more that I cannot imagine.
Evolution is hard to predict, but I can imagine such an environment would create intelligent and capable meta-organisms, like fully automated businesses, that outcompete human equivalents.
There are so, so many versions of an AI future where we wind up on the wrong end of Darwin.
If the agents can mutate/learn, and if more successful agents replicate, then there will be natural selection for the most effective replicators. This would undermine any kind of alignment in much the same way that cancer eventually escapes the body’s defenses.
You don’t want to be on the wrong end of Darwin, especially not in the long run.
The obvious next step is a Cambrian explosion of emergent behaviors. Coordination among groups of agents, competition among others, shared infrastructure, roles, rules, and many more that I cannot imagine.
Evolution is hard to predict, but I can imagine such an environment would create intelligent and capable meta-organisms, like fully automated businesses, that outcompete human equivalents.
Needless to say this may not end well for us.
There are so, so many versions of an AI future where we wind up on the wrong end of Darwin.
If the agents can mutate/learn, and if more successful agents replicate, then there will be natural selection for the most effective replicators. This would undermine any kind of alignment in much the same way that cancer eventually escapes the body’s defenses.
You don’t want to be on the wrong end of Darwin, especially not in the long run.