Maybe useful: an analogy this post brought to mind for me: Replacing “AI” with “Animals”.
Hypothetical alien civilization, observing Early Earth and commenting on whether it poses a risk.
Doesn’t optimization nature produce non-agentic animals? It mostly does, but those aren’t the ones we’re concerned with. The risk is all concentrated in the agentic animals.
Basically every animal ever is not agentic. I’ve studied animals for my entire career and I haven’t found an agentic animal yet. That doesn’t preclude them showing up in the future. We have reasons to believe that not only are agents possible, but they are likely.
Even if agentic animals showed up, they would be vastly outnumbered by all the other animals. We believe that agency will give the agentic animals such a drastic advantage, that they will seem to take over the world in a very short amount of time.
(Etc etc)
(Possible that this is in one of the things you cite, and either I missed it or I am failing to remember it)
Maybe useful: an analogy this post brought to mind for me: Replacing “AI” with “Animals”.
Hypothetical alien civilization, observing Early Earth and commenting on whether it poses a risk.
Doesn’t
optimizationnature produce non-agentic animals? It mostly does, but those aren’t the ones we’re concerned with. The risk is all concentrated in the agentic animals.Basically every animal ever is not agentic. I’ve studied animals for my entire career and I haven’t found an agentic animal yet. That doesn’t preclude them showing up in the future. We have reasons to believe that not only are agents possible, but they are likely.
Even if agentic animals showed up, they would be vastly outnumbered by all the other animals. We believe that agency will give the agentic animals such a drastic advantage, that they will seem to take over the world in a very short amount of time.
(Etc etc)
(Possible that this is in one of the things you cite, and either I missed it or I am failing to remember it)