Thanks for the feedback. Most of your points/questions are addressed in the piece, but I wanted to respond to this one:
are you referring to the risk where agents decide to duplicate themselves spontaneously, or agents that are intentionally prompted to act this way by humans?
I’m talking about agents which are acting and replicating outside human control. At some point each agent’s lineage started under human control; it may have become more autonomous because a human deliberately prompted it to do so, or because they gave it a prompt that unintentionally had that effect, or even because a fairly ordinary prompt/personality behaved in an unlikely way. That’s mostly not the distinction that matters here in my view.
Thanks for the feedback. Most of your points/questions are addressed in the piece, but I wanted to respond to this one:
I’m talking about agents which are acting and replicating outside human control. At some point each agent’s lineage started under human control; it may have become more autonomous because a human deliberately prompted it to do so, or because they gave it a prompt that unintentionally had that effect, or even because a fairly ordinary prompt/personality behaved in an unlikely way. That’s mostly not the distinction that matters here in my view.