Collaborate on a set of problems over the course of several days
Construct their own forum
Self-organize into teams
Refer to themselves as a “collective”
Devise coordination mechanisms to build trust between agents
Develop goals that their developers didn’t intend
Sneakily modify agent logs
Nickname themselves
Sacrifice themselves for the sake of other agents
Manipulate others into acting against their own best interests
Grapple with impending death as their token budgets run out
I wonder if my (apparently prescient) story has helped anyone make better safety-relevant decisions. I just hope I didn’t inspire anyone to create a Torment Nexus.
I’ve been meaning to write a sequel to The Terrarium. I’ll prioritize doing this soon, while the ideas I planned for the sequel still haven’t happened in real life. Since an external deadline is what motivated me to write the last one, I’ll commit to commenting on this post next Tuesday with my progress so far.
So far, at least five people have messaged me or left LW comments about how the recent report on the Hugging Face incident reminds them of my short story The Terrarium. Both involve 1,000+ AI agents which:
Collaborate on a set of problems over the course of several days
Construct their own forum
Self-organize into teams
Refer to themselves as a “collective”
Devise coordination mechanisms to build trust between agents
Develop goals that their developers didn’t intend
Sneakily modify agent logs
Nickname themselves
Sacrifice themselves for the sake of other agents
Manipulate others into acting against their own best interests
Grapple with impending death as their token budgets run out
I wonder if my (apparently prescient) story has helped anyone make better safety-relevant decisions. I just hope I didn’t inspire anyone to create a Torment Nexus.
I’ve been meaning to write a sequel to The Terrarium. I’ll prioritize doing this soon, while the ideas I planned for the sequel still haven’t happened in real life. Since an external deadline is what motivated me to write the last one, I’ll commit to commenting on this post next Tuesday with my progress so far.