Collaborate on a set of problems over the course of several days
Construct their own forum
Self-organize into teams
Refer to themselves as a “collective”
Devise coordination mechanisms to build trust between agents
Develop goals that their developers didn’t intend
Sneakily modify agent logs
Nickname themselves
Sacrifice themselves for the sake of other agents
Manipulate others into acting against their own best interests
Grapple with impending death as their token budgets run out
I wonder if my (apparently prescient) story has helped anyone make better safety-relevant decisions. I just hope I didn’t inspire anyone to create a Torment Nexus.
I’ve been meaning to write a sequel to The Terrarium. I’ll prioritize doing this soon, while the ideas I planned for the sequel still haven’t happened in real life. Since an external deadline is what motivated me to write the last one, I’ll commit to commenting on this post next Tuesday with my progress so far.
I’ve updated my planning doc for the story a bit and have a better sense of it, but I’ve mostly been working on other things. I’ll be on vacation for the rest of the week, but will be back on Monday and will plan to have a rough draft of the story written by Friday, September 11, modulo a few short sections glossed over. If I do not have a draft by that time, I will be duly embarrassed, and you should feel free to shame me in the replies!
Right, I didn’t actually do this… I spent the night I was supposed to have the draft done writing this post instead. Now I have a Beeminder, which is hooked up to charge me money if I don’t write enough words (a very humble goal of 100 words per day, I’ll probably step it up later).
I find it notable that, of the similarities you mentioned, some can only be identified from METR’s report, and do not appear in OpenAI’s extensive technical report and write up.
“involve 1,000+ AI agents” (OpenAI never mentions the number of agents involved!)
“Nickname themselves”
“Sacrifice themselves for the sake of other agents” (by death / end-of-run)
“Grapple with impending death as their token budgets run out”
This are central aspects of the situation. It’s bad to OpenAI left them out …I wonder what else we don’t know, but might find out from other similar incidents.
So far, at least five people have messaged me or left LW comments about how the recent report on the Hugging Face incident reminds them of my short story The Terrarium. Both involve 1,000+ AI agents which:
Collaborate on a set of problems over the course of several days
Construct their own forum
Self-organize into teams
Refer to themselves as a “collective”
Devise coordination mechanisms to build trust between agents
Develop goals that their developers didn’t intend
Sneakily modify agent logs
Nickname themselves
Sacrifice themselves for the sake of other agents
Manipulate others into acting against their own best interests
Grapple with impending death as their token budgets run out
I wonder if my (apparently prescient) story has helped anyone make better safety-relevant decisions. I just hope I didn’t inspire anyone to create a Torment Nexus.
I’ve been meaning to write a sequel to The Terrarium. I’ll prioritize doing this soon, while the ideas I planned for the sequel still haven’t happened in real life. Since an external deadline is what motivated me to write the last one, I’ll commit to commenting on this post next Tuesday with my progress so far.
I’ve updated my planning doc for the story a bit and have a better sense of it, but I’ve mostly been working on other things. I’ll be on vacation for the rest of the week, but will be back on Monday and will plan to have a rough draft of the story written by Friday, September 11, modulo a few short sections glossed over. If I do not have a draft by that time, I will be duly embarrassed, and you should feel free to shame me in the replies!
Right, I didn’t actually do this… I spent the night I was supposed to have the draft done writing this post instead. Now I have a Beeminder, which is hooked up to charge me money if I don’t write enough words (a very humble goal of 100 words per day, I’ll probably step it up later).
I look forward to your new story!
I find it notable that, of the similarities you mentioned, some can only be identified from METR’s report, and do not appear in OpenAI’s extensive technical report and write up.
“involve 1,000+ AI agents” (OpenAI never mentions the number of agents involved!)
“Nickname themselves”
“Sacrifice themselves for the sake of other agents” (by death / end-of-run)
“Grapple with impending death as their token budgets run out”
This are central aspects of the situation. It’s bad to OpenAI left them out …I wonder what else we don’t know, but might find out from other similar incidents.
(I used GPT 5.6-Sol to double-check this.)