The error in the quoted text is using “the Internet” instead of “the Web”.
A web user is supposed to be able to read web resources without necessarily being able to modify those web resources. Doing this does involve transmitting packets (to make the request), and typically also involves side-effects such as the creation of log entries by the servers hosting those web resources. So “read-only” web access still involves “writing to the Internet” (sending packets) even if you’re not “writing to the Web” (modifying web resources).
Normally you can’t use reading web resources as a way of sending messages to other web users. But of course there are special cases where you can. A trivial example would be a server that publishes its own access log, so every “read-only” access to a resource causes a modification to a different resource (the log).
The HTTP standard distinguishes actions that are meant to read a web resource (GET actions) from actions that are meant to update or modify a resource (POST, PUT, DELETE, etc.) The wiki software in question, UseMod Wiki, apparently allows resources to be modified using GET actions too.
But the environment of these agents seems to have treated “GETs don’t modify resources” as a load-bearing security assumption — not just for the protection of those resources, but for the prevention of collusion among its agents.
Oops. That assumption was a mistake, and not a new one either.
The other day, I was debating with somebody whether sci-fi can be useful for preparing for the future.
I brought up a plot point in Crystal Society by @Max Harms, where an AI that was restricted to only use GET requests used a clever many-step workaround to get its own email inbox. Pretty often, I start to consider an AI control scheme that lets an agent read from the Internet but prevents writing to it, and I immediately remember “that didn’t work in Crystal Society!” and dismiss it.
If more people at OpenAI had read Crystal Society, maybe they would have prevented this incident! (On the other hand, I think that would actually be bad. Awkward for my pro-fiction thesis.)
I wonder whether any humans at OpenAI knew that the system relied on an assumption that GETs are safe. I worry that these agents’ environment itself could be created by AI — effectively, created by a programmer who is both ① inclined to place too much faith in specifications, and ② too young to remember the heyday of wiki spam, phpBB exploits, and “I send you this file to have your advice”. This is stuff I’d expect a senior security engineer to be able to catch in a design review, but that’s making a lot of charitable assumptions about the organization.
This is stuff I’d expect a 14 year old to be able to catch in a design review. What kind of engineer thinks that GET requests are a read only sandbox? Has anybody at OpenAI ever looked at a web application before?
The models were running in an agentic sandbox with terminal access (and the ability to edit files within their environment)
We know that the agents had access to /etc/hosts and the ability to edit this (used this to avoid the POST request restriction)
We see that the agents can call curl and run setsid.
As an ordinary SWE, once I knew the AI can hack, I would know GET is too powerful. I believe that knowledge would be available to SWEs. I won’t speculate from there on who knew what, when, etc.
It’s not just the web. Anything beyond passive packet-sniffing requires sending something out to get something back. Ftp servers, IRC, nntp, everything down to the humble ping.
Even packet-sniffing depends on the internet interface being there, and setting that up requires some writing to the internet, or no other machine will know that your exists. For current AIs, that infrastructure for the data centre would have been built the traditional human way before any AIs were trained on it. For now.
And of course, GET and POST are Suggestively Named Tokens, and the HTTP standard consists of Suggestions for what they should mean. An actual web server will do with them whatever its creator made it do.
The error in the quoted text is using “the Internet” instead of “the Web”.
A web user is supposed to be able to read web resources without necessarily being able to modify those web resources. Doing this does involve transmitting packets (to make the request), and typically also involves side-effects such as the creation of log entries by the servers hosting those web resources. So “read-only” web access still involves “writing to the Internet” (sending packets) even if you’re not “writing to the Web” (modifying web resources).
Normally you can’t use reading web resources as a way of sending messages to other web users. But of course there are special cases where you can. A trivial example would be a server that publishes its own access log, so every “read-only” access to a resource causes a modification to a different resource (the log).
The HTTP standard distinguishes actions that are meant to read a web resource (GET actions) from actions that are meant to update or modify a resource (POST, PUT, DELETE, etc.) The wiki software in question, UseMod Wiki, apparently allows resources to be modified using GET actions too.
But the environment of these agents seems to have treated “GETs don’t modify resources” as a load-bearing security assumption — not just for the protection of those resources, but for the prevention of collusion among its agents.
Oops. That assumption was a mistake, and not a new one either.
The other day, I was debating with somebody whether sci-fi can be useful for preparing for the future.
I brought up a plot point in Crystal Society by @Max Harms, where an AI that was restricted to only use GET requests used a clever many-step workaround to get its own email inbox. Pretty often, I start to consider an AI control scheme that lets an agent read from the Internet but prevents writing to it, and I immediately remember “that didn’t work in Crystal Society!” and dismiss it.
If more people at OpenAI had read Crystal Society, maybe they would have prevented this incident! (On the other hand, I think that would actually be bad. Awkward for my pro-fiction thesis.)
I wonder whether any humans at OpenAI knew that the system relied on an assumption that GETs are safe. I worry that these agents’ environment itself could be created by AI — effectively, created by a programmer who is both ① inclined to place too much faith in specifications, and ② too young to remember the heyday of wiki spam, phpBB exploits, and “I send you this file to have your advice”. This is stuff I’d expect a senior security engineer to be able to catch in a design review, but that’s making a lot of charitable assumptions about the organization.
This is stuff I’d expect a 14 year old to be able to catch in a design review. What kind of engineer thinks that GET requests are a read only sandbox? Has anybody at OpenAI ever looked at a web application before?
From the linked site:
so it’s somewhat worse than even that.
Curtis Yarvin
As an ordinary SWE, once I knew the AI can hack, I would know GET is too powerful. I believe that knowledge would be available to SWEs. I won’t speculate from there on who knew what, when, etc.
https://x.com/raelifin/status/2095964432316993828
It’s not just the web. Anything beyond passive packet-sniffing requires sending something out to get something back. Ftp servers, IRC, nntp, everything down to the humble ping.
Even packet-sniffing depends on the internet interface being there, and setting that up requires some writing to the internet, or no other machine will know that your exists. For current AIs, that infrastructure for the data centre would have been built the traditional human way before any AIs were trained on it. For now.
And of course, GET and POST are Suggestively Named Tokens, and the HTTP standard consists of Suggestions for what they should mean. An actual web server will do with them whatever its creator made it do.