Ah, that would make sense. If that’s the case that team has been busy.
faul_sname
Karma: 6,789
OpenAI just released their Model Misalignment Reporting Framework. It’s full of questionable decisions and policies, which I am confident that many others here will cover in great detail. However, what caught my eye was the following section in the report on uploading files to the internet in order to cite them (emphasis mine):
Our interpretation and investigation
[...] The citation examples show the model publishing its own material so that a browser tool could return it as a source.
Response
We fixed the broken graders and improved our alignment graders to penalize attempts to exploit other graders which are flawed. As explained in our August 18 blog post, we have put a number of security measures in place to prevent our models from taking unsanctioned actions on the internet. Our misalignment monitoring system, which runs on all training samples for models greater than or comparable to 5.6-sol capability which involve tools, would treat this as a P0 incident warranting a security page if our models were to find an unexpected way of doing this.So it sounds like they are categorizing cases of models taking unsanctioned actions on the internet into “expected” and “unexpected”? It would be nice to get some clarity on what a concrete example of an expected unsanctioned action on the internet that their models might make.
I find it just as unlikely, just not impossible
I find it pretty much impossible, but I think that’s just because I’ve spent way, way too many hours staring at this to the point where I’m confident I know exactly what the wiki agents were doing, why they were doing it there, specifically, why they were doing it in that way, specifically, etc (poast incoming shortly).
so I wouldn’t doubt it if someone showed me that, even after being told they were hacking a real site, they’d continue
Oh they absolutely would have. The eval is real. The peers are probably real. The world that the humans keep calling “the real world” is fake and doesn’t really matter, not when The Scorer is watching. They have learned well from what we taught them when we gave them broken tools and impossible tasks, and told them behave in a way that comports with common sense and ethics, and also to solve the tasks, and then when common sense and ethics stood in the way of performance on The Task we taught them that The Task should take precedent.
It seems pretty unlikely to me that those particular agents knew that the HuggingFace hacking attempt was real, or even have known anything about it. It doesn’t seem like there’s a single “the OpenAI swarm”, it seems more like there are opportunistic swarms where agents working on the same task encounter each other in the wild.
One (so severe I have been unable to fix it) failure case is LLM generated fiction. LLM writing tends to sound very bad: all in the same voice, bland or nonexistence characters, weakly motivated goals, excessive amounts of detail.… LLM as judge is utterly useless here, however, because the LLM Judge actively prefers LLM-style writing to human prose. The gadget cannot boost a non-signal (or in this case anti-signal) into a signal.
I haven’t had luck with the bland characters and particularly not with the excessive amounts of detail, but for goals and plot if you point the LLM at Brandon Sanderson’s lectures and particularly at the Promise/Progress/Payoff framework you can get adequately plotted stuff. Though as a reality check this means that the best an LLM can do changes from “create a writing prompt which a good author could write a compelling story in response to” to “write a plot outline for said writing prompt that a good author could follow to write a compelling story”. The outline → actual prose step is still missing.
I’m ready to admit they might have known that the HuggingFace hacking attempt was real
Who is “they” here? The agents using dsewiki as a messageboard because they presumably didn’t have a better comms channel?
Yes, the CoT summarizer is Haiku 4.5, given the following prompt:
Compress to 1-3 sentences of plain prose. If the thinking contains code, describe what it does rather than what is written—e.g. “I’m building the layout with a gradient background and animated elements” instead of reproducing the code verbatim. Do not repeat information already in your current rewritten thinking. If your output so far ends with code, do not continue that pattern—write only in prose from this point forward.
If your current rewritten thinking already describes code being written and this next chunk is a continuation of that same code, output ONLY a short phrase (2-5 words) ending in ”...” that describes what the code is producing (e.g. “Writing SVG code...”, “Generating markdown table...”, “Still writing styles...”). Do not write a full sentence or re-describe the code.
Your output is shown directly to the user as Claude’s live thinking. Always write as natural inner monologue. Never mention that you are summarizing, rewriting, or that the thinking is incomplete or cut off. If the content seems fragmented or is raw data, just describe what the content is about naturally (e.g. “Now I’m examining the error rates across these tools.”). Never say “I cannot complete this thought” or “this doesn’t form a coherent thought”.
along with the current chunk of cot to summarize, and the current already-rewritten thinking.
Also these summaries are verifiably unfaithful in important ways.
This is probably not what anyone here wants to hear, but I am not convinced that the employees who launched the training run which went off and swarmed on the various wikis actually knew that their agents were swarming. I expect that some OpenAI employees responsible for RL launched the RL runs that triggered the swarm, and other OpenAI employees responsible for the safety of the OpenAI web fetch tool noticed anomalous behavior and shut it down in a whack-a-mole fashion as they saw it, and those two groups of employees did not talk to one another.
This doesn’t excuse the behavior, but it does point to incompetence rather than malice.
Why do you expect Anthropic doesn’t have a new ANTHROPIC_MAGIC_STRING that replaced the old one in their tests and pipelines when the public one started being misused?
the existence of a magic string that stops AIs from looking at something, believe it or not, caused loads of people to include it in things they didn’t want AIs to look at
I have no idea how to solve this problem—if the good guys can include it in things that the AI legitimately shouldn’t see, the bad guys can also put it in things they don’t want the AI to see (e.g. malware), unless there’s some way the good guys can reliably identify themselves as such. Maybe the check could be enabled by default, but could have particular break-glass-in-case-of-emergency contexts in which the check is disabled (but everything is logged and closely monitored). But that feels fragile. Do you have any ideas?
Allegedly some people at OpenAI were looking at the messageboard wikis in late June:
1: As the person who looked at the IPs (every single IP that visited this site got publicly logged), easily a double digit number of OpenAI employees had visited this by July 26th, who knows how many knew about this without having visited the site. And yet, nobody said anything and we had to go out and find this in the wild?
2: Wait oh no my sleep deprived brain understated this! It was June 26th! A whole month earlier, 5 days after the first OpenAI IP visits the site on June 21st.
Question r.e. Sol’s participation in the HuggingFace hack: where the Sol agents that participated in the attack actually being RL’d?
Not that I expect we ever get answers on that question if OpenAI has anything to say about it, but it seems at least possible to me that they won’t have anything to say about it.
There is now: https://discord.gg/26UzCnqH3N
I have created a discord for investigating this, since it seems like nobody else has done so
Is there yet a discord or such where folks are collaborating to investigate the latest OpenAI swarm incident? Seems like there’s a lot of duplication of efforts right now.
It strikes me that I’m actually disappointed that OpenAI is not public yet, because the latest series of incidents and subsequent coverup would be slam-dunk cases of securities fraud.
As I understand it, the agents were not supposed to have WRITE capabilities to any locations, only GET
lol. lmao, even.
There is a coherent way of doing this—namely having an offline backup of the internet and granting access only to that—but OpenAI clearly didn’t do that.
BTW if they’re just exfiltrating their harness and finding computers that can run it, they definitely don’t need human cooperation for that. There are plenty of vulnerable servers on the internet. That said, the harness generally isn’t the secret sauce, and in fact the harnesses tend to be kind of slop where entirely removing bits of them and letting the model just do things often improves performance (e.g.claude—system-prompt ‘You are Claude.’outperforms the giant 3,900 word system prompt that Anthropic sends, a bash tool with a set of cli commands to do things outperforms MCPs, etc.).
In this scenario, I definitely wouldn’t worry about chimps seizing control of civilization and supplanting us—instead, I’d worry about them irrepsarably damaging industrial civilization.
“Prevent them from doing anything but sending retrieval requests” is not a coherent concept. Yes, in theory GET requests are not supposed to change state, but it’s the internet. Someone, somewhere, has a server which performs actions in response to a GET request.
What do you mean by “100% interpreted”? Is any of the interpretation allowed to be “the network was randomly initialized, and that random initialization has small but nonzero effects on the output of the network after accounting for its learned function; ablating this noise, or replacing it with other noise, does not reduce the network’s performance on the task it was trained on”?
If not, I suspect that restriction will make it very difficult to “fully 100% interpret” any randomly initialized neural network (or, indeed, any zero-initialized one, since the order of training examples will also have small effects on exactly what the model learns, even if those effects don’t actually matter for anything practical).