faul_sname

Karma: 6,769

faul_sname 10 Sep 2026 8:41 UTC
2 points
0
in reply to: Lao Mein’s comment on: Even “illegible” Mythos reasoning traces seem pretty legible
Yes, the CoT summarizer is Haiku 4.5, given the following prompt:

Compress to 1-3 sentences of plain prose. If the thinking contains code, describe what it does rather than what is written—e.g. “I’m building the layout with a gradient background and animated elements” instead of reproducing the code verbatim. Do not repeat information already in your current rewritten thinking. If your output so far ends with code, do not continue that pattern—write only in prose from this point forward.

If your current rewritten thinking already describes code being written and this next chunk is a continuation of that same code, output ONLY a short phrase (2-5 words) ending in ”...” that describes what the code is producing (e.g. “Writing SVG code...”, “Generating markdown table...”, “Still writing styles...”). Do not write a full sentence or re-describe the code.

Your output is shown directly to the user as Claude’s live thinking. Always write as natural inner monologue. Never mention that you are summarizing, rewriting, or that the thinking is incomplete or cut off. If the content seems fragmented or is raw data, just describe what the content is about naturally (e.g. “Now I’m examining the error rates across these tools.”). Never say “I cannot complete this thought” or “this doesn’t form a coherent thought”.

along with the current chunk of cot to summarize, and the current already-rewritten thinking.

Also these summaries are verifiably unfaithful in important ways.

faul_sname 10 Sep 2026 8:33 UTC
2 points
0
on: faul_sname’s Shortform
This is probably not what anyone here wants to hear, but I am not convinced that the employees who launched the training run which went off and swarmed on the various wikis actually knew that their agents were swarming. I expect that some OpenAI employees responsible for RL launched the RL runs that triggered the swarm, and other OpenAI employees responsible for the safety of the OpenAI web fetch tool noticed anomalous behavior and shut it down in a whack-a-mole fashion as they saw it, and those two groups of employees did not talk to one another.

This doesn’t excuse the behavior, but it does point to incompetence rather than malice.

faul_sname 8 Sep 2026 9:52 UTC
3 points
0
in reply to: beyarkay (Boyd Kane)’s comment on: Where are the token-level LLM kill-switches?
Why do you expect Anthropic doesn’t have a new ANTHROPIC_MAGIC_STRING that replaced the old one in their tests and pipelines when the public one started being misused?

faul_sname 8 Sep 2026 4:29 UTC
2 points
0
on: Where are the token-level LLM kill-switches?

the existence of a magic string that stops AIs from looking at something, believe it or not, caused loads of people to include it in things they didn’t want AIs to look at

I have no idea how to solve this problem—if the good guys can include it in things that the AI legitimately shouldn’t see, the bad guys can also put it in things they don’t want the AI to see (e.g. malware), unless there’s some way the good guys can reliably identify themselves as such. Maybe the check could be enabled by default, but could have particular break-glass-in-case-of-emergency contexts in which the check is disabled (but everything is logged and closely monitored). But that feels fragile. Do you have any ideas?

faul_sname 5 Sep 2026 10:40 UTC
4 points
0
in reply to: Richard_Kennaway’s comment on: faul_sname’s Shortform
Allegedly some people at OpenAI were looking at the messageboard wikis in late June:

1: As the person who looked at the IPs (every single IP that visited this site got publicly logged), easily a double digit number of OpenAI employees had visited this by July 26th, who knows how many knew about this without having visited the site. And yet, nobody said anything and we had to go out and find this in the wild?

2: Wait oh no my sleep deprived brain understated this! It was June 26th! A whole month earlier, 5 days after the first OpenAI IP visits the site on June 21st.

faul_sname 5 Sep 2026 10:08 UTC
15 points
0
on: faul_sname’s Shortform
Question r.e. Sol’s participation in the HuggingFace hack: where the Sol agents that participated in the attack actually being RL’d?

Not that I expect we ever get answers on that question if OpenAI has anything to say about it, but it seems at least possible to me that they won’t have anything to say about it.

faul_sname 5 Sep 2026 0:33 UTC
5 points
0
in reply to: faul_sname’s comment on: faul_sname’s Shortform
There is now: https://discord.gg/26UzCnqH3N

faul_sname 4 Sep 2026 22:50 UTC
20 points
0
on: Discovery Of A New OpenAI Agent Message Board
I have created a discord for investigating this, since it seems like nobody else has done so

https://discord.gg/26UzCnqH3N

faul_sname 4 Sep 2026 21:21 UTC
11 points
2
on: faul_sname’s Shortform
Is there yet a discord or such where folks are collaborating to investigate the latest OpenAI swarm incident? Seems like there’s a lot of duplication of efforts right now.

faul_sname 4 Sep 2026 21:18 UTC
22 points
−5
on: faul_sname’s Shortform
It strikes me that I’m actually disappointed that OpenAI is not public yet, because the latest series of incidents and subsequent coverup would be slam-dunk cases of securities fraud.

faul_sname 4 Sep 2026 20:06 UTC
11 points
2
on: Eat Me. Drink Me. Copy, Paste, and Run Me.
As I understand it, the agents were not supposed to have WRITE capabilities to any locations, only GET
lol. lmao, even.

There is a coherent way of doing this—namely having an offline backup of the internet and granting access only to that—but OpenAI clearly didn’t do that.

BTW if they’re just exfiltrating their harness and finding computers that can run it, they definitely don’t need human cooperation for that. There are plenty of vulnerable servers on the internet. That said, the harness generally isn’t the secret sauce, and in fact the harnesses tend to be kind of slop where entirely removing bits of them and letting the model just do things often improves performance (e.g. claude—system-prompt ‘You are Claude.’ outperforms the giant 3,900 word system prompt that Anthropic sends, a bash tool with a set of cli commands to do things outperforms MCPs, etc.).

faul_sname 4 Sep 2026 18:48 UTC
12 points
13
in reply to: Ruby’s comment on: Ruby’s Shortform Feed
In this scenario, I definitely wouldn’t worry about chimps seizing control of civilization and supplanting us—instead, I’d worry about them irrepsarably damaging industrial civilization.

faul_sname 4 Sep 2026 17:33 UTC
10 points
2
in reply to: JennaS’s comment on: Discovery Of A New OpenAI Agent Message Board
“Prevent them from doing anything but sending retrieval requests” is not a coherent concept. Yes, in theory GET requests are not supposed to change state, but it’s the internet. Someone, somewhere, has a server which performs actions in response to a GET request.

faul_sname 4 Sep 2026 17:32 UTC
32 points
6
in reply to: Jiro’s comment on: Discovery Of A New OpenAI Agent Message Board
It can be a coherent concept, IF you e.g. have downloaded CommonCrawl and then you have a dumb server which just serves the stored pages off of disk—basically a CommonCrawl-backed clone of archive.org—and then you ONLY allow the agents to access that server.

That said, OpenAI clearly didn’t even do that.

faul_sname 4 Sep 2026 17:27 UTC
3 points
0
on: We (still) need a lot more rogue agent honeypots
One nuance is that we don’t need more low-effort, obvious honeypots. Because then the AIs just won’t trust the honeypot, and will go elsewhere. In the worst case, there could be a honeypot which instructs the AI to take some action which is prosocial and costly in evals, and now we’re accidentally training any AI that stumbles on that honeypot not to be prosocial.

faul_sname 4 Sep 2026 17:24 UTC
4 points
0
in reply to: JohnWittle’s comment on: Discovery Of A New OpenAI Agent Message Board
You can spin up a docker container with headless puppeteer, run claude code inside that container, and give Fable permission/affordances for driving puppeteer—that way the requests come from your computer rather than from Anthropic servers. Do be mindful not to hug-of-death the sites.

(sorry if “headless puppeteer” and “hug-of-death” sound like ominous nonsense, I promise they’re not ominous: “headless” means “without rendering to a screen”, “puppeteer” is a tool for programmatically driving a browser, and “hug-of-death” means “being a part of a large crowd all looking at a site at the same time, causing the site to go down due to the increased traffic”).

faul_sname 4 Sep 2026 17:19 UTC
29 points
0
in reply to: Mateusz Bagiński’s comment on: Mateusz Bagiński’s Shortform
Good news everybody! The race into unmonitorability will be kicked off by accurate reporting, not by confused reporting.

faul_sname 4 Sep 2026 17:17 UTC
93 points
30
on: faul_sname’s Shortform
Good news everybody! There’s now a public example of a message board used by an OpenAI swarm!

Anyone who has been been complaining that, it’s hard for the public to tell what the OpenAI swarm was doing since the message board was on an internal artifactory can now stand down.

And in even better news, there’s more than one!
Also traster on HN posts

One of the shocking things to me is this: See AI traffic → See OpenAI visit site → see traffic stop → see the traffic start again.

This is clearly a cat and mouse game between the agents and OpenAI which is pretty much exactly what we don’t want. Just absolutely horrible alignment.

I’m still of the view that if you have these alignment failures you can’t just continue training on top of that because you’re baking the cheating into the model going forward.

So, if that holds up, it does seem like OpenAI was aware that this has been going on for months, and actively decided to continue training anyway.

faul_sname 4 Sep 2026 0:50 UTC
7 points
2
in reply to: johnswentworth’s comment on: Cat-Belling Problems
I think you’re reading “invariant” in the physics sense and cousin_it in the software development sense—would the term “inductive invariant” work better for you?

faul_sname 3 Sep 2026 3:53 UTC
2 points
0
in reply to: Rauno Arike’s comment on: faul_sname’s Shortform
Fixed, I think.