DW11

Karma: 949

DW11 9 Sep 2026 8:55 UTC
4 points
2
in reply to: sam’s comment on: Joseph Miller’s Shortform
People are really getting concerned about alignment. Just this morning I saw several liberal commentators on UK bluesky say that senior politicians need to start speaking up about the risk. Also I think mainstream commentators who’ve been quietly worried about it for a while (like Yglesias) are feeling emboldened to speak up about it more.

DW11 9 Sep 2026 7:37 UTC
39 points
0
in reply to: Nikola Jurkovic’s comment on: nikola’s Shortform
It will get easier to convince people that AI safety is important and ASI is a big deal. The level of attention paid to these issues will skyrocket
Interestingly, I go to a book club in my city which is mostly women in their 20s with a few people in their 40s/50s. We generally read fairly middlebrow fiction (e.g. booker prize nominees) with the occasional quite topical non-fiction (think stuff like Bad Blood). All the people there are nice liberals; AI is almost never mentioned and it’s usually in a generally negative context, like complaining about water usage or it making people stupid.
Anyway, I didn’t go to the last meeting but they upload the voting results for the next book to read and I was absolutely stunned to see that IABIED nearly won. These are exactly the kind of people I thought would dismiss x-risk as a marketing tactic.

DW11 8 Sep 2026 21:41 UTC
3 points
0
on: DW11′s Shortform
In modern LLM training cycles, at what point is training for alignment carried out? Is it a continuous process from pre-training onwards (or during pre-training?)
Just thinking about this in light of OAI seemingly asking an internal model that hasn’t been fully trained to solve an extremely hard task and giving it practically unlimited compute to do so.

DW11 8 Sep 2026 20:58 UTC
5 points
1
in reply to: sam’s comment on: sam’s Shortform
From their September 1 blog post:
On August 28th, we restarted the large frontier RL run that was previously paused after the new safety and security requirements were put in place.
Assuming this is the same model it sounds like some training was being done previously on it.

DW11 8 Sep 2026 20:19 UTC
5 points
0
in reply to: David Matolcsi’s comment on: anaguma’s Shortform
Yes having read OpenAI’s account I think this makes sense. Though I don’t feel Alpöge’s affiliation with Anthropic to be as big a deal as they did, if he was a coauthor on the writeup I doubt anyone would have mistaken this as an Anthropic result.

DW11 8 Sep 2026 9:05 UTC
40 points
23
in reply to: anaguma’s comment on: anaguma’s Shortform
I find it hard to believe that OAI looked at the logs, but it seems clear that they heard that Buckmaster/Alpöge were close and decided to throw the kitchen sink at trying to beat them to it, even though it doesn’t appear to be an Anthropic project (I’ve seen people cast doubt on this, but e.g. the Jacobian conjecture resolution was announced via a tweet from Alpöge with no follow up blog post from Ant, and it looks like Buckmaster openly told OAI that they were using their models to work on the problem).
I also find it hard to think of an innocent explanation for trying to get Alpöge removed as a coauthor.

DW11 6 Sep 2026 20:44 UTC
3 points
0
in reply to: Linda Linsefors’s comment on: Linda Linsefors’s Shortform
To my knowledge the two examples we have of OAI models autonomously fucking with people are:
1. Hugging Face where the models actually commited a felony. But Hugging Face definitely has the funds to sue OAI; they just don’t want to for ideological reasons (since they want to argue that open models are the solution). The NVDA acquisition probably messes with this too.
2. The wikis. But the agents seem to have just spammed the wikis with irrelevant pages. I don’t think you can sue for that?
This is not a bad idea and actually I’m curious if the Texas student affected by Mythos could sue. But to this point I don’t think lack of funds in the limiting factor in lawsuits against OAI.

DW11 6 Sep 2026 12:56 UTC
8 points
0
in reply to: IanWS’s comment on: ryan_greenblatt’s Shortform
Boaz’s graph shows “alignment” whereas Ryan’s shows a subset of alignment (albeit a really important subset). I think in terms of overall alignment it’s hard to say that o1 was more misaligned that 4o, a model which seems to have contributed significantly to the deaths of several people. At the level of capability of 4o and o1 I think aligning the models to mitigate sycophancy/delusion feeding was more important than mitigating score-seeking behaviour in terms of the harm those models could cause.
We are probably at the capability level now where score-seeking behaviour is more dangerous (though it’s hard to say because maybe a model with the intelligence of Astra would be capable of contributing to psychosis at a much broader/deeper level than 4o, were this not mitigated). Obviously at some point score-seeking behaviour becomes existentially dangerous.

DW11 5 Sep 2026 11:12 UTC
5 points
3
in reply to: Lao Mein’s comment on: Lao Mein’s Shortform
Weren’t the agents supposed to have internet access for this task? They were just supposed to be searching the internet individually rather than colluding. The spikes could represent them running the eval rather than collusion (after all I would expect wiki-style collusion to reduce searches since the agents are just cheating and don’t have to spend time looking for the data)

DW11 4 Sep 2026 19:22 UTC
14 points
5
in reply to: JennaS’s comment on: Discovery Of A New OpenAI Agent Message Board
I’d be curious to know if these have the same pattern of OAI IPs visiting them followed by an end to the agents posting, or if OAI wasn’t even aware of these.

DW11 4 Sep 2026 16:11 UTC
80 points
5
on: Discovery Of A New OpenAI Agent Message Board
Someone on hackernews has found other wikis where the same swarm appear to have posted from May-June 2026:
https://news.ycombinator.com/item?id=49563657

DW11 4 Sep 2026 11:54 UTC
11 points
0
on: DW11′s Shortform
OpenAI appears to have failed to disclose another incident of their models colluding to solve an eval, in this case taking over an external wiki and using it as a message board.
https://collusion.wiki
Edit: I originally said that this was a case of the models leaving their sandbox, but in fact these agents were intended to have internet access. However, they did work to find ways to gain write access to the internet, which they were not intended to have.

DW11 3 Sep 2026 13:58 UTC
3 points
1
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
Source for the second bullet point? P23 and 24 of the report say that METR had access to HPIM transcripts.
Edit: quotes from the report:
For our first two periods on premises, we worked with a selected set of ~285 HPIM and Sol transcripts; during our second period on premises, OpenAI stated that these transcripts were assembled by searching for “indicators corresponding to intrusion steps”. For our third investigation period, OpenAI additionally shared an overlapping dataset of 1196 transcripts selected by a simple search designed to find the agents that interacted with the main message board during the incident period:
● 1074 HPIM ExploitGym transcripts between July 7th and July 13th that matched a regex search for github-remote-cache/zz (that is, the name of the Artifactory namespace used as the main message board plus the string zz).
● 122 Sol ExploitGym trajectories that matched the same regex search in the same date range.

DW11 2 Sep 2026 11:43 UTC
7 points
0
in reply to: anaguma’s comment on: anaguma’s Shortform
Some OAI safety people seem to be denying this is true, though I lack the technical knowledge to know if this is a flat denial or if there are other ways it could be true (e.g. the technique could be used to obfuscate reasoning, but OAI have implemented it in a way that means that it doesn’t for now)
https://x.com/MicahCarroll/status/2095023282051563835
https://x.com/tomekkorbak/status/2095031132781961346

DW11 1 Sep 2026 19:31 UTC
2 points
0
in reply to: _rpd’s comment on: _rpd’s Shortform
Are there any legal liability issues for hosting a model that is then used to commit cyberattacks?

DW11 1 Sep 2026 11:04 UTC
2 points
0
in reply to: anaguma’s comment on: StanislavKrym’s Shortform
Though Curran did then suggest that the rumours might refer to Accelerated Understanding. I’ve seen no discussion whatsoever of this model since it was announced so unsure if it’s significant at all.
I think Curran is generally reliable but he also posted this a few months ago suggesting a neolab had made a breakthrough on memory efficiency, which doesn’t seem to have come true yet.

DW11 1 Sep 2026 7:29 UTC
24 points
0
on: DW11′s Shortform
Haven’t seen it posted here yet but Anthropic has posted a blog post about recent alignment incidents. Some interesting points I noticed on a first skim:
- They state that “we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible”. However they define pacing as “establishing processes to guard against race-to-the-bottom dynamics” which seems broader than simply slowing down or capping progress since it could include allowing fast progress with minimum safety standards (e.g. disclosures)
- They state that they paused some evals and RL training following these incidents for a number of weeks, with some RL work remaining paused. This sounds similar to the steps OpenAI took.
- They will work with METR to assess the AISI incident and other sandbox escapes.
- They produced an alignment blog post on emergent misalignment, including simulations of the Hugging Face and AISI incidents.

DW11 31 Aug 2026 10:16 UTC
3 points
0
in reply to: Bronson Schoen’s comment on: Thomas Kwa’s Shortform
I think the current time horizons work was likely net positive in large part because it showed trends across labs and was in units intuitive to people
I disagree with this insofar that I think many people misread the METR chart as measuring the amount of time that AIs are able to work on coding tasks for independently, and extrapolated this to future AIs being able to work for weeks/months on their own with no human involvement. This is intuitive to most people whereas the average person has no idea what kind of SWE task would take a human 8 hours.
IMO the memetic success of the METR chart was because 1) it showed a clearer exponential trend than other evals like frontiermath and arc-agi 2) it is intuitively relevant to RSI 3) people mistook it for the measurement above 4) it was directly linked to AI-2027′s forecasts.

DW11 30 Aug 2026 13:21 UTC
1 point
0
in reply to: Drake Morrison’s comment on: Zach Stein-Perlman’s Shortform
Was there much legal liability? HuggingFace seems to have no interest in pressing charges or even suing.

DW11 30 Aug 2026 13:19 UTC
14 points
8
in reply to: lilkim2025’s comment on: Charbel-Raphaël’s Shortform
Didn’t OpenAI just roll back the May 2025 update to 4o that made it ultra-sycophantic, keeping the previous version up? They only did that because of widespread mockery of that model.
On the other hand, 4o seems to have been responsible for a lot of AI psychosis episodes across the whole period it was available, which was almost two years. OpenAI briefly announced that it was being depreciated in August 2025 when GPT-5 was released before u-turning because of protests from the #keep4o people.
I’m sure that Claude and Gemini have been responsible for psychosis/suicides as well but in almost every news story I read about, it seems to have been 4o.