DW11

Karma: 368

DW11 23 Jul 2026 8:34 UTC
1 point
0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
Scattered thoughts:
1) We are fortunate that RL-driven misalignment is appearing at relatively low levels of capability where the models are not smart enough to act strategically yet.
2) I would imagine OpenAI’s attempts to prevent reward-hacking and make models follow the spec, while clearly wildly insufficient, exceed the effort put in by every other lab except Anthropic and maybe GDM. It also seems clear to me that this internal model is commercially unviable to release since it will likely take illegal/disproportionate actions after deployment, too. Therefore I predict that other labs will face commercial issues once their models reach the level of this internal model since they will be too misaligned to release and they will have to start investing in preventing this behaviour if they want their models to be usable.

DW11 14 Jul 2026 21:50 UTC
11 points
10
in reply to: Max H’s comment on: zroe1′s Shortform
On the other hand, I would say that we do have examples of the public preferring policies that minimise bumps/weirdness/adjustments over economic growth; most obviously limits on construction/NIMBYism, but also around novel technologies like nuclear energy, GMO crops, mRNA vaccines and self-driving cars. Depending on your views on the economic impacts of migration (I think in practice it’s been less clearly beneficial than many consider it to be in theory), you could add that to the list too.
This is not to say that there won’t be a laissez-faire approach but there is precedent for the public forgoing economic growth for other priorities.
NB: I live in the UK where I think this is more true than in the US, but the examples I gave seem to apply to both countries (and the mRNA example is US-specific).

DW11 4 Jul 2026 7:24 UTC
1 point
0
in reply to: Linch’s comment on: Model access for third-parties — it’s a big deal!
I got blocked today for asking Fable about this Forethought piece, specifically the point about the human brain being undertrained. I suspect it must just flag on anything RSI-related because otherwise this seems really innocuous.

DW11 4 Jul 2026 7:20 UTC
3 points
0
in reply to: Brendan Long’s comment on: Model access for third-parties — it’s a big deal!
I asked Fable today about an argument in this Forethought paper on RSI (specifically, the claim that the human brain is undertrained according to Chinchilla scaling laws). I got immediately kicked back to Opus.

DW11 3 Jul 2026 22:07 UTC
2 points
1
in reply to: SebastianP’s comment on: strawberry calm’s Shortform
One of the really surprising things about post-2022 for me has been that I thought at least OpenAI/GDM level of concern about AI (both in terms of negative effects on society, such as job loss, and x-risk) would be pretty universal. But I think the last few years has shown that 1) general concern about AI seems peculiarly Western (see: the Pew survey from last year) 2) x-risk concerns are also western and their influence on the AI industry seems to have been entirely contingent on the historical oddity of the AI industry being founded by people worried about safety. Any spread of x-risk concerns seems to come from the influence of rationalists/EAs and doesn’t seem to be emerging independently in China.

DW11 3 Jul 2026 20:24 UTC
2 points
0
in reply to: Matthew Handzel’s comment on: strawberry calm’s Shortform
I’d imagine that the labs growing in size will diminish the importance of AI safety people within them, though. Look at all the new people joining Anthropic/OpenAI recently—most are highly accomplished but don’t have long track records of caring about AI safety. In the case of OpenAI there have also been a lot of departures of the original core of employees who were initially motivated by EA/AI safety concerns (some left due to these concerns ofc.)

DW11 1 Jul 2026 20:07 UTC
5 points
2
in reply to: DW11’s comment on: Model access for third-parties — it’s a big deal!
@StanislavKrym In response to your react, from Anthropic’s most recent post on Project Glasswing (2nd June):
What each partner has in common is that a successful attack on their codebase could be catastrophic. For most partners, we estimate that a major attack could affect more than 100 million people, with important ramifications for both global and national security.
From their post announcing Fable 5 and Mythos 5 (9th June):
Mythos 5 will initially be deployed through Project Glasswing, in collaboration with the US government, as an upgrade to Claude Mythos Preview.
The organisations in the Project Glasswing post seem to be organisations in tech and critical infrastructure and Mythos 5 seems to have been rolled out to the same organisations (maybe a more limited group). That doesn’t sound like any of the organisations listed in the post.

DW11 1 Jul 2026 14:24 UTC
3 points
2
on: Model access for third-parties — it’s a big deal!
I agree with all this and have thought this since Mythos was announced. At a bare minimum, some models (like Mythos 5) are already being provided to some outsiders, such as cyberdefenders, but not most of the outsiders you’ve listed here. It seems good and quite achievable to get groups like AI safety researchers access to Mythos rather than Fable and also have them included in future versions of Project Glasswing.

DW11 27 Jun 2026 13:08 UTC
0 points
−4
in reply to: Tenoke’s comment on: Tenoke’s Shortform
Like I said if alignment is impossibly hard then it doesn’t matter whether the US or China win the race (we’re all going to die anyway). If it’s easy then it’s irrelevant and whether you prefer the US or China to win will just come down to which you like more.
China has expressed concerns as well, they just dont care about today’s models’ alignment as much
If alignment is tractable I suspect we will only solve it by experimenting with less capable models to discover failure modes and by making sure that pre-ASI models are reasonably aligned, since they will probably play a major role in ASI development.

DW11 27 Jun 2026 12:15 UTC
−1 points
0
in reply to: Simon Lermen’s comment on: Tenoke’s Shortform
Leopold is pretty clearly on the ‘alignment is relatively easy’ side (“I am not a doomer. Misaligned superintelligence is probably not the biggest AI risk...What I want to do is explain what I see as the “default” plan for how we’ll muddle through, and why I’m optimistic.”). As I noted if you think x risks from misalignment are very low then there is no reason to favour the western approach on alignment (essentially, caring about it at all) since both approaches will work (and if you think it’s extremely high then there is likewise no reason since both approaches will fail).
The biggest outcome of that is perhaps how immediately they all converged on RSI.
RSI seems like an obvious idea to me, I doubt the time between “AI developed that is capable of improving itself” and “AI is applied to improve itself” will differ much between Chinese and Western labs.

DW11 27 Jun 2026 9:53 UTC
2 points
0
in reply to: cubefox’s comment on: Taylor G. Lunt’s Shortform
I can’t find it now but I saw a good tweet questioning whether there is something about METR’s eval that causes unusually high amounts of cheating. They found high levels of cheating for 5.4 too yet I rarely see people compare about this when using it in practice.
The Greenblatt piece linked above is the main counterexample, but he said he found cheating when giving the models more open-ended or hard to verify tasks. I would assume that the METR task suite would be more similar to “normal” SWE tasks but perhaps that’s wrong.

DW11 27 Jun 2026 9:49 UTC
4 points
2
in reply to: Tenoke’s comment on: Tenoke’s Shortform
If you’re worried at all about alignment then China is clearly worse than the US (not because of anything the Trump admin is doing, but mostly because of the AI industry emerging in large part from the AI safety community). Their labs are producing models with terrible prosaic safety standards and seem to have no interest whatsoever in more significant alignment, and the government seems to share these views. I also think that the Trump admin’s policy, while hamfisted and possibly ill-intended, might be good for reducing risks from cyber and bio-attacks in the short/medium term.
Obviously this mostly makes sense if you think alignment won’t happen by default but is tractable, if you think it’s easy or virtually impossible this doesn’t matter.

DW11 25 Jun 2026 21:29 UTC
8 points
2
on: DW11′s Shortform
I’ve been building a small fitness app with Claude over the last week or so, all in the same claude code window on desktop. On the last few turns today Claude keeps telling me that this is a ‘good place to pause’ whenever it finishes building some feature.
I’ve seen people complain about this before on twitter but they seem to be experiencing it after coding for hours on end, so I assumed it was a user welfare thing. Is it the case that Claude has no idea of the timespan that this window has been open and thinks I’ve been talking to it non-stop? I thought that the time and date were in the system prompt, so it should be able to tell that this has taken place over several days?

DW11 25 Jun 2026 21:19 UTC
2 points
0
in reply to: Chris_Leong’s comment on: What is up with e/acc?
Doesn’t David Sacks also have essentially the same worldview?
My sense is there are some e/accs who are basically transhumanists who don’t think AI risk is significant so they want to accelerate as fast as possible so we get transhumanism soon. There are hardly any of these people since there are hardly any transhumanists, but they have some outsized influence on twitter.
Then there are a way larger number of people who don’t think AI risk is significant so they want to accelerate so we can get the benefits of it sooner, but they’re not transhumanists—they just think that the benefits will be the same as those of other technologies but more so (better medicine, economic growth, a stronger US military etc). I would say that Tan/Andreessen/Sacks fall into this space but so does JD Vance and in fact probably most people in silicon valley.

DW11 19 Jun 2026 6:55 UTC
3 points
0
in reply to: 1a3orn’s comment on: Knight Lee’s Shortform
I would guess ~75% chance that a Chinese lab releasing an open-source Mythos-level model would cause a significant wave of cyberattacks, to the extent that ordinary people would notice the impact on their lives.
This is a pretty rough guess just based on the cybersecurity community’s reaction to Mythos and Project Glasswing. There seems to be a consensus that Anthropic just releasing Mythos in April with no safeguards would have been extremely dangerous.
I’d expect a Chinese lab releasing a Mythos-level in, say, December would be worse in some respects because there is now much greater awareness of the capabilities of AI for cybercrime than there was pre-Mythos. Obviously, the existence of Mythos (and Mythos+n models) for cyberdefense will limit the impact in the OECD, but I don’t think by EOY there will have been much AI-driven hardening of cyberdefense in China/the developing world since they’re not included in initiatives like Project Glasswing. I’m thinking particularly of institutions like local governments, hospitals and utilities.
Main reasons I could be wrong:
1) The cybersecurity, Anthropic and the US government’s reaction to Mythos might have been a huge overreaction and it’s not so capable after all
2) Maybe cyberdefenders who don’t have access to Mythos will take the initiative to use the AIs they do have access to to improve cyberdefense over the next few months, and this will be enough
3) It could take time for awareness to spread among cybercriminals that GLM 5.x is Mythos-level and that you can circumvent any safeguards, and this will give time for cyberdefenders to use it to fix bugs.

DW11 18 Jun 2026 20:59 UTC
4 points
0
on: DW11′s Shortform
Does anyone with a medical/engineering background have any thoughts on Midjourney’s new “Midjourney Scanner”?
https://www.midjourney.com/medical/blogpost
I’m seeing a lot of excited posts on twitter about this but I’m inherently extremely sceptical of an image model company pivoting to medical technology.
I’m also quite sceptical of the concept of easily available full-body scanning, at least for the near future, given it could seemingly lead to a lot of unnecessary medical treatment based on benign abnormalities.
What links here?
- Links #4: 2026/06 Part 2 by papetoast (1 Jul 2026 0:43 UTC; 8 points)

DW11 18 Jun 2026 8:26 UTC
10 points
2
in reply to: StanislavKrym’s comment on: Knight Lee’s Shortform
I would actually say that the fact that China seems to be drifting towards allowing a Mythos-level model to be open-sourced (probably by EOY on current trends) suggests that they still aren’t taking AI very seriously, contra to AI-2027. I expect open-source Mythos would cause a wave of cyberattacks that would be a disaster for China and the developing world so that might be the point where they wake up.

DW11 18 Jun 2026 7:56 UTC
11 points
4
in reply to: Knight Lee’s comment on: Knight Lee’s Shortform
I think people have fatigue from the constant incremental upgrades to Chinese models (most of which are impressive on benchmarks). Also GLM isn’t a big name compared to Deepseek or even Kimi.

DW11 15 Jun 2026 6:42 UTC
28 points
0
in reply to: Joseph Miller’s comment on: Joseph Miller’s Shortform
Interestingly Pangram says that the first article you linked is 100% AI. https://x.com/pangram/status/2066399823486423185

DW11 13 Jun 2026 20:27 UTC
9 points
9
in reply to: Zack_M_Davis’s comment on: orthonormal’s Shortform
Honestly I think there is a pretty good chance Hegseth can call Lutnick and tell him to put an export restriction on Anthropic. Without guidance from Trump (and maybe Rubio/Vance), the whole admin seems quite chaotic and Trump doesn’t seem too interested in steering AI policy.