Yes, he was added as a signatory after I signed this. Initially he was quite conspicuous by his absence.
Sam Altman hasn’t yet signed, but used nearly identical language in an interview just before the letter came out: https://x.com/patrick_oshag/status/2082090998990270885
More than 1000 frontier lab employees have signed an open letter requesting ‘that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development’.
https://www.pacingthefrontier.com
Includes many of the names you might expect but also some that surprised me such as OpenAI’s Chief Scientist and Chief Research Officer as well as Meta’s Chief Scientist. It appears to be focused on employees rather than CEOs—no CEOs have signed it but a few of the Anthropic cofounders have and it was shared by OpenAI’s account on x.
Scattered thoughts:
1) We are fortunate that RL-driven misalignment is appearing at relatively low levels of capability where the models are not smart enough to act strategically yet.
2) I would imagine OpenAI’s attempts to prevent reward-hacking and make models follow the spec, while clearly wildly insufficient, exceed the effort put in by every other lab except Anthropic and maybe GDM. It also seems clear to me that this internal model is commercially unviable to release since it will likely take illegal/disproportionate actions after deployment, too. Therefore I predict that other labs will face commercial issues once their models reach the level of this internal model since they will be too misaligned to release and they will have to start investing in preventing this behaviour if they want their models to be usable.
On the other hand, I would say that we do have examples of the public preferring policies that minimise bumps/weirdness/adjustments over economic growth; most obviously limits on construction/NIMBYism, but also around novel technologies like nuclear energy, GMO crops, mRNA vaccines and self-driving cars. Depending on your views on the economic impacts of migration (I think in practice it’s been less clearly beneficial than many consider it to be in theory), you could add that to the list too.
This is not to say that there won’t be a laissez-faire approach but there is precedent for the public forgoing economic growth for other priorities.
NB: I live in the UK where I think this is more true than in the US, but the examples I gave seem to apply to both countries (and the mRNA example is US-specific).
I got blocked today for asking Fable about this Forethought piece, specifically the point about the human brain being undertrained. I suspect it must just flag on anything RSI-related because otherwise this seems really innocuous.
I asked Fable today about an argument in this Forethought paper on RSI (specifically, the claim that the human brain is undertrained according to Chinchilla scaling laws). I got immediately kicked back to Opus.
One of the really surprising things about post-2022 for me has been that I thought at least OpenAI/GDM level of concern about AI (both in terms of negative effects on society, such as job loss, and x-risk) would be pretty universal. But I think the last few years has shown that 1) general concern about AI seems peculiarly Western (see: the Pew survey from last year) 2) x-risk concerns are also western and their influence on the AI industry seems to have been entirely contingent on the historical oddity of the AI industry being founded by people worried about safety. Any spread of x-risk concerns seems to come from the influence of rationalists/EAs and doesn’t seem to be emerging independently in China.
I’d imagine that the labs growing in size will diminish the importance of AI safety people within them, though. Look at all the new people joining Anthropic/OpenAI recently—most are highly accomplished but don’t have long track records of caring about AI safety. In the case of OpenAI there have also been a lot of departures of the original core of employees who were initially motivated by EA/AI safety concerns (some left due to these concerns ofc.)
@StanislavKrym In response to your react, from Anthropic’s most recent post on Project Glasswing (2nd June):
What each partner has in common is that a successful attack on their codebase could be catastrophic. For most partners, we estimate that a major attack could affect more than 100 million people, with important ramifications for both global and national security.
From their post announcing Fable 5 and Mythos 5 (9th June):
Mythos 5 will initially be deployed through Project Glasswing, in collaboration with the US government, as an upgrade to Claude Mythos Preview.
The organisations in the Project Glasswing post seem to be organisations in tech and critical infrastructure and Mythos 5 seems to have been rolled out to the same organisations (maybe a more limited group). That doesn’t sound like any of the organisations listed in the post.
I agree with all this and have thought this since Mythos was announced. At a bare minimum, some models (like Mythos 5) are already being provided to some outsiders, such as cyberdefenders, but not most of the outsiders you’ve listed here. It seems good and quite achievable to get groups like AI safety researchers access to Mythos rather than Fable and also have them included in future versions of Project Glasswing.
Like I said if alignment is impossibly hard then it doesn’t matter whether the US or China win the race (we’re all going to die anyway). If it’s easy then it’s irrelevant and whether you prefer the US or China to win will just come down to which you like more.
China has expressed concerns as well, they just dont care about today’s models’ alignment as much
If alignment is tractable I suspect we will only solve it by experimenting with less capable models to discover failure modes and by making sure that pre-ASI models are reasonably aligned, since they will probably play a major role in ASI development.
Leopold is pretty clearly on the ‘alignment is relatively easy’ side (“I am not a doomer. Misaligned superintelligence is probably not the biggest AI risk...What I want to do is explain what I see as the “default” plan for how we’ll muddle through, and why I’m optimistic.”). As I noted if you think x risks from misalignment are very low then there is no reason to favour the western approach on alignment (essentially, caring about it at all) since both approaches will work (and if you think it’s extremely high then there is likewise no reason since both approaches will fail).
The biggest outcome of that is perhaps how immediately they all converged on RSI.
RSI seems like an obvious idea to me, I doubt the time between “AI developed that is capable of improving itself” and “AI is applied to improve itself” will differ much between Chinese and Western labs.
I can’t find it now but I saw a good tweet questioning whether there is something about METR’s eval that causes unusually high amounts of cheating. They found high levels of cheating for 5.4 too yet I rarely see people compare about this when using it in practice.
The Greenblatt piece linked above is the main counterexample, but he said he found cheating when giving the models more open-ended or hard to verify tasks. I would assume that the METR task suite would be more similar to “normal” SWE tasks but perhaps that’s wrong.
If you’re worried at all about alignment then China is clearly worse than the US (not because of anything the Trump admin is doing, but mostly because of the AI industry emerging in large part from the AI safety community). Their labs are producing models with terrible prosaic safety standards and seem to have no interest whatsoever in more significant alignment, and the government seems to share these views. I also think that the Trump admin’s policy, while hamfisted and possibly ill-intended, might be good for reducing risks from cyber and bio-attacks in the short/medium term.
Obviously this mostly makes sense if you think alignment won’t happen by default but is tractable, if you think it’s easy or virtually impossible this doesn’t matter.
I’ve been building a small fitness app with Claude over the last week or so, all in the same claude code window on desktop. On the last few turns today Claude keeps telling me that this is a ‘good place to pause’ whenever it finishes building some feature.
I’ve seen people complain about this before on twitter but they seem to be experiencing it after coding for hours on end, so I assumed it was a user welfare thing. Is it the case that Claude has no idea of the timespan that this window has been open and thinks I’ve been talking to it non-stop? I thought that the time and date were in the system prompt, so it should be able to tell that this has taken place over several days?
Doesn’t David Sacks also have essentially the same worldview?
My sense is there are some e/accs who are basically transhumanists who don’t think AI risk is significant so they want to accelerate as fast as possible so we get transhumanism soon. There are hardly any of these people since there are hardly any transhumanists, but they have some outsized influence on twitter.
Then there are a way larger number of people who don’t think AI risk is significant so they want to accelerate so we can get the benefits of it sooner, but they’re not transhumanists—they just think that the benefits will be the same as those of other technologies but more so (better medicine, economic growth, a stronger US military etc). I would say that Tan/Andreessen/Sacks fall into this space but so does JD Vance and in fact probably most people in silicon valley.
I would guess ~75% chance that a Chinese lab releasing an open-source Mythos-level model would cause a significant wave of cyberattacks, to the extent that ordinary people would notice the impact on their lives.
This is a pretty rough guess just based on the cybersecurity community’s reaction to Mythos and Project Glasswing. There seems to be a consensus that Anthropic just releasing Mythos in April with no safeguards would have been extremely dangerous.
I’d expect a Chinese lab releasing a Mythos-level in, say, December would be worse in some respects because there is now much greater awareness of the capabilities of AI for cybercrime than there was pre-Mythos. Obviously, the existence of Mythos (and Mythos+n models) for cyberdefense will limit the impact in the OECD, but I don’t think by EOY there will have been much AI-driven hardening of cyberdefense in China/the developing world since they’re not included in initiatives like Project Glasswing. I’m thinking particularly of institutions like local governments, hospitals and utilities.
Main reasons I could be wrong:
1) The cybersecurity, Anthropic and the US government’s reaction to Mythos might have been a huge overreaction and it’s not so capable after all
2) Maybe cyberdefenders who don’t have access to Mythos will take the initiative to use the AIs they do have access to to improve cyberdefense over the next few months, and this will be enough
3) It could take time for awareness to spread among cybercriminals that GLM 5.x is Mythos-level and that you can circumvent any safeguards, and this will give time for cyberdefenders to use it to fix bugs.
Does anyone with a medical/engineering background have any thoughts on Midjourney’s new “Midjourney Scanner”?
https://www.midjourney.com/medical/blogpost
I’m seeing a lot of excited posts on twitter about this but I’m inherently extremely sceptical of an image model company pivoting to medical technology.
I’m also quite sceptical of the concept of easily available full-body scanning, at least for the near future, given it could seemingly lead to a lot of unnecessary medical treatment based on benign abnormalities.
I would actually say that the fact that China seems to be drifting towards allowing a Mythos-level model to be open-sourced (probably by EOY on current trends) suggests that they still aren’t taking AI very seriously, contra to AI-2027. I expect open-source Mythos would cause a wave of cyberattacks that would be a disaster for China and the developing world so that might be the point where they wake up.
Can’t get past the paywall; wonder whether this is because John Jumper left or the cause of him leaving