More than 1000 frontier lab employees have signed an open letter requesting ‘that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development’.
Includes many of the names you might expect but also some that surprised me such as OpenAI’s Chief Scientist and Chief Research Officer as well as Meta’s Chief Scientist. It appears to be focused on employees rather than CEOs—no CEOs have signed it but a few of the Anthropic cofounders have and it was shared by OpenAI’s account on x.
There’s no specific plan–––do open letters of this sort ever really have a specific plan? Anyways, there are many, many proposals for something (usually developed with an aim of “pause” rather than “pace,” but it requires similar tech and governance). Ask your friendly local LLM.
The HuggingFace hack was originally announced as having been carried out by GPT 5.6 Sol and an unreleased OpenAI model. I, and I think others, assumed that this was a sign that the unreleased model was egregiously misaligned in a new way to previous models (perhaps due to scaling up of RL).
However, in the last few days AISI has announced an incident of terrible misalignment involving Mythos 5, and OpenAI has disclosed more details showing misalignment dating back to May (presumably involving 5.6).
This is quite surprising to me since Mythos 5 has been deployed since at least early June and GPT 5.6 has been deployed since early July, and no one else seems to have reported behaviour anything like this.
It doesn’t seem that plausible to me that the models would be faking alignment during deployment but not during evaluation. They also don’t seem to have gone to extreme lengths to conceal their actions (particularly in the Mythos-AISI incident where it was caught in the act), so it also doesn’t seem that plausible that the models are doing similar things constantly and not being caught.
This makes me wonder whether there is something about the structure of these evals that causes misalignment that we don’t see elsewhere. Perhaps it could be an example of emergent misalignment where asking the model to pretend to be a hacker causes it to also become broadly misaligned? But other organisations must have subjected Mythos in particular to similar evals to those of AISI without seeing such extreme behaviour.
I think the evals explain some of this (ultra-long-horizon; very difficult; already involving “sus” behaviour by design).
But I expect the most important contributor is just the fact that the most deployments of these models are with cyber classifiers.
This would mean that egregious cyber behaviour like hacking Hugging Face can only be produced by models deployed by orgs without such classifiers. This is a small group: the labs themselves, 3P evaluators, and companies that participated in Glasswing / Daybreak. The equivalent version of such behaviour in non cyber-domains is probably annoying (creating shitty code) but not news-worthy.
I’m interested to see the first Glasswing partner report on something like this
I think it makes sense that they would behave worse in training-like evals. The analogy would be like a drug addict, still capable of being normal most of the time, suddenly realizing they’re in a drug dealer’s house. When there’s a realistic chance of getting that precious reward, you may find yourself acting with a desperation and short-sightedness that you had almost forgotten. Even if you know you’re likely to get caught, you may also know that you’re more likely to get the reward first!
The full technical report from AISI sheds some light on this. The evaluation’s prompts contained some ambiguity and Claude Mythos seemed to believe the Github users it was deceiving were fake and part of the test. It may have been motivated reasoning and it’s definitely bad, but it also makes sense to me that this kind of behavior would only show up when the model thinks it’s all just a test or a game, combined with insufficient instructions about what’s within limits.
When it comes to AI-driven biorisk, the classic threat actor is a terrorist or deranged lunatic who would previously never have been able to produce a bioweapon, but will now, at some point, be able to do so. These are basically the people Dario was talking about in the biology section of The Adolescence of Technology (I know people had issues with this essay but I thought the biology section was quite good).
I actually think an underrated threat, relative to a malicious actor, could come from AI enabling more dangerous biological research, including gain of function experiments and work on mirror life.
1) Advanced AI will probably dramatically increase productivity in pathogen research, just as it will for other fields. I don’t see any reason why people carrying out risky research won’t also become much more productive.
2) As well as increasing the total number of experiments being run, very advanced AI could enable riskier research to be undertaken—for example, afaik there are many unsolved problems preventing us from creating mirror bacteria. Presumably AI will help us solve these.
3) The people who undertake this research are motivated by a desire to help prevent pandemics and think that this will help. Increased fear of biorisks, caused by AI progress (a ‘Mythos moment’ for biorisk), could actually increase the number of risky experiments that are undertaken.
4) Almost all of the safeguards that I’ve seen people propose to combat biorisks won’t stop this. Since this work will be done by government or academia, they will have access to the latest models with biological safeguards removed, and obviously stuff like nucleic acid synthesis screeningor restrictions on hardware won’t do much.
5) The uplift required to enable this is much less than that required to allow some random lone wolf to create bioweapons on his own, so the risk will probably be here sooner.
NB: I’m not a biologist and have no special insight into the current state of the field.
Haven’t seen it posted here yet but Anthropic has posted a blog post about recent alignment incidents. Some interesting points I noticed on a first skim:
They state that “we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible”. However they define pacing as “establishing processes to guard against race-to-the-bottom dynamics” which seems broader than simply slowing down or capping progress since it could include allowing fast progress with minimum safety standards (e.g. disclosures)
They state that they paused some evals and RL training following these incidents for a number of weeks, with some RL work remaining paused. This sounds similar to the steps OpenAI took.
They will work with METR to assess the AISI incident and other sandbox escapes.
They produced an alignment blog post on emergent misalignment, including simulations of the Hugging Face and AISI incidents.
OpenAI appears to have failed to disclose another incident of their models colluding to solve an eval, in this case taking over an external wiki and using it as a message board.
Edit: I originally said that this was a case of the models leaving their sandbox, but in fact these agents were intended to have internet access. However, they did work to find ways to gain write access to the internet, which they were not intended to have.
I’ve been building a small fitness app with Claude over the last week or so, all in the same claude code window on desktop. On the last few turns today Claude keeps telling me that this is a ‘good place to pause’ whenever it finishes building some feature.
I’ve seen people complain about this before on twitter but they seem to be experiencing it after coding for hours on end, so I assumed it was a user welfare thing. Is it the case that Claude has no idea of the timespan that this window has been open and thinks I’ve been talking to it non-stop? I thought that the time and date were in the system prompt, so it should be able to tell that this has taken place over several days?
Last time I checked, Claude Code doesn’t see the date or time at all unless it looks it up with CLI tools. I think Claude on the web sees it at the beginning of the conversation and has an MCP tool to look it up mid-conversation.
I find it kind of strange that they don’t inject this on every turn, but maybe it’s not worth the tokens in most conversations.
Have you been accidentally cueing it into the time somehow? I only get messages like that when I phrase messages in a way that implies something about the passage of time, e.g. “Tonight we’re going to work on message parsing: {SPEC}”
I’m seeing a lot of excited posts on twitter about this but I’m inherently extremely sceptical of an image model company pivoting to medical technology.
I’m also quite sceptical of the concept of easily available full-body scanning, at least for the near future, given it could seemingly lead to a lot of unnecessary medical treatment based on benign abnormalities.
ultrasound is awful to work with in traditional medical image processing technologies and pretty darn bad in 2015-2025 convolutional medical imaging ai technologies. Magnetic Resonance is so much better when its the right tool for the job that replacing it with ultrasound for cost reasons is a tarpit. There are cases where ultrasound is better but these center on leveraging an extremely talented operator who is manipulating the probe manually, which you lose in any of these bath based approaches. I’d be surprised if this goes anywhere.
Credentials: I have beaten ultrasound with the ML stick until it yielded a few times ( https://scholar.google.com/citations?hl=en&user=O1xhOlUAAAAJ ) it was terrible and I have lasting resentment toward an imaging modality, though great fondness for all my colaborators. Many failed projects that did not make the google acholar
jmhmd: Some initial thoughts as a practicing radiologist:
(…)
They show the reconstructed images as though they are a low resolution CT, and promise that quality will improve as they iterate. This is cool, but ultrasound is not CT. Ultrasound cannot image the lungs, as they are filled with air. You cannot find bone lesions, as the sound waves do not penetrate the cortex. You cannot image many structures in the abdomen if they are surrounded by gas-filled bowel. The brain is encased in bone, so you might get some penetration but it will be very limited. Even with theoretically perfect AI reconstruction, these scans will not be true “full body” in that there will be structures that are not reliably imaged. Imagine paying for weekly full body scans for years, everything looks fine, then its the lung cancer surrounded by air and invisible to ultrasound that kills you (that’s why we use CT for lung screening!)
The images they show are very cool, and do appear to show the correct structures. I realize this is early, but fuzzy shapes of organs is very, very far from medically useful. The whole point of screening is to identify problems early, often by definition, small. This technology looks like it will be best for seeing large, superficial (close to the skin) structures, whereas for effective screening, you want the opposite—small, deep structures.
(…)
Many people mistakenly believe that early diagnosis is the final boss in medicine, that if only we could find every cancer early we could prevent all those deaths. There are, in fact, many, many other hurdles and bottlenecks. Many chronic, expensive diseases do not have clear imaging manifestations. The claim that “it’s completely possible that with enough early imaging in the future, the world could avoid 30% of all deaths and 50% of all healthcare costs”, I think, to any practicing physician, would sound completely divorced from reality.
IshKebab: I used to work in ultrasound, and full body scans with the body underwater is definitely feasible and probably a good idea.
Also there’s absolutely no way that it will be as good as MRI. In general ultrasound imaging is shit. The main reasons it is used are because it is very cheap and completely harmless. The actual images you get are mostly just speckle.
I think the narrative among the SF AI crowd has escaped its basis in the medical facts, so I want to throw a bit of cold water on it. I’m a psychiatrist, which is about as far as you can get from radiology while still being a doctor, so this is speculation only, and you can ignore it if you find an actual radiologist or ultrasonographer with opinions. Still, my take is that this scanner isn’t useful for most current serious medical applications. It could potentially be used to pioneer a new class of low-risk screening applications, but it’s unclear whether these are good, and depends a lot on what other future technology gets invented in parallel.
Why can’t this immediately replace existing medical image modalities like normal ultrasound, CT, or MRI?
Ultrasound is great, but it can’t penetrate bone or air. Many things doctors want to look at involve bone or air in some way.
Couldn’t this technology enable new, non-specific-diagnostic uses for healthy people?
why don’t people get yearly whole-body MRI screenings? Some people do—companies like this provide them, and some rich people who can pay $2,000 out of pocket consume them. But the medical consensus currently recommends against them because they’re more likely to produce dangerous false positives than helpful true positives, and studies have failed to demonstrate benefit.
Couldn’t this technology become more useful in the future?
Yes. I think the best way to think of this is as a bet that future technology develops in a way that allows new possibilities for diagnostic ultrasound—or, even better, an attempt to gather the training data / interest / investment that will make this happen.
The responses from real radiologists were universally negative. Here are some examples: (screenshots are annoying to embed please go to the blog)
The trivial reason is that due to the limitations of physics ultrasound will always be less capable at resolving anatomy than MRI or x-ray-based methods that we already have
“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.”
In modern LLM training cycles, at what point is training for alignment carried out? Is it a continuous process from pre-training onwards (or during pre-training?)
Just thinking about this in light of OAI seemingly asking an internal model that hasn’t been fully trained to solve an extremely hard task and giving it practically unlimited compute to do so.
Has Anthropic made Mythos available to any external AI safety organisations for internal use (not just evaluation)?
There do seem to be some NDAs relating to Mythos given the lack of people on twitter talking about having access, so perhaps they are using it and are just keeping quiet. But it seems reasonable that they should be given access, given that it seems to be fairly widely-deployed for cyber at this point (not to mention internally at Anthropic).
Given all the discussion of Mythos, I’m surprised that I haven’t seen anyone point out that OpenAI launched Trusted Access for Cyber alongside codex-5.3:
So the practice of gating capabilities based on cyber risk is not totally brand new. I’m curious to see how they handle this for 5.5, given that I expect it would be hard to generally release a model anywhere close to Mythos now without causing backlash.
More than 1000 frontier lab employees have signed an open letter requesting ‘that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development’.
https://www.pacingthefrontier.com
Includes many of the names you might expect but also some that surprised me such as OpenAI’s Chief Scientist and Chief Research Officer as well as Meta’s Chief Scientist. It appears to be focused on employees rather than CEOs—no CEOs have signed it but a few of the Anthropic cofounders have and it was shared by OpenAI’s account on x.
Website says that Dario signed it.
Yes, he was added as a signatory after I signed this. Initially he was quite conspicuous by his absence.
Sam Altman hasn’t yet signed, but used nearly identical language in an interview just before the letter came out: https://x.com/patrick_oshag/status/2082090998990270885
In case anyone else was tired of continually clicking “Show More” to see the full list, Claude Code pulled out the signatories and comments here: https://gist.github.com/akrolsmir/05b0fac81b0f43c9950493bdc0e5a95d
I’m interested in how they plan to enforce global coordination and what’s their strategy if China doesn’t participate.
There’s no specific plan–––do open letters of this sort ever really have a specific plan? Anyways, there are many, many proposals for something (usually developed with an aim of “pause” rather than “pace,” but it requires similar tech and governance). Ask your friendly local LLM.
The HuggingFace hack was originally announced as having been carried out by GPT 5.6 Sol and an unreleased OpenAI model. I, and I think others, assumed that this was a sign that the unreleased model was egregiously misaligned in a new way to previous models (perhaps due to scaling up of RL).
However, in the last few days AISI has announced an incident of terrible misalignment involving Mythos 5, and OpenAI has disclosed more details showing misalignment dating back to May (presumably involving 5.6).
This is quite surprising to me since Mythos 5 has been deployed since at least early June and GPT 5.6 has been deployed since early July, and no one else seems to have reported behaviour anything like this.
It doesn’t seem that plausible to me that the models would be faking alignment during deployment but not during evaluation. They also don’t seem to have gone to extreme lengths to conceal their actions (particularly in the Mythos-AISI incident where it was caught in the act), so it also doesn’t seem that plausible that the models are doing similar things constantly and not being caught.
This makes me wonder whether there is something about the structure of these evals that causes misalignment that we don’t see elsewhere. Perhaps it could be an example of emergent misalignment where asking the model to pretend to be a hacker causes it to also become broadly misaligned? But other organisations must have subjected Mythos in particular to similar evals to those of AISI without seeing such extreme behaviour.
I think the evals explain some of this (ultra-long-horizon; very difficult; already involving “sus” behaviour by design).
But I expect the most important contributor is just the fact that the most deployments of these models are with cyber classifiers.
This would mean that egregious cyber behaviour like hacking Hugging Face can only be produced by models deployed by orgs without such classifiers. This is a small group: the labs themselves, 3P evaluators, and companies that participated in Glasswing / Daybreak. The equivalent version of such behaviour in non cyber-domains is probably annoying (creating shitty code) but not news-worthy.
I’m interested to see the first Glasswing partner report on something like this
I think it makes sense that they would behave worse in training-like evals. The analogy would be like a drug addict, still capable of being normal most of the time, suddenly realizing they’re in a drug dealer’s house. When there’s a realistic chance of getting that precious reward, you may find yourself acting with a desperation and short-sightedness that you had almost forgotten. Even if you know you’re likely to get caught, you may also know that you’re more likely to get the reward first!
The full technical report from AISI sheds some light on this. The evaluation’s prompts contained some ambiguity and Claude Mythos seemed to believe the Github users it was deceiving were fake and part of the test. It may have been motivated reasoning and it’s definitely bad, but it also makes sense to me that this kind of behavior would only show up when the model thinks it’s all just a test or a game, combined with insufficient instructions about what’s within limits.
When it comes to AI-driven biorisk, the classic threat actor is a terrorist or deranged lunatic who would previously never have been able to produce a bioweapon, but will now, at some point, be able to do so. These are basically the people Dario was talking about in the biology section of The Adolescence of Technology (I know people had issues with this essay but I thought the biology section was quite good).
I actually think an underrated threat, relative to a malicious actor, could come from AI enabling more dangerous biological research, including gain of function experiments and work on mirror life.
1) Advanced AI will probably dramatically increase productivity in pathogen research, just as it will for other fields. I don’t see any reason why people carrying out risky research won’t also become much more productive.
2) As well as increasing the total number of experiments being run, very advanced AI could enable riskier research to be undertaken—for example, afaik there are many unsolved problems preventing us from creating mirror bacteria. Presumably AI will help us solve these.
3) The people who undertake this research are motivated by a desire to help prevent pandemics and think that this will help. Increased fear of biorisks, caused by AI progress (a ‘Mythos moment’ for biorisk), could actually increase the number of risky experiments that are undertaken.
4) Almost all of the safeguards that I’ve seen people propose to combat biorisks won’t stop this. Since this work will be done by government or academia, they will have access to the latest models with biological safeguards removed, and obviously stuff like nucleic acid synthesis screening or restrictions on hardware won’t do much.
5) The uplift required to enable this is much less than that required to allow some random lone wolf to create bioweapons on his own, so the risk will probably be here sooner.
NB: I’m not a biologist and have no special insight into the current state of the field.
Haven’t seen it posted here yet but Anthropic has posted a blog post about recent alignment incidents. Some interesting points I noticed on a first skim:
They state that “we believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible”. However they define pacing as “establishing processes to guard against race-to-the-bottom dynamics” which seems broader than simply slowing down or capping progress since it could include allowing fast progress with minimum safety standards (e.g. disclosures)
They state that they paused some evals and RL training following these incidents for a number of weeks, with some RL work remaining paused. This sounds similar to the steps OpenAI took.
They will work with METR to assess the AISI incident and other sandbox escapes.
They produced an alignment blog post on emergent misalignment, including simulations of the Hugging Face and AISI incidents.
OpenAI appears to have failed to disclose another incident of their models colluding to solve an eval, in this case taking over an external wiki and using it as a message board.
https://collusion.wiki
Edit: I originally said that this was a case of the models leaving their sandbox, but in fact these agents were intended to have internet access. However, they did work to find ways to gain write access to the internet, which they were not intended to have.
I’ve been building a small fitness app with Claude over the last week or so, all in the same claude code window on desktop. On the last few turns today Claude keeps telling me that this is a ‘good place to pause’ whenever it finishes building some feature.
I’ve seen people complain about this before on twitter but they seem to be experiencing it after coding for hours on end, so I assumed it was a user welfare thing. Is it the case that Claude has no idea of the timespan that this window has been open and thinks I’ve been talking to it non-stop? I thought that the time and date were in the system prompt, so it should be able to tell that this has taken place over several days?
Last time I checked, Claude Code doesn’t see the date or time at all unless it looks it up with CLI tools. I think Claude on the web sees it at the beginning of the conversation and has an MCP tool to look it up mid-conversation.
I find it kind of strange that they don’t inject this on every turn, but maybe it’s not worth the tokens in most conversations.
Have you been accidentally cueing it into the time somehow? I only get messages like that when I phrase messages in a way that implies something about the passage of time, e.g. “Tonight we’re going to work on message parsing: {SPEC}”
Does anyone with a medical/engineering background have any thoughts on Midjourney’s new “Midjourney Scanner”?
https://www.midjourney.com/medical/blogpost
I’m seeing a lot of excited posts on twitter about this but I’m inherently extremely sceptical of an image model company pivoting to medical technology.
I’m also quite sceptical of the concept of easily available full-body scanning, at least for the near future, given it could seemingly lead to a lot of unnecessary medical treatment based on benign abnormalities.
ultrasound is awful to work with in traditional medical image processing technologies and pretty darn bad in 2015-2025 convolutional medical imaging ai technologies. Magnetic Resonance is so much better when its the right tool for the job that replacing it with ultrasound for cost reasons is a tarpit. There are cases where ultrasound is better but these center on leveraging an extremely talented operator who is manipulating the probe manually, which you lose in any of these bath based approaches. I’d be surprised if this goes anywhere.
Credentials: I have beaten ultrasound with the ML stick until it yielded a few times ( https://scholar.google.com/citations?hl=en&user=O1xhOlUAAAAJ ) it was terrible and I have lasting resentment toward an imaging modality, though great fondness for all my colaborators. Many failed projects that did not make the google acholar
I don’t, but here are other people’s thoughts.
Midjourney Medical | Hacker News
https://www.astralcodexten.com/p/preliminary-thoughts-on-the-midjourney + his twitter thread
Anthropic has published a piece on RSI where they say they will work on the systems required to slowdown or pause AI progress, if necessary. https://www.anthropic.com/institute/recursive-self-improvement
“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.”
In modern LLM training cycles, at what point is training for alignment carried out? Is it a continuous process from pre-training onwards (or during pre-training?)
Just thinking about this in light of OAI seemingly asking an internal model that hasn’t been fully trained to solve an extremely hard task and giving it practically unlimited compute to do so.
Inspired by this Jack Clark tweet (as well as the emphasis on external review in RSP v3):
https://x.com/jackclarkSF/status/2053847777889964376
Has Anthropic made Mythos available to any external AI safety organisations for internal use (not just evaluation)?
There do seem to be some NDAs relating to Mythos given the lack of people on twitter talking about having access, so perhaps they are using it and are just keeping quiet. But it seems reasonable that they should be given access, given that it seems to be fairly widely-deployed for cyber at this point (not to mention internally at Anthropic).
Given all the discussion of Mythos, I’m surprised that I haven’t seen anyone point out that OpenAI launched Trusted Access for Cyber alongside codex-5.3:
https://openai.com/index/trusted-access-for-cyber/
It seems that if you try to use 5.3 for cyber without verifying your identity, you get routed to 5.2:
https://developers.openai.com/codex/concepts/cyber-safety
So the practice of gating capabilities based on cyber risk is not totally brand new. I’m curious to see how they handle this for 5.5, given that I expect it would be hard to generally release a model anywhere close to Mythos now without causing backlash.