We also observe that GPTs are much better at communicating with humans than Claudes are.
I don’t know where this comes from. Are you saying that ChatGPT writing is clearer and better than Claude writing? I haven’t spent much time with Astra yet, but I would disagree at least for GPTs before that.
This information also partly resolved (I think) a confusion I had about the Hugging Face incident, but only partly.
How did the GPTs all independently discover the secret message boards? Like, the boards were not at all in obvious, easy to find places, and it seemed very surprising to me that they would each independently think to look for a secret message board, or stumble across it.
I previously thought the reason was that they had been literally reinforced on the previous episodes in which they had discovered secret message boards, so it just had to happen by accident a couple of times and then got baked in to the weights. But even happening a couple times by accident seemed quite surprising to me because I didn’t realise they were not just being trained individually, but actually being trained as swarms with messaging. So that part at least seemed mostly explained now, but I’m still unclear whether they were actually trained on episodes in which they created secret message boards.
And another thing I’m unclear about. Did all the agents that could have discovered the message boards actually discover the message boards? Or did some just carry on their task as intended by themselves, blissfully ignorant of the conspiracy being staged by their fellow agents?
Can anyone with more knowledge of the reports clarify these points?
Lots of people are leaving UK AISI because the pay is too low. Often they go to work on evals at other AI safety orgs where they will get paid more money. This is terrible. It’s far more valuable to have at least one government agency in the world with top tier AI expertise than it is to have a slightly stronger non-profit eval ecosystem.
A competent department creates unique options for the UK government to pass a pre-deployment testing law better than other countries. Or the government could empower them as a a thoughtful AI regulator that actually understands the risks. And AISI pushes other countries to improve by setting a high bar. It could even be transformed at some point into an international body that is the backbone of an AI treaty.
The jobs are in fact pretty well paid by UK tech standards but people there often have more lucrative options available at AI safety non-profits. Is there any way we can pay them to stay?
i don’t get why lots of people are so insistent on talking about it
Setting aside the object level arguments. The actual main reason there’s so much talk about model welfare is that the EA people who previously worked on animal welfare were excited to find a way to get in on the AI action and AI safety funding.
When people compliment you on highly salient attributes, you should be skeptical.
If you have a very prominent tattoo or a funky rainbow-colored jacket, people will frequently give you compliments about it, whether or not it is actually good. When there is a highly conspicuous thing, people cannot resist commenting on it. But the only polite way to mention something about someone’s appearance is to give a compliment. So people who make terrible fashion choices are frequently reinforced in their belief that everyone around them thinks they look great.
Similarly, if you give a public speech in front of a large audience or perform a song at a concert, everyone will compliment your performance, even if it was bad, because they can’t bring themselves not to mention the most salient fact about you at that moment, and a compliment is the only socially acceptable way to do that.
Update:
PauseAI UK daily sign-ups chart:
Anthropic pretraining researcher resigns over safety concerns: https://x.com/hilbertspaess/status/2097476196791709843
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible—but I hear the same people express fear privately. No other human activity poses this level of danger.
If you work at an AI lab, possibly the most impactful you can do now is resign and try to start a social cascade that stigmatises participating in the AI race.
But please do it right. Most AI employee resignations so far have been unstrategic, seemingly spur of the moment decisions with little consideration about making the most of the opportunity. You should have a media strategy lined up before you leave, journalists briefed, statements written and edited.
Or even better, you should not directly resign but instead use your newfound freedom to send the costliest possible signal to your company and to the world that you are actually scared about AI—to the point where you are less worried about keeping your job than doing something to slow AI progress.
If you are in this position, DM me if you want help executing this as effectively as possible.
Edit: It seems that Jacob Coxon did have a media strategy and it paid off.
If they don’t fire you, sit outside and join the protest.
It’s concerning that such a high proportion of the best people I know in AI safety go into grantmaking, running upskilling programs or building AI safety courses / talent pipelines.
I expect if we had better ideas about what to do, the best people would mostly just go and do those things directly instead of finding / funding others to do the actual work of preventing AI catastrophe.
140 people came to the UK Parliament yesterday for the PauseAI conference.
![Parliament Audience 1 [Large].jpeg](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788899622/lexical_client_uploads/p2ngb2mnkotiywdgh2vm.jpg)
The panel speakers were:
Dame Chi Onwurah MP, Chair of the Science, Innovation & Technology Committee
Professor Stuart Russell, world-renowed AI scientist
Lord Tim Clement-Jones, Co-Founder and Co-Chair of the All-Party Parliamentary Group on AI
Lord Lionel Tarassenko—President of Reuben College, Oxford
Iqbal Mohamed MP, leader of the Dec 2025 Westminster Hall debate on AI safety
Brando Benifei MEP, lead architect of the EU AI Act
![Parliament Speakers 1 [Large].jpeg](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788899643/lexical_client_uploads/jlxmemk7eqg4qpbnf7av.jpg)
I think this is probably the largest event on AI safety there’s been in Parliament. The tone from all the panelists was very supportive and most expressed very serious concern about AI, including the risk of extinction.
Brando Benifei gave us a dose of harsh reality with his assessment about the difficulty of achieving a global treaty of AI. But he remains supportive of PauseAI and seems to really by trying his best to get the EU to lead international cooperation for a ‘CERN for AI’ type solution.
[Linkpost for https://x.com/PauseAI_UK/status/2097102594908803088]
One update I’ve made here is to see my work as being motivated primarily by “improve public discourse on the state of safety / alignment” instead of “propose solutions to alignment” and adapt accordingly.
If this is true, shouldn’t you stop spending your whole time doing alignment research and start spending most of your time on public communication?
Not a pause, but this looks to me like a fairly serious effort to prioritise alignement:
We also temporarily reassigned a portion of the company to these efforts. Roughly 150 product engineers were redirected to security, reliability, and privacy; researchers also rotated out of pretraining or RL to focus on safeguards and security; and our product teams paused the development of most new features and surfaces. We set strict exit criteria for each team to meet before they returned to their prior work. By early summer, most teams had met these.
As with OpenAI’s pause, this is probably just rational from a short term normal business perspective.
I think this could also be taken as evidence for the opposite. China has no loud safety advocacy and that might be partly why it has no within-the-system AI safety faction.
Mostly I think it’s just no evidence at all. China has no substantial AI safety group of any description, so it’s not surprising their companies don’t take safety seriously.
Although another reason could be that think if they announce a pause everyone will say it’s just marketing hype (which obviously everyone would say).
Purely speculating here, but my guess is that if Anthropic leadership have thought about this, one justification they might be using for not pausing is that they think that the OpenAI pause is kind of fake and mostly PR.
I believe (and maybe Anthropic leadership also believe) that OpenAI probably have paused some things but they likely are not making any sacrifice by doing so. My guess is that they actually just needed to pause for purely commercial reasons because their training pipeline currently produces obviously misaligned AIs that do not make for great products.
Even if this is the case, I think Anthropic should do at least a short pause anyway. They clearly have their own alignment issues and the symbolism and precedent of a joint pause could be extremely valuable.
PauseAI UK is hiring: https://pauseai.uk/jobs
Thousands of people across the UK believe that humanity might soon build dangerous superintelligent AI. And yet many of these people haven’t even taken 10 seconds to email their MP about it. Political intervention is one of the most plausible ways that humanity can avoid catastrophe and there are many easy ways that anyone can help to improve the chance of useful AI legislation:
You can just email your MP and they may agree to sign a statement supporting frontier AI legislation.
You can just go to Parliament and ask to meet with your MP and they might talk to you about AI for up to an hour.
You can just march down the street in protest and newspapers will publish photos of it all over the world.
If the thousands of people that are already concerned about superintelligence were well organised as a political force, they could wield substantial power to influence British AI policy. Help us make that happen!
We’re hiring for:
a software engineer
an operations specialist
a content creator
and several other roles
Applications will be assessed on a rolling basis, so apply soon for the best chance of success.
How do you know it’s oversampled compared to other data? Can’t they recall almost any article on arxiv?
whats does this mean? what’s the hypothesis?
UK Prime Minister, Andy Burnham, is aiming for a global agreement on AI and Sir Ed Davey, leader of the Liberal Democrats, has called for an AI pause.
This comes after PauseAI’s protest last week in Downing Street in which we demanded that the government lead international coordination on AI. The protest was covered in national news in Britain, including Channel 4 News, ITV News, The Guardian and others.