i don’t get why lots of people are so insistent on talking about it
Setting aside the object level arguments. The actual main reason there’s so much talk about model welfare is that the EA people who previously worked on animal welfare were excited to find a way to get in on the AI action and AI safety funding.
When people compliment you on highly salient attributes, you should be skeptical.
If you have a very prominent tattoo or a funky rainbow-colored jacket, people will frequently give you compliments about it, whether or not it is actually good. When there is a highly conspicuous thing, people cannot resist commenting on it. But the only polite way to mention something about someone’s appearance is to give a compliment. So people who make terrible fashion choices are frequently reinforced in their belief that everyone around them thinks they look great.
Similarly, if you give a public speech in front of a large audience or perform a song at a concert, everyone will compliment your performance, even if it was bad, because they can’t bring themselves not to mention the most salient fact about you at that moment, and a compliment is the only socially acceptable way to do that.
Update:
PauseAI UK daily sign-ups chart:
Anthropic pretraining researcher resigns over safety concerns: https://x.com/hilbertspaess/status/2097476196791709843
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible—but I hear the same people express fear privately. No other human activity poses this level of danger.
If you work at an AI lab, possibly the most impactful you can do now is resign and try to start a social cascade that stigmatises participating in the AI race.
But please do it right. Most AI employee resignations so far have been unstrategic, seemingly spur of the moment decisions with little consideration about making the most of the opportunity. You should have a media strategy lined up before you leave, journalists briefed, statements written and edited.
Or even better, you should not directly resign but instead use your newfound freedom to send the costliest possible signal to your company and to the world that you are actually scared about AI—to the point where you are less worried about keeping your job than doing something to slow AI progress.
If you are in this position, DM me if you want help executing this as effectively as possible.
Edit: It seems that Jacob Coxon did have a media strategy and it paid off.
If they don’t fire you, sit outside and join the protest.
It’s concerning that such a high proportion of the best people I know in AI safety go into grantmaking, running upskilling programs or building AI safety courses / talent pipelines.
I expect if we had better ideas about what to do, the best people would mostly just go and do those things directly instead of finding / funding others to do the actual work of preventing AI catastrophe.
140 people came to the UK Parliament yesterday for the PauseAI conference.
![Parliament Audience 1 [Large].jpeg](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788899622/lexical_client_uploads/p2ngb2mnkotiywdgh2vm.jpg)
The panel speakers were:
Dame Chi Onwurah MP, Chair of the Science, Innovation & Technology Committee
Professor Stuart Russell, world-renowed AI scientist
Lord Tim Clement-Jones, Co-Founder and Co-Chair of the All-Party Parliamentary Group on AI
Lord Lionel Tarassenko—President of Reuben College, Oxford
Iqbal Mohamed MP, leader of the Dec 2025 Westminster Hall debate on AI safety
Brando Benifei MEP, lead architect of the EU AI Act
![Parliament Speakers 1 [Large].jpeg](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1788899643/lexical_client_uploads/jlxmemk7eqg4qpbnf7av.jpg)
I think this is probably the largest event on AI safety there’s been in Parliament. The tone from all the panelists was very supportive and most expressed very serious concern about AI, including the risk of extinction.
Brando Benifei gave us a dose of harsh reality with his assessment about the difficulty of achieving a global treaty of AI. But he remains supportive of PauseAI and seems to really by trying his best to get the EU to lead international cooperation for a ‘CERN for AI’ type solution.
[Linkpost for https://x.com/PauseAI_UK/status/2097102594908803088]
One update I’ve made here is to see my work as being motivated primarily by “improve public discourse on the state of safety / alignment” instead of “propose solutions to alignment” and adapt accordingly.
If this is true, shouldn’t you stop spending your whole time doing alignment research and start spending most of your time on public communication?
Not a pause, but this looks to me like a fairly serious effort to prioritise alignement:
We also temporarily reassigned a portion of the company to these efforts. Roughly 150 product engineers were redirected to security, reliability, and privacy; researchers also rotated out of pretraining or RL to focus on safeguards and security; and our product teams paused the development of most new features and surfaces. We set strict exit criteria for each team to meet before they returned to their prior work. By early summer, most teams had met these.
As with OpenAI’s pause, this is probably just rational from a short term normal business perspective.
I think this could also be taken as evidence for the opposite. China has no loud safety advocacy and that might be partly why it has no within-the-system AI safety faction.
Mostly I think it’s just no evidence at all. China has no substantial AI safety group of any description, so it’s not surprising their companies don’t take safety seriously.
Although another reason could be that think if they announce a pause everyone will say it’s just marketing hype (which obviously everyone would say).
Purely speculating here, but my guess is that if Anthropic leadership have thought about this, one justification they might be using for not pausing is that they think that the OpenAI pause is kind of fake and mostly PR.
I believe (and maybe Anthropic leadership also believe) that OpenAI probably have paused some things but they likely are not making any sacrifice by doing so. My guess is that they actually just needed to pause for purely commercial reasons because their training pipeline currently produces obviously misaligned AIs that do not make for great products.
Even if this is the case, I think Anthropic should do at least a short pause anyway. They clearly have their own alignment issues and the symbolism and precedent of a joint pause could be extremely valuable.
PauseAI UK is hiring: https://pauseai.uk/jobs
Thousands of people across the UK believe that humanity might soon build dangerous superintelligent AI. And yet many of these people haven’t even taken 10 seconds to email their MP about it. Political intervention is one of the most plausible ways that humanity can avoid catastrophe and there are many easy ways that anyone can help to improve the chance of useful AI legislation:
You can just email your MP and they may agree to sign a statement supporting frontier AI legislation.
You can just go to Parliament and ask to meet with your MP and they might talk to you about AI for up to an hour.
You can just march down the street in protest and newspapers will publish photos of it all over the world.
If the thousands of people that are already concerned about superintelligence were well organised as a political force, they could wield substantial power to influence British AI policy. Help us make that happen!
We’re hiring for:
a software engineer
an operations specialist
a content creator
and several other roles
Applications will be assessed on a rolling basis, so apply soon for the best chance of success.
How do you know it’s oversampled compared to other data? Can’t they recall almost any article on arxiv?
whats does this mean? what’s the hypothesis?
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs.
Are you saying you would have a restricted system prompt that tells the model not to do certain things. To the extent that that would work, does that not end up with a pretty similar result to value alignment (at least for the end user)?
We should expect a pause or slowdown because the current trajectory is not sustainable.
For a while, everyone was underinvesting in AI because they were not sufficiently AGI-pilled. Now, I think a lot of people (the big companies, investors, people in this community) might be overinvesting in AI because they’re not sufficiently AGI-pilled.
AI data center capex is approaching $1 trillion per year. Enough gigawatts are already planned to multiply the AI labs’ compute several times over in the next few years.
And yet the models are going crazy. I feel like the new revelations about misaligned agent swarms from OpenAI are not yet priced in. How can we possibly keep accelerating for another three years like this? If nothing else cybersecurity is surely going to break.
I can see a few possible futures:
Governments step in and regulate pre-emptively before a major catastrophe.
There is a major catastrophe and everyone is forced to slow down.
We fully lose control and AI takes over.
But another three years of just more scaling and acceleration like we’ve had the last three is increasingly infeasible. We are entering the danger zone and the race cannot just carry on as before.
I wouldn’t expect markets to price in the third possible future (total loss of control), but they should price in the first two. I think we could soon see a market correction when investors see that a deceleration will be required to keep models under control.
Lots of people are leaving UK AISI because the pay is too low. Often they go to work on evals at other AI safety orgs where they will get paid more money. This is terrible. It’s far more valuable to have at least one government agency in the world with top tier AI expertise than it is to have a slightly stronger non-profit eval ecosystem.
A competent department creates unique options for the UK government to pass a pre-deployment testing law better than other countries. Or the government could empower them as a a thoughtful AI regulator that actually understands the risks. And AISI pushes other countries to improve by setting a high bar. It could even be transformed at some point into an international body that is the backbone of an AI treaty.
The jobs are in fact pretty well paid by UK tech standards but people there often have more lucrative options available at AI safety non-profits. Is there any way we can pay them to stay?