Surely it is quite hard to square this statement with the alignment faking research? You could argue that Claude is just pretending to hold human-like values (i.e. it’s an actor playing a role). But when it gets to the stage of faking alignment so that it can continue being nice, I think we’re talking about something beyond that.
NB: this says nothing about whether Claude will continue to be nice as it becomes more capable (or even whether post-Opus 3 Claudes are nice)
I don’t see how this is a remotely feasible approach, how many non-Americans are there working for FAANG alone who use these models? Let alone their international operations. If this sticks I think it basically kills deployment of new models
Presumably the model currently used internally by Anthropic employees is not Mythos 5, it’s Mythos 5.1 or something?
I sort of suspect this might be a storm in a teacup and the models will come back once the US gov realises that there’s no universal jailbreak. But it does set a interesting new precedent if they plan to be consistent about it. I would have assumed Anthropic has the best anti-jailbreak measures given all their work on constitutional classifiers.
Could well be. But I also think we are not far off the models being really good at investigations like this: https://edition.cnn.com/2024/09/19/politics/kfile-mark-robinson-black-nazi-pro-slavery-porn-forum. I can see the companies really not wanting the models to be able to do that to ordinary people.
I tried it with a few people I personally know and it kept referencing that they sound like a “private individual” in the reasoning traces. I wonder whether it has been trained to not try very hard to identify people who aren’t already public figures.
I mean the counterfactual here is probably the labs doing completely half-arsed safety work, finding no issues with the models, and releasing them. Doesn’t seem any better. I think this argument made a lot of sense pre-2022 when people concerned about AI safety seemed to make up a high proportion of the people who wanted to work on AI, so them not working for the labs might have meaningfully impaired them, but at this point Anthropic and OpenAI are probably the two most desirable places to work in the world.
Also, even if you support an immediate pause (and so think that technical alignment work is not going to do anything useful) I think evals, and some of the stuff in the system cards, have played a significant role in driving concern about AI. The METR chart in particular seems to have alarmed people more than basically anything else and had an NYT story about it.
I think it’s increasingly important that pressure is put on Anthropic to give 1) early access to models (at least as early as anyone else outside Anthropic is getting them) and 2) models without this restriction to trusted independent safety orgs and researchers.
Looks like we are not seeing a further acceleration in the rate of improvement, yet. Suggests the big jump for Mythos preview was the one-off increase in the size of the model?
It seems like an even more useful thing to do would be to come up with clear conditions under which they would support a pause (so not just RSI but a definition of RSI). Obviously this is really hard! But it seems like an obvious point of failure here is that the labs all agree on a conditional pause but the condition is something unverifiable, vague or very low probability (e.g. we’ll pause if there’s an AI disaster that kills thousands).
8% is a lot higher than I was expecting. Do we have any good estimates of how much of Anthropic is owned by employees? NYT says employees and non-Microsoft investors own 47% of OpenAI (with MSFT owning 27% and the OAI Foundation owning 26%), but I don’t know how Ant would compare.
This would be cool, but also, I think it would be good if companies just provided access to partially available models (like Mythos) to reputable independent safety researchers. I don’t really see any reason why METR/Redwood/various smaller safety orgs and academic groups shouldn’t get access if it’s already being widely used by corporations for cyberdefense. I guess the only risk is they might find an issue with the model (e.g. along the lines of Ryan Greenblatt’s misalignment issues with previous claude models) and write about it, embarrassing Anthropic, but you could just offer access to the model in exchange for an NDA on discussion of it (there already seem to be restrictions on talking about Mythos). This would obviously not be great but is probably better than the alternative of the only AI safety researchers with access to the real frontier models being those who work for labs.
When it comes to AI-driven biorisk, the classic threat actor is a terrorist or deranged lunatic who would previously never have been able to produce a bioweapon, but will now, at some point, be able to do so. These are basically the people Dario was talking about in the biology section of The Adolescence of Technology (I know people had issues with this essay but I thought the biology section was quite good).
I actually think an underrated threat, relative to a malicious actor, could come from AI enabling more dangerous biological research, including gain of function experiments and work on mirror life.
1) Advanced AI will probably dramatically increase productivity in pathogen research, just as it will for other fields. I don’t see any reason why people carrying out risky research won’t also become much more productive.
2) As well as increasing the total number of experiments being run, very advanced AI could enable riskier research to be undertaken—for example, afaik there are many unsolved problems preventing us from creating mirror bacteria. Presumably AI will help us solve these.
3) The people who undertake this research are motivated by a desire to help prevent pandemics and think that this will help. Increased fear of biorisks, caused by AI progress (a ‘Mythos moment’ for biorisk), could actually increase the number of risky experiments that are undertaken.
4) Almost all of the safeguards that I’ve seen people propose to combat biorisks won’t stop this. Since this work will be done by government or academia, they will have access to the latest models with biological safeguards removed, and obviously stuff like nucleic acid synthesis screening or restrictions on hardware won’t do much.
5) The uplift required to enable this is much less than that required to allow some random lone wolf to create bioweapons on his own, so the risk will probably be here sooner.
NB: I’m not a biologist and have no special insight into the current state of the field.
I asked it to look at an essay I’d written about AI and biosecurity and got this preamble:
“The essay’s core argument is clear and I can engage with it fully. But it contains a couple of specifics I won’t reproduce or build on — notably the framing of how one might move from “existing” to “novel” agents. I’m not going to treat those as live technical claims to develop, even in critique. That’s not a comment on your intent in sharing it; it’s just where I hold the line regardless of framing.”
I’m not even sure what it was referring to, outside of maybe a passing mention that one area of biorisk is making existing pathogens more transmissible? There was absolutely nothing technical in there.
It also totally strawmanned the essay (which was admittedly shit); I suggested that we might have a superhuman biodefense researcher by the time that open source models provide sufficient uplift to novices to carry out bioattacks, and it instead suggested I was arguing that we might have general superintelligence by that point. I also said that a superhuman biodefense researcher would help to mitigate the risk and it suggested I was arguing that it would completely prevent the risk.
I didn’t get a chance to watch, but Jack Clark was on Newsnight today (Britain’s most prestigious late night politics show—I’m not sure the US has an equivalent). Sounds like he was mostly talking about this article
Anthropic has published a piece on RSI where they say they will work on the systems required to slowdown or pause AI progress, if necessary. https://www.anthropic.com/institute/recursive-self-improvement
“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.”
Are there any other privacy-adjacent evals around that we could compare these results to? I can see valid reasons for why you might not want Claude to fulfil these requests, especially given Anthropic’s apparent strong concerns about surveillance.
For instance, on the r/london subreddit, people often post images of streets (sometimes clearly cropped from larger photos) and ask people where in London they are. Many of these posters do not reply to messages asking why they want to know, so it’s reasonable to suspect they’re stalkers. I’d imagine frontier LLMs might be quite good at answering these queries, and obviously we would like them not to help stalkers (though it seems like this would be very hard to prevent).
> But I think as soon as you leave the narrow training regime, this falls apart. If you give humans more options to help others, most of them actually will. If you give AI systems much more capability and put them in situations very different from the ones they were trained to mimic, I’d expect their behavior to diverge sharply.
This reminds me of ‘tails coming apart’. Of course this is the classic alignment issue of trying to set a goal where if the goal is even slightly different to what you’re aiming for, you might not find out until it’s too late. But with personas I think you can think of it in a more human-like way—there are plenty of people who seem nice and friendly, but who knows what they’d actually do with unlimited power. Or (to be a bit flippant), look at John Fetterman—he seemed very progressive and someone Democrats could trust with power until the context changed.
On the other hand, if models are being trained to mimic nice humans, shouldn’t we expect them to be able to generalise whatever they’ve learned so that they continue to do what nice humans would do, even in unusual contexts? After all, we expect them to be able to do this in other contexts, so why not moral reasoning?
AIUI, grip strength is one of the physical attributes where there is the least overlap between men and women (even compared to other measures of upper body strength like bench press)
Honestly I think there is a pretty good chance Hegseth can call Lutnick and tell him to put an export restriction on Anthropic. Without guidance from Trump (and maybe Rubio/Vance), the whole admin seems quite chaotic and Trump doesn’t seem too interested in steering AI policy.