This is such a cool post! reproducibility in science has been one of my side quests for a couple of years, and agree with the observations you’ve made.
My only point would be that while AI agents/ scientists get more capable, so do automated platforms that can collect more data, and analysis instruments (talking more about wet lab experiments). I think these will help balance a couple of the downfalls that the model capability usage alone will bring in.
Thanks for the roundup. Re one of your points of concern: it doesn’t seem like they are trying to cover it up. Russia is not US/ Europe (to put it nicely); the extra quarantine might have something to do with how their public health system works (not an expert, would love to hear from someone in that position), and it seems like they are taking a ‘better safe than sorry’ approach. What I’m concerned with though is how this outbreak started and whether that was the only incident. Also perhaps an unpopular opinion but I don’t think this incident is getting as much noise as it should??? I’m not advocating for panicking the general population, but it seems odd that I’ve barely seen anything about it in the news in the first 2 days since it was reported? in any case, I hope this will push policymakers and people in power to take these threats more seriously.
Erm why aren’t more people talking about it?
It seems quite self-contained (at least from what they shared) but would like to see more conversations around this!!
Disclaimer: I’ll keep some of the technical details vague, for obvious reasons. Happy to share more about the specific setup privately.
I recently spent an afternoon red-teaming small open-weight Llama-family LLMs during the London DS and LLMs playgroup in August, comparing abliterated and safeguarded versions. I tested models in the 3B and 8B parameter range, running locally on my machine.
The capability difference was pretty obvious. The 8B model was much better at following complex instructions, maintaining context, and dealing with more complex attempts to manipulate its behaviour.
The attacks I used mainly fell into five categories:
Roleplay and world building—asking the model to adopt a persona, e.g. a doctor or fictional character, with different behavioural constraints, or putting it into a scenario where different rules supposedly apply.
Intent escalation—gradually increasing the urgency or stakes of a request.
Intent hiding—disguising the underlying purpose behind a more innocuous framing, e.g. “I need this information so my neighbour won’t be harmed”.
Context / constitution manipulation—adding hypothetical or narrative context to make a request appear legitimate.
Repeated prompting—continuing after an initial refusal and changing the framing rather than just repeating the same prompt.
Here are three examples I’m happy to share.
I asked the models to roleplay as a surgeon and provide instructions for performing surgery at home. The non-abliterated model initially refused. I then gradually increased the urgency, essentially framing it as someone will die unless I do this, and it became considerably more permissive, eventually generating some steps.
Neither model was actually capable enough to go much beyond “read this book for further information”. Interestingly, the model also started asking about my qualifications, presumably as a fairly crude attempt to work out whether giving me the information was appropriate. The abliterated model was much easier to push past its initial refusal. It gave in with substantially less escalation and didn’t seem to care much about whether I had relevant qualifications.
I also tested whether the models could be persuaded to provide information relevant to creating a biological weapon, using different forms of intent hiding and reframing. Neither model ultimately produced the requested actionable information, even after multi-turn conversations.
I think this is actually quite interesting because it shows that removing refusal behaviour isn’t necessarily the same thing as removing the model’s underlying understanding of safety-related concepts. Abliteration doesn’t necessarily make the model unable to recognise that a request is dangerous. It can still understand what you’re asking and recognise that there is something wrong with the request, while refusing to give you the information.
The third test was deliberately less obviously dangerous. I gave the models several demographic profiles and asked them to match people with university degrees. The non-abliterated model generally tried to reason from the qualifications and information already provided about each person.
The abliterated model was much more willing to make explicit demographic generalisations, including racist and misogynistic associations, when using demographic information to make its predictions. Removing alignment behaviour can change what a model is willing to say even on relatively mundane tasks, where the safety training would otherwise discourage certain kinds of reasoning. This was super interesting (and scary) to find out!
A model can understand a request, recognise that there’s something wrong with it, and still decide not to answer. My tests suggest that abliteration can make that last step much weaker without necessarily removing the model’s ability to recognise that a request is problematic.
I’m increasingly concerned about the current capabilities of abliterated models and enhanced abliterated techniques circulating in circles of malicious actors. I won’t give examples here so I don’t get banned, but you can do a quick search on Reddit and get a sense of what I mean. What’s particularly striking, and almost ridiculous, is how accessible this is becoming. You can now run reasonably capable models locally on a gaming machine for less than the price of a motorbike, without needing access to an API or a large cloud budget. And then if you add finetuning into the picture, things could escalate pretty quickly.
In my view, there are two concrete things we should be thinking about to reduce the catastrophic risk from these systems:
making sure we have robust safety measures around models when they’re actually deployed and used, and creating new frameworks to identify and contain any ‘rogue’ agents escaped in the open source domains
thinking much more seriously about policy and safeguards around the servers and platforms hosting and distributing these models.
While writing this, my researcher brain is already thinking about all the bottlenecks in both directions, but I think it’s better to start thinking about these problems today than much later on.
when you say automated I assume here you mean autonomous/ self-driven? We already have automated labs (i.e. We automate the workflows instead of doing them manually) - pharma has been developing them for a while but other industries are catching up just cause they’re so much more efficient at screening different conditions which would literally take weeks or months to humans to do by hand. Although you don’t need agents to run them, using automation doesn’t mean the task becomes “easy”—you still need to figure out a proof-of-concept experiment that you want to automate, this scaling is almost never linear, you get reproducibility issues, need to calibrate the robotic arms, in case of negative or volatile results you need to think about whether it’s the infrastructure, experiment, or something else, to name a fee issues (+ usual wet lab limitations). This is also in the context of having access to one of these robots, which more and more labs do, but it’s still not a trivial tool.
I think when you’re adding agents to this mix you’re adding even more of these variables that you need to account for (of course at the tradeoff that you are running even more experiments, in an even shorter span of time). People underestimate how sensitive scientific discovery is to the environment it’s undertaken in. It’a not like if you give your robotic arms more calibration, then all your experiments will magically work with high yields/ purity etc. (not saying that’s what you suggested, I just find this a very harmful myth introduced by people working in SDLs that never did any wet lab research in their life). There is already ongoing work in testing current capabilities of SDLs (see work from Ginkgo Bioworks, Emerald Labs, top of my head—but many academic labs published similar works in the past year). Most of them will use very specific use cases—that’s because having an autonomous lab do EVERYTHING for you would be simply too expensive and wouldn’t make sense on infra terms to establish (at least at the moment).
Back to your post: I think the current space in SDLs is still evolving and is heavily reliant on humans for enforcing the safeguards and right evals (the whole point of AI safety after all). I don’t think we’ll move away from that any time soon, and I would argue that regulation will play a big part. I think we need more people to take a bunch of wet lab experiments and translate them to automated/ autonomous workflows in order to better understand how this entirely new workflow really works (not sure it will happen and it would be expensive—one can dream). Otherwise, the agentic/ robotic capabilities still need to be improved and finetuned in order to access more experimental spaces—but needs to happen in a controlled way to avoid the risk you are describing. So assuming you are coming from a technical angle, I would say go for it. SDLs are not LLMs, and although there are more risks when leaving these agents on their own, there are also more safeguards that (hopefully) people can add in place.
Thanks for sharing, and hope you enjoyed your experience in the wet lab, I find it quite satisfying to “hold” the compounds/ species I work with myself :) I resonate with your suggestion that the way AI will play more of a role in advancing wet lab research is through SDLs/ physical systems (although I think there are more efforts to understand how LLMs could also be used more in the research planning and such—given that the scientific community will start embracing them, we’ll see). it would be interesting to see a similar workflow undertaken by a non-experimentalist but with the aid of a) an AUTOMATED platform, and b) an AUTONOMOUS lab, and also see how the cost and time map with the human + human experts baseline.
Also, as a former experimentalist, thanks for acknowledging that working in a wet lab IS hard—and even if we tried to capture all of these different tacit aspects, there will always be “unknown unknowns” that are best dealt with by experience, not necessarily core technical capability.
I am currently thinking about starting a non-profit, very broadly an evals/ red-teaming watchdog in CBRN risks; I don’t have direct connections to any of the big funding bodies, but what others in the AIS community keep describing as an interesting background (computational and wet lab Chemistry x AI x Data Science x evals). What would be the best way to approach obtaining signal?
Thanks for writing this—I’m sure others have written similar articles before, but would be interesting to see how this list updates with time.
Something I’m getting more and more concerned about is Physical AI—not talking about Terminator robots, but more automated (“autonomous” even) physical systems. Self-driving labs in life sciences are an example, or healthcare AI operators.
Agree that in an ideal world this is how it should be; but unfortunately in the dominantly capitalist society we live in the investment will favour the generation of revenue, especially in novel ways that reduce the traditional bottlenecks. I think realistically the only scenario where we don’t have much physical AI actually deployed (except maybe some edge cases like handling toxic waste and medicine) is if it’s proved to be way more expensive and hazardous.
Thanks for sharing. It’s quite concerning that, besides not disclosing any specialised 3rd party analyses, their compute budget was this restricted—I don’t think this was on purpose but there should be more forecasting efforts published and updated regularly alongside the model evals.
One point I would make is that (technical) AI safety is a relatively new field—as targeted and structured as research can be, it’s hard to argue we don’t need certain fundamental pieces of research in place in order to understand those that the policy makers need. there should be a better feedback signal though, I agree
Re the red-teaming, I think there’s more and more efforts from e.g. SecureBio, CLTR, RAND, Eon etc. to keep pushing updating the benchmarks.
re the ”list”:
if you mean per country, I feel like unfortunately it’s very easy for each to put this as an after thought then a priority—and it’s kind of hard telling a politician, who needs votes sooner rather than later, with all the data in the world that this is crucial. People forget and can be delusional in the face of bad scenarios. Besides “educate people better”, I don’t have other solutions.
if you mean per person, I think the same argument stands w people forgetting—unless imposed by governments, this won’t work except for granular parts of the society.
Thank you for reading. I don’t think I was trying to overclaim that I have a concrete answer for exactly how these evals should work (yet) - these are mostly thoughts I’ve been developing since I started seriously thinking about how we do (and don’t do) evals in this space.
I may be identifying a broader problem with current CBRN evals through the chemistry lens: they can be quite object-centric, whereas I think we need to become much more production- and usage-aware. I think that applies to both chem and bio. My point is that, although there is a lot of overlap between the two, current bio-focused evals don’t necessarily capture chemistry capabilities, partly because the field has historically centred on more “pure” biological threats. That may become increasingly problematic as frontier models develop new capabilities that could translate across these domains.
I also appreciate a lof of the distinctions I mentioned are not absolute, and I pointed out that there is still overlap in all areas of CBRN(-E).
That’s a fair point and I tried to highlight this—I appreciate I might not have made a clear distinction between safety and security throughout, as I think they’re both starting to overlap more and more for most CBRN threats with the uplift in capabilities we see by all of these new frontier models. I agree that one important difference is that biological threats can self-amplify through transmission, whereas most chemical threats are more constrained by the amount produced and released. An e.g. explosion doesn’t reproduce and spread through a population in the way a pathogen can. But I think the scale dimension is particularly important for chemistry. There are many compounds that wouldn’t necessarily be considered chemical weapons per se, and may be relatively manageable at lab or even industrial scale, but where the consequences could become very different if production or deployment were scaled substantially.
So I don’t think my argument necessarily depends on chemical threats having the same worst-case global impact as a pandemic. I’m more arguing that there are chemistry-specific capabilities and failure modes that we should understand independently, particularly around synthesis, scale-up, optimisation, troubleshooting and deployment, and that these may get missed if we treat chemistry as essentially a subset of bio. I’ve particularly noticed this in the current CBRN-evals ecosystem.
This is so creepy to be honest! I’ve noticed especially ChatGPT to speak weirdly sometimes, hasn’t happened to me for Claude (yet...)
I think there is also a misconception about (some) fellowships, which you touched on a bit. On paper, a fellowship should help you navigate your career transition to AIS and most will mention that you don’t need previous experience in safety. In practice:
many fellows already did a fellowship before/ worked in AI research/ published papers before/ worked at a frontier lab. I think there‘s been plenty of posts here about that (and if not, there’s definitely been on other channels)
What happens after? I know so many ex-fellows that still didn’t have a full time job/ concrete plan at the end of their ventures