I agree that it might not make sense for an AI company itself to unleash bioweapons. But it could make sense to a giant pharma company to reap profits by creating a biovirus along with manufacturing its antidote. The incentives are huge for this anyways for any actor who gets away with this undetected.
We are already having GPT-5 with biorisk capabilities and which could be exploited using jailbreaks and its report suggests that out of 46 jailbreaks, they think only 3 could practical help in bioweapon development. Which they say have now been blocked by the monitor. Considering this, even more red-teaming efforts like 50 more novel ways of jailbreaks could easily have minimum two or three chances to get sufficient practical insights to create bioweapons. These chances could be more than claimed by these AI companies because there are already questions about whether the labs are actually correctly evaluating and fully measuring the bioweapon capabilities of a model.
And evaluating a model for absence of certain capability is even more difficult when running evals, which means a model could have hidden more severe capability and risk that the evals have failed to cover potentially exacerbating risks across all domains including biorisk.
Additionally, I don’t think any bad actor needs to have access to the weights to capable models. At best, it needs to have sufficient biology/virology knowledge along with jailbreaking a capable model (or fine-tuning an open-source AI model with similar capability exploiting dual-use risks). And IMO it’s only going to get more feasible with time.
I agree that it might not make sense for an AI company itself to unleash bioweapons. But it could make sense to a giant pharma company to reap profits by creating a biovirus along with manufacturing its antidote. The incentives are huge for this anyways for any actor who gets away with this undetected.
We are already having GPT-5 with biorisk capabilities and which could be exploited using jailbreaks and its report suggests that out of 46 jailbreaks, they think only 3 could practical help in bioweapon development. Which they say have now been blocked by the monitor. Considering this, even more red-teaming efforts like 50 more novel ways of jailbreaks could easily have minimum two or three chances to get sufficient practical insights to create bioweapons. These chances could be more than claimed by these AI companies because there are already questions about whether the labs are actually correctly evaluating and fully measuring the bioweapon capabilities of a model.
And evaluating a model for absence of certain capability is even more difficult when running evals, which means a model could have hidden more severe capability and risk that the evals have failed to cover potentially exacerbating risks across all domains including biorisk.
Additionally, I don’t think any bad actor needs to have access to the weights to capable models. At best, it needs to have sufficient biology/virology knowledge along with jailbreaking a capable model (or fine-tuning an open-source AI model with similar capability exploiting dual-use risks). And IMO it’s only going to get more feasible with time.