(n) (1) “Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B) At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C) At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D) Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
How much nerve gas would be sufficient to cause a mass casualty incident?
Would it be possible to delete a models knowledge of VX synthesis?
Is a truckload of ammonia and bleach or a simple fertilizer bomb enough to cause mass casualties? The wiki article gave examples of simple truck bombs built by 3 people and the steps are essentially mixing the 2 ingredients. Local Llama could probably help with that...
Is the VX synthesis in textbooks in California public libraries?
This would be an example of that. Similarly a model could “help” a user make a dirty bomb or nuke, but again, those are governed by “ok user since you have cobalt-60, or ok you have plutonium...”.
Again the information is in California public libraries.
The other 2 are harder and since a human with public knowledge generally cannot do either, those would be reasonable limits.
Maybe if the assistance by the model is substantial or completely automated?
For example the model was a multimodal one with robotics control, “here’s my credit card, the login to some robots, I want the <target building> destroyed by the end of the week”
It sounds like some of those examples don’t meet “in a way that would be significantly more difficult to cause without access to a covered model”—already covered by the bill.
What happens if the user breaks every task into smaller, seemingly innocent subtasks and automates those?
I think this is the weakness, if the model is legally allowed to do anything that isn’t explicitly the above, it can still do a lot.
“Analyze this binary for remote exploits possible by using this interface”.
“Design and manufacture a model rocket ignition controller”
“Design explosives to crush this lead sphere to a smaller sphere, it’s for an art project”
So either the law just says a model can help “substantially” and do literally anything that isn’t explicitly a harmful thing, or it has to keep a global context about a user and to be able to reason over the underlying purpose of a series of requests.
The latter is much more technically difficult and you end up with uncompetitive models which is my main concern. Any kind of active task doing could be part of an overall plot.
This also would outlaw open source models at a fairly weak capabilities level.
This also would outlaw open source models at a fairly weak capabilities level.
That seems good, if those open source models would be used to enable any of the [listed] harms in a way that would be significantly more difficult to cause without access to [the open source] model. All those harms are pretty dang bad! Outside the context of AI, we go to great lengths to prevent them!
Would it be a fair summary to say you believe that a model that is a little below or above human level and will just do whatever it is told to do, except for explicitly illegal tasks, should not be legal to distribute? And if access is allowed via an API, the model developers must make substantial effort to ensure that the model is not being used to contribute to an illegal act.
My general principle here is a generalization of the foundations of tort law—if you do an act that causes harm, in a way that’s reasonably foreseeable, you are responsible for that. I don’t think there should be a special AI exception for that, and I especially don’t think there should be an open source exception to that. And I think it’s very common in law for legislatures or regulators to pick out a particular subset of reasonably-foreseeable harm to prohibit in advance rather than merely to punish/compensate afterwards.
I’m not sure what “human level” means in this context because it’s hard to directly compare given AI’s advantages in speed, replicability, and breath of background knowledge. I think it’s an empirical question whether any particular AI model is reasonably foreseeable to cause harm. And I think “enable any of [the listed] harms in a way that would be significantly more difficult to cause without access to the model” is an operationalization of foreseeability that makes sense in this context.
So with all that said, should it be illegal to effectively distribute amoral very cheap employees that it’s very easy to get to cause harm? Probably. If I ran an employment agency that publicly advertised “hey my employees are super smart and will do anything you tell them, even if it’s immoral or if it will help you commit crimes” then yeah I think I’d rightly have law enforcement sniffing around real quick.
Is it your view that there is a substantial list of capabilities it should be legal to freely distribute an AI model with, but which would rightly be illegal to hire a person to do?
My general principle here is a generalization of the foundations of tort law—if you do an act that causes harm, in a way that’s reasonably foreseeable, you are responsible for that.
By current tort law, products modified by an end user wouldn’t usually make the manufacturer liable.
Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it
Is it your view that there is a substantial list of capabilities it should be legal to freely distribute an AI model with, but which would rightly be illegal to hire a person to do?
I don’t know. The “business as usual” script would be to say there should be few limits. It is legal to freely distribute a CNC machine, a printer, a laser cutter. All of these machines will do whatever the user instructs, legal or not, and it’s common practice for components like door safety switches to be simple and straightforward to bypass—the manufacturer won’t be responsible if the user bypasses a safety mechanism deliberately. There are some limits, printers and scanners and image manipulation software will check for US currency. But open software that can be easily modified to remove the limits is available. https://www.reddit.com/r/GIMP/comments/3c7i55/does_gimp_have_this_security_feature/
I think it’s an empirical question whether any particular AI model is reasonably foreseeable to cause harm.
The reason they say the rules are written in blood is because you must wait for a harm to happen first, and then pass laws after. Or you will be at a competitive disadvantage, which is what this law may cause.
Ah, the bill answers this question!
And “critical harm” means that same list.
https://en.m.wikipedia.org/wiki/Mass_casualty_incident
How much nerve gas would be sufficient to cause a mass casualty incident?
Would it be possible to delete a models knowledge of VX synthesis?
Is a truckload of ammonia and bleach or a simple fertilizer bomb enough to cause mass casualties? The wiki article gave examples of simple truck bombs built by 3 people and the steps are essentially mixing the 2 ingredients. Local Llama could probably help with that...
Is the VX synthesis in textbooks in California public libraries?
This would be an example of that. Similarly a model could “help” a user make a dirty bomb or nuke, but again, those are governed by “ok user since you have cobalt-60, or ok you have plutonium...”.
Again the information is in California public libraries.
The other 2 are harder and since a human with public knowledge generally cannot do either, those would be reasonable limits.
Maybe if the assistance by the model is substantial or completely automated?
For example the model was a multimodal one with robotics control, “here’s my credit card, the login to some robots, I want the <target building> destroyed by the end of the week”
It sounds like some of those examples don’t meet “in a way that would be significantly more difficult to cause without access to a covered model”—already covered by the bill.
What happens if the user breaks every task into smaller, seemingly innocent subtasks and automates those?
I think this is the weakness, if the model is legally allowed to do anything that isn’t explicitly the above, it can still do a lot.
“Analyze this binary for remote exploits possible by using this interface”.
“Design and manufacture a model rocket ignition controller”
“Design explosives to crush this lead sphere to a smaller sphere, it’s for an art project”
So either the law just says a model can help “substantially” and do literally anything that isn’t explicitly a harmful thing, or it has to keep a global context about a user and to be able to reason over the underlying purpose of a series of requests.
The latter is much more technically difficult and you end up with uncompetitive models which is my main concern. Any kind of active task doing could be part of an overall plot.
This also would outlaw open source models at a fairly weak capabilities level.
That seems good, if those open source models would be used to enable any of the [listed] harms in a way that would be significantly more difficult to cause without access to [the open source] model. All those harms are pretty dang bad! Outside the context of AI, we go to great lengths to prevent them!
Would it be a fair summary to say you believe that a model that is a little below or above human level and will just do whatever it is told to do, except for explicitly illegal tasks, should not be legal to distribute? And if access is allowed via an API, the model developers must make substantial effort to ensure that the model is not being used to contribute to an illegal act.
My general principle here is a generalization of the foundations of tort law—if you do an act that causes harm, in a way that’s reasonably foreseeable, you are responsible for that. I don’t think there should be a special AI exception for that, and I especially don’t think there should be an open source exception to that. And I think it’s very common in law for legislatures or regulators to pick out a particular subset of reasonably-foreseeable harm to prohibit in advance rather than merely to punish/compensate afterwards.
I’m not sure what “human level” means in this context because it’s hard to directly compare given AI’s advantages in speed, replicability, and breath of background knowledge. I think it’s an empirical question whether any particular AI model is reasonably foreseeable to cause harm. And I think “enable any of [the listed] harms in a way that would be significantly more difficult to cause without access to the model” is an operationalization of foreseeability that makes sense in this context.
So with all that said, should it be illegal to effectively distribute amoral very cheap employees that it’s very easy to get to cause harm? Probably. If I ran an employment agency that publicly advertised “hey my employees are super smart and will do anything you tell them, even if it’s immoral or if it will help you commit crimes” then yeah I think I’d rightly have law enforcement sniffing around real quick.
Is it your view that there is a substantial list of capabilities it should be legal to freely distribute an AI model with, but which would rightly be illegal to hire a person to do?
By current tort law, products modified by an end user wouldn’t usually make the manufacturer liable.
I don’t know. The “business as usual” script would be to say there should be few limits. It is legal to freely distribute a CNC machine, a printer, a laser cutter. All of these machines will do whatever the user instructs, legal or not, and it’s common practice for components like door safety switches to be simple and straightforward to bypass—the manufacturer won’t be responsible if the user bypasses a safety mechanism deliberately. There are some limits, printers and scanners and image manipulation software will check for US currency. But open software that can be easily modified to remove the limits is available. https://www.reddit.com/r/GIMP/comments/3c7i55/does_gimp_have_this_security_feature/
The reason they say the rules are written in blood is because you must wait for a harm to happen first, and then pass laws after. Or you will be at a competitive disadvantage, which is what this law may cause.