Okay I’ll layout the theory and demonstrate my conclusion.
Assume we have two parties who are developing AI, Company A and Company B. They both can either develop AGI or not with probabilities and
Let be the probability that a party engaging in AGI development will succeed. Let the reward for creating AGI be .Let the cost of failure to build AGI be and let the cost/reward of researching AGI but not coming in first be
Then for A,
Notice that the more likely AGI appears the less reward is needed to trigger research, and the only risk comes from the term representing the ‘coming in second’ cost. So the optimal decision is AGI research as long as R>0 and P_Bx<R. In fact, the more likely AGI seems, the less incentive companies need to research it. Research causes improvement which incentivizes more research. The game pushes all players towards developing AGI harder the closer AGI appears.
So the only thing that can stop this mechanism is an actual technological or physical barrier to development. A moratorium doesn’t break the theory, it just forces development underground.
Please correct me if I made a mistake.
[EDIT]
Fix to the math to account for the risk of getting caught breaking the ‘No AGI’ law. Thank you Brendan Long!
Let be the probability the government detects the corporation engaging in AGI development. Let be the cost of being caught and let be the cost intrinsic to failing to produce AGI but not getting caught.
Assuming that breaking the law and successfully creating AGI functionally invalidates the law itself, we have:
Because of this assumption that “success in AGI development the law no longer applies”, it forces the cost to be attached to the term carrying the probability of failing to achieve AGI, and the same logic I mentioned above applies. As you get a feedback loop pushing corporations to race towards the ‘forbidden fruit’ so to speak.
If I’m following your math correctly, it seems like this is a fully-general argument that it’s impossible to prevent any action with non-zero reward and non-zero cost of failure. I’m not really a math person, but it seems like something must be wrong with this argument because people fail to do things with non-zero reward and non-zero cost of failure all the time.
It also seems suspicious that your equation has no term for the cost of getting caught breaking the hypothetical anti-AI law/international norm.
I appreciate your feedback and I’ll change the math to account for the cost of getting caught, since I think that’s a significant oversight on my part. A few subtle distinctions I want to point out.
First: I said corporations/companies, not people. Sorry if I didn’t clarify this. The reason behind the corporate framing instead of the personal framing is to motivate the reward to money relationship, and make the quantifiable argument more sound.
Second: AGI works differently. We’re concerned not with a rate of success but with success categorically, i.e. ‘who gets there first?’. This changes how we should look at prevention. With regular crime we focus on reducing the rate when we draft laws, but with AGI we focus on absolutely preventing the action from occurring. Thus, it must be not only upheld legally, but intrinsicly. It must be such that no company will ever participate in such a project, as it fundamentally goes against basic properties of corporate reward functions (sign, relative magnitude, dynamics, etc).
Okay I’ll layout the theory and demonstrate my conclusion.
Assume we have two parties who are developing AI, Company A and Company B. They both can either develop AGI or not with probabilities
Let
Then for A,
Notice that the more likely AGI appears
So the only thing that can stop this mechanism is an actual technological or physical barrier to development. A moratorium doesn’t break the theory, it just forces development underground.
Please correct me if I made a mistake.
[EDIT]
Fix to the math to account for the risk of getting caught breaking the ‘No AGI’ law. Thank you Brendan Long!
Let
Because of this assumption that “success in AGI development
If I’m following your math correctly, it seems like this is a fully-general argument that it’s impossible to prevent any action with non-zero reward and non-zero cost of failure. I’m not really a math person, but it seems like something must be wrong with this argument because people fail to do things with non-zero reward and non-zero cost of failure all the time.
It also seems suspicious that your equation has no term for the cost of getting caught breaking the hypothetical anti-AI law/international norm.
I appreciate your feedback and I’ll change the math to account for the cost of getting caught, since I think that’s a significant oversight on my part. A few subtle distinctions I want to point out.
First: I said corporations/companies, not people. Sorry if I didn’t clarify this. The reason behind the corporate framing instead of the personal framing is to motivate the reward to money relationship, and make the quantifiable argument more sound.
Second: AGI works differently. We’re concerned not with a rate of success but with success categorically, i.e. ‘who gets there first?’. This changes how we should look at prevention. With regular crime we focus on reducing the rate when we draft laws, but with AGI we focus on absolutely preventing the action from occurring. Thus, it must be not only upheld legally, but intrinsicly. It must be such that no company will ever participate in such a project, as it fundamentally goes against basic properties of corporate reward functions (sign, relative magnitude, dynamics, etc).