For a different analogy, think about a software that fits on a floppy disk that somehow turns any laptop into an explosive device with a nuclear bomb level yield.
Okay. I get that AGI would be this powerful. What I don’t get is that the code for it would fit onto a floppy disk. When you say I am making a mistake analogizing AGI to existing software projects, what precisely do you mean to say? Is it that it really wouldn’t need very many programmers? Is it that problems with sloppy, rushed coding would be irrelevant? I’m not sure exactly how this counters my point.
I’m not sure I’d be happy to settle with “code gets stolen anyway, so let’s make sure everyone gets access to it”.
I’m not happy with it. I think it’s better than the alternative. See next point.
An actual working AGI could be extremely weaponizable both for very cheap and into something much more dangerous than any software engineering analogy gives reason to suppose, and significantly less useful as a defensive than as an offensive measure.
Agreed. That is precisely why everyone should have it. Because it’s “the one ring”. They say, “absolute power corrupts absolutely” because there are a billion examples of humans abusing power throughout history. You can’t trust anybody with that much power. It will ruin the checks and balances between governments and the people they’re supposed to serve, it will ruin the checks and balances between branches of governments and it will make hackers, spies and any criminal or criminal organization who are capable of stealing the software (this might be terrorists, the mafia, gangs, corrupt government leaders, cult leaders, etc.) into superpowers.
To check and balance the power there needs to be a mutually assured destruction type threat between the following:
The people and the governments they serve.
Each branch of governments and the other branches of those governments.
The pirates, hackers, spies and criminals and the good people in the world.
The reason the US government was set up the way it was—with the right to bear arms and with and balances between branches of government—is because power corrupts and mutually assured destruction keeps the humans accountable, and this type of accountability is necessary to keep the system healthy. In a world where AGI exists, the right to bear arms needs to include AGI, or power imbalances will probably ruin everything.
We can’t assume the AGIs will all be friendly. Even if we succeed in the incredibly hard task of making sure every AGI released is initially friendly, this won’t guarantee they won’t be hacked or fooled into being unfriendly. To think that there’s a way to ensure that they won’t be hacked is foolish.
What would solve the problem of the power of AGI corrupting people if not checks and balances?
Okay. I get that AGI would be this powerful. What I don’t get is that the code for it would fit onto a floppy disk. When you say I am making a mistake analogizing AGI to existing software projects, what precisely do you mean to say? Is it that it really wouldn’t need very many programmers? Is it that problems with sloppy, rushed coding would be irrelevant? I’m not sure exactly how this counters my point.
I’m not happy with it. I think it’s better than the alternative. See next point.
Agreed. That is precisely why everyone should have it. Because it’s “the one ring”. They say, “absolute power corrupts absolutely” because there are a billion examples of humans abusing power throughout history. You can’t trust anybody with that much power. It will ruin the checks and balances between governments and the people they’re supposed to serve, it will ruin the checks and balances between branches of governments and it will make hackers, spies and any criminal or criminal organization who are capable of stealing the software (this might be terrorists, the mafia, gangs, corrupt government leaders, cult leaders, etc.) into superpowers.
To check and balance the power there needs to be a mutually assured destruction type threat between the following:
The people and the governments they serve.
Each branch of governments and the other branches of those governments.
The pirates, hackers, spies and criminals and the good people in the world.
The reason the US government was set up the way it was—with the right to bear arms and with and balances between branches of government—is because power corrupts and mutually assured destruction keeps the humans accountable, and this type of accountability is necessary to keep the system healthy. In a world where AGI exists, the right to bear arms needs to include AGI, or power imbalances will probably ruin everything.
We can’t assume the AGIs will all be friendly. Even if we succeed in the incredibly hard task of making sure every AGI released is initially friendly, this won’t guarantee they won’t be hacked or fooled into being unfriendly. To think that there’s a way to ensure that they won’t be hacked is foolish.
What would solve the problem of the power of AGI corrupting people if not checks and balances?