You are risking to oversaturate the AGI research market relative to other life-changing potential technologies, such as engineered life extension or organizational mechanism design.
Such oversaturation may result in two or more internally consistent but poorly compatible AI Safety theories (e.g. distillation-amplification vs. agent foundation), which will end up in two different friendly AI and the the war for correct friendliness will start.
If we have two distinct AI safety plans, the researchers are sensible to have a big discussion on which is better and only turn that one on. If not, and neither AI is fatally flawed, I would expect them to cooperate, they have very similar goals and neither wants war.
Historically, it didn’t work. A most bitter conflicts were between two main branches of, say, Islam, - Shia and Sunni, or between different socialists groups. We could hope though that rationalists are better in cooperation.
Distillation-amplification, if it works, should only start a war if the amplified human would have wanted that. Agent foundations theorists afaic predict that the first mover in AI has enough strategic advantage that there’ll be nothing worthy of the word war.
If the amplified human could take over the world but hasn’t because he’s not evil, and predicts that this other AI system would do such evil, yes.
It’s plausible, though, that the decision theory used by the new AI would tell it to act predictably non-evilly, in order to make the amplified human see this coming and not destroy the new AI before it’s turned on.
Note that this amplified human has thereby already taken over the world in all but name.
You are risking to oversaturate the AGI research market relative to other life-changing potential technologies, such as engineered life extension or organizational mechanism design.
Such oversaturation may result in two or more internally consistent but poorly compatible AI Safety theories (e.g. distillation-amplification vs. agent foundation), which will end up in two different friendly AI and the the war for correct friendliness will start.
If we have two distinct AI safety plans, the researchers are sensible to have a big discussion on which is better and only turn that one on. If not, and neither AI is fatally flawed, I would expect them to cooperate, they have very similar goals and neither wants war.
Historically, it didn’t work. A most bitter conflicts were between two main branches of, say, Islam, - Shia and Sunni, or between different socialists groups. We could hope though that rationalists are better in cooperation.
Distillation-amplification, if it works, should only start a war if the amplified human would have wanted that. Agent foundations theorists afaic predict that the first mover in AI has enough strategic advantage that there’ll be nothing worthy of the word war.
Ok, so should amplified human try to turn off first the agent-foundation-based project which is going to turn off this human if complete?
If the amplified human could take over the world but hasn’t because he’s not evil, and predicts that this other AI system would do such evil, yes.
It’s plausible, though, that the decision theory used by the new AI would tell it to act predictably non-evilly, in order to make the amplified human see this coming and not destroy the new AI before it’s turned on.
Note that this amplified human has thereby already taken over the world in all but name.