But most dangerous thing is creating of many incomparable theories of friendliness, and even AIs based on them which would result in AI wars and extinction.
I strongly disagree.
First, because there are multiple reasons that the creation of many distinct theories of friendliness would not be dangerous: The first one to get to superintelligence should be able to establish a monopoly on power, and then we wouldn’t have to worry about the others. Even if that didn’t happen, a reasonable decision theory should be able to cooperate with other agents with different reasonable decision theories, when it is in both of their interests to do so. And even if we end up with multiple friendly AIs that are not great at cooperation, it is a particularly easy problem to cooperate with agents that have similar goals (as is implied by all of them being friendly). And even if we end up with a “friendly AI” that is incapable of establishing a monopoly on power but that will cause a great deal of destruction when another similarly capable but differently designed agent comes into existence, even if both agents have broadly similar goals (I would not call this a successful friendly AI), convincing people not to create such AIs does not actually get much easier if the people planning to create the AI have not been thinking about how to make it friendly, so preventing people from developing different theories of friendliness still doesn’t help.
But beyond all that, I would also say that not creating many incomparable theories of friendliness is dangerous. If there is only one that anyone is working on, it will likely be misguided, and by the time anyone notices, enough time may have been wasted that friendliness will have lost too much ground in the race against general AI.
Of course I meant not only creating, but implementing of different friendliness theories. Your objection about cooperation based on good decision theories also seems to be sound.
But from history we know that christian countries had wars between them, and socialist countries also had mutual wars. Sometimes small difference leaded to sectarian violence, like between shia and sunni. So adopting a value system which promote future good for everybody doesn’t prevent a state-agent to have wars this another agent with simillar positive value system.
For example we have two FAIs, and they both know that for the best one of them should be switched off. But how they would decide about which one will be switched off?
Also FAI may work fine until creation of the another AI, but could have instrumental value to switched off all other AIs and not be switched off by any AI, and they have to go to war because of this flaw in design which only appear if we have two FAIs.
Just pointing out I upvoted Turchin’s comment above, but I agree with your clarification above here, of the last part of his comment. Nothing I’ve read thus far raises concern about warring superintelligences.
I strongly disagree.
First, because there are multiple reasons that the creation of many distinct theories of friendliness would not be dangerous: The first one to get to superintelligence should be able to establish a monopoly on power, and then we wouldn’t have to worry about the others. Even if that didn’t happen, a reasonable decision theory should be able to cooperate with other agents with different reasonable decision theories, when it is in both of their interests to do so. And even if we end up with multiple friendly AIs that are not great at cooperation, it is a particularly easy problem to cooperate with agents that have similar goals (as is implied by all of them being friendly). And even if we end up with a “friendly AI” that is incapable of establishing a monopoly on power but that will cause a great deal of destruction when another similarly capable but differently designed agent comes into existence, even if both agents have broadly similar goals (I would not call this a successful friendly AI), convincing people not to create such AIs does not actually get much easier if the people planning to create the AI have not been thinking about how to make it friendly, so preventing people from developing different theories of friendliness still doesn’t help.
But beyond all that, I would also say that not creating many incomparable theories of friendliness is dangerous. If there is only one that anyone is working on, it will likely be misguided, and by the time anyone notices, enough time may have been wasted that friendliness will have lost too much ground in the race against general AI.
Of course I meant not only creating, but implementing of different friendliness theories. Your objection about cooperation based on good decision theories also seems to be sound.
But from history we know that christian countries had wars between them, and socialist countries also had mutual wars. Sometimes small difference leaded to sectarian violence, like between shia and sunni. So adopting a value system which promote future good for everybody doesn’t prevent a state-agent to have wars this another agent with simillar positive value system.
For example we have two FAIs, and they both know that for the best one of them should be switched off. But how they would decide about which one will be switched off?
Also FAI may work fine until creation of the another AI, but could have instrumental value to switched off all other AIs and not be switched off by any AI, and they have to go to war because of this flaw in design which only appear if we have two FAIs.
Just pointing out I upvoted Turchin’s comment above, but I agree with your clarification above here, of the last part of his comment. Nothing I’ve read thus far raises concern about warring superintelligences.