I think that too much investment could result in more noise in the field.
First of all because it will result in large number of published materials, which could exceed capacity of other researchers to read it. In result really interesting works will be not read.
It will also attract in the field more people than actually clever and dedicated people exist. If we have 100 trained ai safety reserchers, which is overestimation , and we hire 1000 people, than real reasesrchers will be dissolved.
In some fields like nanotech overinvestment result even in expel of original reaserchers because they prevent less educated ones to spent money as they want.
But most dangerous thing is creating of many incomparable theories of friendliness, and even AIs based on them which would result in AI wars and extinction.
Yeah, I read Eliezer’s chapter “Artificial Intelligence as a Positive and Negative Factor in Global Risk” in Global Catastrophic Risks, and it was impressed with how far in advance he anticipated reactions to the rising popularity of AI safety, what it might be like when the public finally switched from skepticism to genuine concern, and what it might start to look like. Eliezer has also anticipated even safety-conscious work on AI might increase AI risk.
The idea some existing institutions in AI safety, perhaps MIRI, should expand much faster than others so it can keep up with all the published material coming out, and evaluate it, is neglected.
But most dangerous thing is creating of many incomparable theories of friendliness, and even AIs based on them which would result in AI wars and extinction.
I strongly disagree.
First, because there are multiple reasons that the creation of many distinct theories of friendliness would not be dangerous: The first one to get to superintelligence should be able to establish a monopoly on power, and then we wouldn’t have to worry about the others. Even if that didn’t happen, a reasonable decision theory should be able to cooperate with other agents with different reasonable decision theories, when it is in both of their interests to do so. And even if we end up with multiple friendly AIs that are not great at cooperation, it is a particularly easy problem to cooperate with agents that have similar goals (as is implied by all of them being friendly). And even if we end up with a “friendly AI” that is incapable of establishing a monopoly on power but that will cause a great deal of destruction when another similarly capable but differently designed agent comes into existence, even if both agents have broadly similar goals (I would not call this a successful friendly AI), convincing people not to create such AIs does not actually get much easier if the people planning to create the AI have not been thinking about how to make it friendly, so preventing people from developing different theories of friendliness still doesn’t help.
But beyond all that, I would also say that not creating many incomparable theories of friendliness is dangerous. If there is only one that anyone is working on, it will likely be misguided, and by the time anyone notices, enough time may have been wasted that friendliness will have lost too much ground in the race against general AI.
Of course I meant not only creating, but implementing of different friendliness theories. Your objection about cooperation based on good decision theories also seems to be sound.
But from history we know that christian countries had wars between them, and socialist countries also had mutual wars. Sometimes small difference leaded to sectarian violence, like between shia and sunni. So adopting a value system which promote future good for everybody doesn’t prevent a state-agent to have wars this another agent with simillar positive value system.
For example we have two FAIs, and they both know that for the best one of them should be switched off. But how they would decide about which one will be switched off?
Also FAI may work fine until creation of the another AI, but could have instrumental value to switched off all other AIs and not be switched off by any AI, and they have to go to war because of this flaw in design which only appear if we have two FAIs.
Just pointing out I upvoted Turchin’s comment above, but I agree with your clarification above here, of the last part of his comment. Nothing I’ve read thus far raises concern about warring superintelligences.
I wonder if MIRI’s General Staff or Advisors deal with issues like this.
Your last point was interesting. I tried making a few, narrow comparisons with other fields that are important to people emotionally and physically i.e. cancer research and poverty charities. Upon a cursory glance, things like quacks, deceit and falsification seem present in these areas. So I suppose stuff like that’s possible in AI safety.
Though I guess the people involved in AI safety would try much harder to lock out people like that or publicly challange people who have no clue what they’re saying. However, its possible that some group might emerge that promotes shaky ideas which gain traction.
Though I think the scrutiny of those in the field and their judgements would cut down things like that.
I think that too much investment could result in more noise in the field. First of all because it will result in large number of published materials, which could exceed capacity of other researchers to read it. In result really interesting works will be not read. It will also attract in the field more people than actually clever and dedicated people exist. If we have 100 trained ai safety reserchers, which is overestimation , and we hire 1000 people, than real reasesrchers will be dissolved. In some fields like nanotech overinvestment result even in expel of original reaserchers because they prevent less educated ones to spent money as they want. But most dangerous thing is creating of many incomparable theories of friendliness, and even AIs based on them which would result in AI wars and extinction.
Yeah, I read Eliezer’s chapter “Artificial Intelligence as a Positive and Negative Factor in Global Risk” in Global Catastrophic Risks, and it was impressed with how far in advance he anticipated reactions to the rising popularity of AI safety, what it might be like when the public finally switched from skepticism to genuine concern, and what it might start to look like. Eliezer has also anticipated even safety-conscious work on AI might increase AI risk.
The idea some existing institutions in AI safety, perhaps MIRI, should expand much faster than others so it can keep up with all the published material coming out, and evaluate it, is neglected.
I strongly disagree.
First, because there are multiple reasons that the creation of many distinct theories of friendliness would not be dangerous: The first one to get to superintelligence should be able to establish a monopoly on power, and then we wouldn’t have to worry about the others. Even if that didn’t happen, a reasonable decision theory should be able to cooperate with other agents with different reasonable decision theories, when it is in both of their interests to do so. And even if we end up with multiple friendly AIs that are not great at cooperation, it is a particularly easy problem to cooperate with agents that have similar goals (as is implied by all of them being friendly). And even if we end up with a “friendly AI” that is incapable of establishing a monopoly on power but that will cause a great deal of destruction when another similarly capable but differently designed agent comes into existence, even if both agents have broadly similar goals (I would not call this a successful friendly AI), convincing people not to create such AIs does not actually get much easier if the people planning to create the AI have not been thinking about how to make it friendly, so preventing people from developing different theories of friendliness still doesn’t help.
But beyond all that, I would also say that not creating many incomparable theories of friendliness is dangerous. If there is only one that anyone is working on, it will likely be misguided, and by the time anyone notices, enough time may have been wasted that friendliness will have lost too much ground in the race against general AI.
Of course I meant not only creating, but implementing of different friendliness theories. Your objection about cooperation based on good decision theories also seems to be sound.
But from history we know that christian countries had wars between them, and socialist countries also had mutual wars. Sometimes small difference leaded to sectarian violence, like between shia and sunni. So adopting a value system which promote future good for everybody doesn’t prevent a state-agent to have wars this another agent with simillar positive value system.
For example we have two FAIs, and they both know that for the best one of them should be switched off. But how they would decide about which one will be switched off?
Also FAI may work fine until creation of the another AI, but could have instrumental value to switched off all other AIs and not be switched off by any AI, and they have to go to war because of this flaw in design which only appear if we have two FAIs.
Just pointing out I upvoted Turchin’s comment above, but I agree with your clarification above here, of the last part of his comment. Nothing I’ve read thus far raises concern about warring superintelligences.
I wonder if MIRI’s General Staff or Advisors deal with issues like this.
Your last point was interesting. I tried making a few, narrow comparisons with other fields that are important to people emotionally and physically i.e. cancer research and poverty charities. Upon a cursory glance, things like quacks, deceit and falsification seem present in these areas. So I suppose stuff like that’s possible in AI safety.
Though I guess the people involved in AI safety would try much harder to lock out people like that or publicly challange people who have no clue what they’re saying. However, its possible that some group might emerge that promotes shaky ideas which gain traction.
Though I think the scrutiny of those in the field and their judgements would cut down things like that.
By the way if OpenAI were suggested before Musk, it would likely be regarded as such shaky idea.
Many people do regard OpenAI as a shaky idea.
Do you mean the whole field of AI would regard OpenAI as a shaky idea before Musk, or just safety-conscious AI researchers?
I was speaking about safety researchers.
In that case, yeah, it’s still shaky, albeit less so than if Musk wasn’t involved.