“what can we do to prevent some small group of humans (the SIAI, a secret conspiracy of billionaires, a secret conspiracy of Google employees, whoever) from steering a first-mover scenario in a direction that’s beneficial to themselves and perhaps their blood relatives, but harmful to the rest of humanity?”
Actually, if they managed to do that, then they have managed to build an FAI. The large(/largest ?) risk some (as SIAI I think, but I am not an expert) is that they think they are building an FAI (or perhaps a too weak AI to be really dangerous) but that they are misstaken in that assumption. In reality they have been building an uFAI that takes over the world and humanity as a whole is doomed, including a small minority of humanity that possibly the AI was supposed to be friendly to.
There seems to be three different problems here. To analyse how dangerous AIs in general are. If dangerous, how can one make an FAI, that is an AGI that is at least beneficial to some. And then if an FAI can be built, to whom should it be friendly. As I interprete your post you are discussing the third question and dangers related to that while hypothetically assuming that the small group building the AGI has managed to solve the second question? If so, you are not really discussing why some would build an uFAI half-way by purpose but why some would build an FAI that is unfriendly to most humans?
That’s not how I understood the “on the whole, beneficial to humans and humanity.” It would benefit some humans, but it wouldn’t fulfill the “on the whole” part of the quoted definition of Friendly AI.
That does, though, highlight some of the confusions that seem to surround the term “Friendly AI.”
“what can we do to prevent some small group of humans (the SIAI, a secret conspiracy of billionaires, a secret conspiracy of Google employees, whoever) from steering a first-mover scenario in a direction that’s beneficial to themselves and perhaps their blood relatives, but harmful to the rest of humanity?”
Actually, if they managed to do that, then they have managed to build an FAI. The large(/largest ?) risk some (as SIAI I think, but I am not an expert) is that they think they are building an FAI (or perhaps a too weak AI to be really dangerous) but that they are misstaken in that assumption. In reality they have been building an uFAI that takes over the world and humanity as a whole is doomed, including a small minority of humanity that possibly the AI was supposed to be friendly to.
There seems to be three different problems here. To analyse how dangerous AIs in general are. If dangerous, how can one make an FAI, that is an AGI that is at least beneficial to some. And then if an FAI can be built, to whom should it be friendly. As I interprete your post you are discussing the third question and dangers related to that while hypothetically assuming that the small group building the AGI has managed to solve the second question? If so, you are not really discussing why some would build an uFAI half-way by purpose but why some would build an FAI that is unfriendly to most humans?
Abandoning the 99% may fail the “beneficial to humans” test.
That’s not how I understood the “on the whole, beneficial to humans and humanity.” It would benefit some humans, but it wouldn’t fulfill the “on the whole” part of the quoted definition of Friendly AI.
That does, though, highlight some of the confusions that seem to surround the term “Friendly AI.”