Just a little remark: before “friendly AI” becomes a problem, there’s the usual one: “friendly humans”.
Let’s take the standard assumption:
Evolution is about evolution of Intelligence. Needs our last animal/wetware stage only to develop its “in silico” successor.
Then the conclusion is quite obvious:
We are the danger. Mostly because due to our stupidity we can easily destroy ourselves before advancing to the next level.
It’s quite a simplecstory: animals got technology from another, higher civilization—this smells deadly.
Memetic parallel civilization of a tiny intelligent but powerless fraction of per cent is giving the humanimal “society” (for free and without any say/power over its usage) more and more advanced power based on sci/tech.
Enormous conflict coming from the discrepancy between these advances and animalistic roots of the “society” (ruled by the usual predator/prey jungle-system—call it king/slaves, employer/employees or politician/taxpayers) is clear.
Already nuclear bombs were beyond the limit. Now imagine cheap, almost homemade drone-transported nuclear grenade—coming nanotech can be much worse, as we all know.
Now slip just for a while into the boots of the upcoming Super AGI.
Giving this techpower to the bunch of predatory lunatics (read: governments) often drugged by stupid ideologies? Or even general public?
Are you kidding?
We are much bigger existential risk for the Superintelligence than the AGI is for us (and if it’s really THE Superintelligence, then the humans do not matter anymore, anyway).
Just a little remark: before “friendly AI” becomes a problem, there’s the usual one: “friendly humans”.
Let’s take the standard assumption:
Evolution is about evolution of Intelligence. Needs our last animal/wetware stage only to develop its “in silico” successor.
Then the conclusion is quite obvious:
We are the danger. Mostly because due to our stupidity we can easily destroy ourselves before advancing to the next level.
It’s quite a simplecstory: animals got technology from another, higher civilization—this smells deadly.
Memetic parallel civilization of a tiny intelligent but powerless fraction of per cent is giving the humanimal “society” (for free and without any say/power over its usage) more and more advanced power based on sci/tech.
Enormous conflict coming from the discrepancy between these advances and animalistic roots of the “society” (ruled by the usual predator/prey jungle-system—call it king/slaves, employer/employees or politician/taxpayers) is clear.
Already nuclear bombs were beyond the limit. Now imagine cheap, almost homemade drone-transported nuclear grenade—coming nanotech can be much worse, as we all know.
Now slip just for a while into the boots of the upcoming Super AGI.
Giving this techpower to the bunch of predatory lunatics (read: governments) often drugged by stupid ideologies? Or even general public?
Are you kidding?
We are much bigger existential risk for the Superintelligence than the AGI is for us (and if it’s really THE Superintelligence, then the humans do not matter anymore, anyway).