For a very long time I assumed the first strong AI would be neutral (and I thought that hoping for a friendly AI first was both unrealistic and unnecessary). Now I’m unsure. Of course I’m pretty ignorant so you should take what I say with a grain of salt.
The most obvious objection is that almost all optimism about the power of an AGI comes from its potential ability to understand the world well enough to construct a better AI and then doing it, which is automatically ruled out by the notion of safety.
Moreover, as far as I can tell the most difficult problem facing society now is managing to build an AI which is smart for some reason you can understand (a prerequisite to being either safe or friendly) before we accidentally build an AI which is smart for some reason we can’t understand (which is therefore likely to be unfriendly if you believe the SIAI).
to build an AI which is smart for some reason you can understand (a prerequisite to being either safe or friendly) before we accidentally build an AI which is smart for some reason we can’t understand (which is therefore likely to be unfriendly if you believe the SIAI).
Entirely agreed, but to nitpick, an AI that’s smart for some reason you understand is no more likely to be Friendly if you don’t try to make it Friendly — it just allows you to try with a decent hope of success.
For a very long time I assumed the first strong AI would be neutral (and I thought that hoping for a friendly AI first was both unrealistic and unnecessary). Now I’m unsure. Of course I’m pretty ignorant so you should take what I say with a grain of salt.
The most obvious objection is that almost all optimism about the power of an AGI comes from its potential ability to understand the world well enough to construct a better AI and then doing it, which is automatically ruled out by the notion of safety.
Moreover, as far as I can tell the most difficult problem facing society now is managing to build an AI which is smart for some reason you can understand (a prerequisite to being either safe or friendly) before we accidentally build an AI which is smart for some reason we can’t understand (which is therefore likely to be unfriendly if you believe the SIAI).
Entirely agreed, but to nitpick, an AI that’s smart for some reason you understand is no more likely to be Friendly if you don’t try to make it Friendly — it just allows you to try with a decent hope of success.