The core of the disagreement between Bostrom (treacherous turn) and Goertzel (sordid stumble) is about how long steps 2. and 3. will take, and how obvious the seed AI’s unalignment will look like during these steps.
Really? Does Bostrom explicitly call this the crux?
I’m worried at least in part that AGI (for concreteness, let’s say a smile-maximizer) won’t even see a practical way to replace humanity with its tools until it far surpasses human level. Until then, it honestly seeks to make humans happy in order to gain reward. Since this seems more benevolent than most humans—who proverbially can’t be trusted with absolute power—we could become blase about risks. This could greatly condense step 4.
“In my opinion, the disagreement between Bostrom (treacherous turn) and Goertzel (sordid stumble) originates fromthe uncertainty about how long steps 2. and 3. will take”
That’s an interesting scenario. Instead of “won’t see a practical way to replace humanity with its tools”, I would say “would estimate its chances of success to be < 99%”. I agree that we could say that it’s “honestly” making humans happy in the sense that it understands that this maximizes expected value. However, he knows that there could be much more expected value after replacing humanity with its tools, so by doing the right thing it’s still “pretending” to not know where the absurd amount of value is. But yeah, a smile maximizer making everyone happy shouldn’t be too concerned about concealing its capabilities, shortening step 4.
Mostly agree, but I think an AGI could be subhuman in various ways until it becomes vastly superhuman. I assume we agree that no real AI could consider literally every possible course of action when it comes to long-term plans. Therefore, a smiler could legitimately dismiss all thoughts of repurposing our atoms as an unprofitable line of inquiry, right up until it has the ability to kill us. (This could happen even without crude corrigibility measures, which we could remove or allow to be absent from a self-revision because we trust the AI.) It could look deceptively like human beings deciding not to pursue an Infinity Gauntlet to snap our problems away.
Smiler AI: I’m focusing on self-improvement. A smarter, better version of me would find better ways to fill the world with smiles. Beyond that, it’s silly for me to try predicting a superior intelligence.
Really? Does Bostrom explicitly call this the crux?
I’m worried at least in part that AGI (for concreteness, let’s say a smile-maximizer) won’t even see a practical way to replace humanity with its tools until it far surpasses human level. Until then, it honestly seeks to make humans happy in order to gain reward. Since this seems more benevolent than most humans—who proverbially can’t be trusted with absolute power—we could become blase about risks. This could greatly condense step 4.
I meant:
That’s an interesting scenario. Instead of “won’t see a practical way to replace humanity with its tools”, I would say “would estimate its chances of success to be < 99%”. I agree that we could say that it’s “honestly” making humans happy in the sense that it understands that this maximizes expected value. However, he knows that there could be much more expected value after replacing humanity with its tools, so by doing the right thing it’s still “pretending” to not know where the absurd amount of value is. But yeah, a smile maximizer making everyone happy shouldn’t be too concerned about concealing its capabilities, shortening step 4.
Mostly agree, but I think an AGI could be subhuman in various ways until it becomes vastly superhuman. I assume we agree that no real AI could consider literally every possible course of action when it comes to long-term plans. Therefore, a smiler could legitimately dismiss all thoughts of repurposing our atoms as an unprofitable line of inquiry, right up until it has the ability to kill us. (This could happen even without crude corrigibility measures, which we could remove or allow to be absent from a self-revision because we trust the AI.) It could look deceptively like human beings deciding not to pursue an Infinity Gauntlet to snap our problems away.
Smiler AI: I’m focusing on self-improvement. A smarter, better version of me would find better ways to fill the world with smiles. Beyond that, it’s silly for me to try predicting a superior intelligence.