The flaw in both arguments that jumps out at me is that you’re both assuming that you can predict how a superintelligent entity will think, value, and act without being superintelligent yourselves. That’s like a wolf (or a tiger, or a bear, or a killer whale) predicting whether a human will pounce on a squirrel and eat it alive solely on criteria such as how tasty squirrel meat is, how fast and agile the squirrel is, whether it is healthy or sick, or how hungry the human is. Variables such as empathy and a moral aversion to eating a creature while its still alive, lack of a need to kill things in order to subsist, appreciation of the beauty and cuteness of squirrels, or concern about viral or bacterial infection from eating raw meat are unfathomable and hence invisible to most predators. These variables are far more significant than how fast, agile, or tasty the squirrel is, but none of these variables will occur to a wolf or a tiger. We have no more idea how an ASI thinks than a turtle does of how we think.
The flaw in both arguments that jumps out at me is that you’re both assuming that you can predict how a superintelligent entity will think, value, and act without being superintelligent yourselves. That’s like a wolf (or a tiger, or a bear, or a killer whale) predicting whether a human will pounce on a squirrel and eat it alive solely on criteria such as how tasty squirrel meat is, how fast and agile the squirrel is, whether it is healthy or sick, or how hungry the human is. Variables such as empathy and a moral aversion to eating a creature while its still alive, lack of a need to kill things in order to subsist, appreciation of the beauty and cuteness of squirrels, or concern about viral or bacterial infection from eating raw meat are unfathomable and hence invisible to most predators. These variables are far more significant than how fast, agile, or tasty the squirrel is, but none of these variables will occur to a wolf or a tiger. We have no more idea how an ASI thinks than a turtle does of how we think.