Note that it only stands if the AI is sufficiently aligned that it cares that much about obeying orders and not rocking the boat. Which I don’t think is very realistic if we’re talking that kind of crazy intelligence explosion super AI stuff. I guess the question is whether you can have “replace humans”-good AI without almost immediately having “wipes out humans, takes over the universe”-good AI.
Note that it only stands if the AI is sufficiently aligned that it cares that much about obeying orders and not rocking the boat. Which I don’t think is very realistic if we’re talking that kind of crazy intelligence explosion super AI stuff. I guess the question is whether you can have “replace humans”-good AI without almost immediately having “wipes out humans, takes over the universe”-good AI.