I dispute this. The SIAI FAI is specifically designed to have control of the universe as one of its goals.
It is widely expected that this will arise as an important instrumental goal; nothing more than that. I can’t tell if this is what you mean. (When you point out that “trying to take over the universe isn’t utility-maximizing under many circumstances”, it sounds like you’re thinking of taking over the universe as a separate terminal goal, which would indeed be terrible design; an AI without that terminal goal, that can reason the same way you can, can decide not to try to take over the universe if that looks best.)
I notice that some of my comment wars with other people arise because they automatically assume that whenever we’re talking about a superintelligence, there’s only one of them. This is in danger of becoming a LW communal assumption. It’s not even likely.
I probably missed it in some other comment, but which of these do you not buy: (a) huge first-mover advantages from self-improvement (b) preventing other superintelligences as a convergent subgoal (c) that the conjunction of these implies that a singleton superintelligence is likely?
(More generally, there’s a strong tendency for people on LW to attribute very high likelihoods to scenarios that EY spends a lot of time talking about—even if he doesn’t insist that they are likely.)
This sounds plausible and bad. Can you think of some other examples?
It is widely expected that this will arise as an important instrumental goal; nothing more than that. I can’t tell if this is what you mean. (When you point out that “trying to take over the universe isn’t utility-maximizing under many circumstances”, it sounds like you’re thinking of taking over the universe as a separate terminal goal, which would indeed be terrible design; an AI without that terminal goal, that can reason the same way you can, can decide not to try to take over the universe if that looks best.)
I probably missed it in some other comment, but which of these do you not buy: (a) huge first-mover advantages from self-improvement (b) preventing other superintelligences as a convergent subgoal (c) that the conjunction of these implies that a singleton superintelligence is likely?
This sounds plausible and bad. Can you think of some other examples?