Constant: Corruptibility is a complex evolutionary adaptation. Even the best humans have hard-to-suppress semiconscious selfish motivations that tend to come out when we have power, even if we thought we were gaining that power for idealistic reasons. There’s no reason an AI with an intelligently designed, clean, transparent goal system would be corrupted, or need to be kept in check. “Respect for rights” is similarly anthropomorphic. Creating a society of AIs would be very problematic due to the strong first-mover effect, and the likely outcome amounts to extinction anyway.
Richard: Of course Eliezer should follow the rock; by stipulation, it states exactly those insights he would have if he were perfectly informed, rational, etc. This is nothing like “morality as the will of God”, since any such rock would have a causal dependency on his brain. It’s not clear to me that he’s saying everyone else should as well. Also by stipulation (AFAICS), his volition will be able to convince him through honest argument of any of its moral positions, regardless of how twisted he might be. (I share Marcello’s concern here, though.)
Caledonian: Eliezer is obviously not saying there is no point to looking for convincing arguments regarding ‘morality’, but that there are no arguments that will sway all minds, and there don’t need to be any for morality to be meaningful. You’re being ridiculous. (And how do you define ‘rational mind’?)
inquiringmind: I know Caledonian’s IP address. He’s not Robin.
Constant: Corruptibility is a complex evolutionary adaptation. Even the best humans have hard-to-suppress semiconscious selfish motivations that tend to come out when we have power, even if we thought we were gaining that power for idealistic reasons. There’s no reason an AI with an intelligently designed, clean, transparent goal system would be corrupted, or need to be kept in check. “Respect for rights” is similarly anthropomorphic. Creating a society of AIs would be very problematic due to the strong first-mover effect, and the likely outcome amounts to extinction anyway.
Richard: Of course Eliezer should follow the rock; by stipulation, it states exactly those insights he would have if he were perfectly informed, rational, etc. This is nothing like “morality as the will of God”, since any such rock would have a causal dependency on his brain. It’s not clear to me that he’s saying everyone else should as well. Also by stipulation (AFAICS), his volition will be able to convince him through honest argument of any of its moral positions, regardless of how twisted he might be. (I share Marcello’s concern here, though.)
Caledonian: Eliezer is obviously not saying there is no point to looking for convincing arguments regarding ‘morality’, but that there are no arguments that will sway all minds, and there don’t need to be any for morality to be meaningful. You’re being ridiculous. (And how do you define ‘rational mind’?)
inquiringmind: I know Caledonian’s IP address. He’s not Robin.