Is the Sun King (who doesn’t particularly desire pointless mass murder) more moral than I am? Much harder, and your articulation of “weak Friendliness” seems incapable of even trying to answer. And that doesn’t even get into actual moral problems society actually faces every day (i.e. what is the most moral taxation scheme?).
If weak-FAI can’t solve those types of problems, or even suggest useful directions to look, why should we believe it is a step on the path to strong-FAI?
That’s my point. I’m not sure where the confusion is, here. Why would you call it useless to prevent wireheading, UFAI, and nuclear winter, just because it can’t also do your taxes?
If it’s easier to solve the big problems first, wouldn’t we want to do that? And then afterwards we can take our sweet time figuring out abortion and gay marriage and tax codes, because a failure there doesn’t end the species.
For reasons related to Hidden Complexity of Wishes, I don’t think weak-FAI actually is likely to prevent “wireheading, UFAI, and nuclear winter.” At best, it prohibits the most obvious implementations of those problems. And it is terribly unlikely to be helpful in creating strong-FAI.
And your original claim was that common human preferences already implement weak-FAI preferences. I think that the more likely reason why we haven;t had the disasters you reference is that for most of human history, we lacked the capacity to cause those problems. As actual society shows, hidden complexity of wishes make implementing social consensus hopeless, much less whatever smaller set of preferences is weak-FAI preferences.
As actual society shows, hidden complexity of wishes make implementing social consensus hopeless
My basic point was that we shouldn’t worry about politics, at least not yet, because politics is a wonderful example of all the hard questions in CEV, and we haven’t even worked out the easy questions like how to prevent nuclear winter. My second point was that humans do seem to have a much clearer CEV when it comes to “prevent nuclear winter”, even if it’s still not unanimous.
Implicit in that should have been the idea that CEV is still ridiculously difficult. Just like intelligence, it’s something humans seem to have and use despite being unable to program for it.
So, then, summarized, I’m saying that we should perhaps work out the easy problems first, before we go throwing ourselves against harder problems like politics.
There’s not a clear dividing line between “easy” moral questions and hard moral questions. The Cold War, which massively increased the risk of nuclear winter, was a rational expression of Great Power relations between two powers.
Until we have mutually acceptable ways of resolving disputes when both parties are rationally protecting their interests, we can’t actually solve the easy problems either.
Mass murder is an easy question.
Is the Sun King (who doesn’t particularly desire pointless mass murder) more moral than I am? Much harder, and your articulation of “weak Friendliness” seems incapable of even trying to answer. And that doesn’t even get into actual moral problems society actually faces every day (i.e. what is the most moral taxation scheme?).
If weak-FAI can’t solve those types of problems, or even suggest useful directions to look, why should we believe it is a step on the path to strong-FAI?
That’s my point. I’m not sure where the confusion is, here. Why would you call it useless to prevent wireheading, UFAI, and nuclear winter, just because it can’t also do your taxes?
If it’s easier to solve the big problems first, wouldn’t we want to do that? And then afterwards we can take our sweet time figuring out abortion and gay marriage and tax codes, because a failure there doesn’t end the species.
For reasons related to Hidden Complexity of Wishes, I don’t think weak-FAI actually is likely to prevent “wireheading, UFAI, and nuclear winter.” At best, it prohibits the most obvious implementations of those problems. And it is terribly unlikely to be helpful in creating strong-FAI.
And your original claim was that common human preferences already implement weak-FAI preferences. I think that the more likely reason why we haven;t had the disasters you reference is that for most of human history, we lacked the capacity to cause those problems. As actual society shows, hidden complexity of wishes make implementing social consensus hopeless, much less whatever smaller set of preferences is weak-FAI preferences.
My basic point was that we shouldn’t worry about politics, at least not yet, because politics is a wonderful example of all the hard questions in CEV, and we haven’t even worked out the easy questions like how to prevent nuclear winter. My second point was that humans do seem to have a much clearer CEV when it comes to “prevent nuclear winter”, even if it’s still not unanimous.
Implicit in that should have been the idea that CEV is still ridiculously difficult. Just like intelligence, it’s something humans seem to have and use despite being unable to program for it.
So, then, summarized, I’m saying that we should perhaps work out the easy problems first, before we go throwing ourselves against harder problems like politics.
There’s not a clear dividing line between “easy” moral questions and hard moral questions. The Cold War, which massively increased the risk of nuclear winter, was a rational expression of Great Power relations between two powers.
Until we have mutually acceptable ways of resolving disputes when both parties are rationally protecting their interests, we can’t actually solve the easy problems either.
from you:
and from me:
So, um, we agree, huzzah? :)