This seemed to be a major theme of the OP—see the discussions of “extremal Goodhart”, and “the tails come apart”—so I’m confused that you don’t seem to see that as very central.
I agree that a large part of Joe’s post was about the idea that human values diverge in the limit. But I think if you take the thing he wrote about human selfishness seriously, then it is perfectly reasonable to talk about the ordinary cases of value divergence too, which I think are very common. Joe wrote,
And we can worry about the human-human case for more mundane reasons, too. Thus, for example, it’s often thought that a substantial part of what’s going on with human values is either selfish or quite “partial.” That is, many humans want pleasure, status, flourishing, etc for themselves, and then also for their family, local community, and so on. We can posit that this aspect of human values will disappear or constrain itself on reflection, or that it will “saturate” to the point where more impartial and cosmopolitan values start to dominate in practice – but see above re: “convenient and substantive empirical hypothesis” (and if “saturation” helps with extremal-Goodhart problems, can you make the AI’s values saturate, too?).
I mean, I’m quite sure that it’s false, as an empirical claim about the normal human world, that the normal things Alice chooses to do, will tend to make a random different person Bob worse-off, on-average, as judged by Bob himself, including upon reflection.
I think you’re potentially mixing up two separate claims that have different implications. I’m not saying that people rarely act in such a way that makes random strangers better off. In addition to my belief in a small yet clearly real altruistic element to human nature, there’s the obvious fact that the world is not zero-sum, and people routinely engage in mutually beneficial actions that make both parties in the interaction better off.
I am claiming that people are mostly selfish, and that the majority of economic and political behavior seems to be the result of people acting in their own selfish interests, rather than mostly out of the kindness of their heart. (Although by selfishness I’m including concern for one’s family and friends; I’m simply excluding concern for total strangers.)
That is exactly Adam Smith’s point: it is literally in the self-interest of the baker for them to sell us dinner. Even if the baker were completely selfish, they’d still sell us food. Put yourself in their shoes. If you were entirely selfish and had no regard for the preferences of other people, wouldn’t you still try to obey the law and engage in trade with other people?
You said that, “if the butcher and brewer and everyone else were selfish to the point of being sociopathic, it would be a catastrophe.” But I live in the world where the average person donates very little to charity. I’m already living in a world like the one you are describing; it’s simply less extreme along this axis. In such a world, maybe charitable donations go from being 3% of GDP to being 0% of GDP, but presumably we’d still have to obey laws and trade with each other to get our dinner, because those things are mainly social mechanisms we use to coordinate our mostly selfish values.
The economy (and world) relies on some minimal level of trust between employees, coworkers, business partners and so on, trust that they will generally follow norms and act with a modicum of integrity, even when nobody is looking.
Trust is also valuable selfishly, if you can earn it from others, or if you care about not being deceived by others yourself. Again, put yourself in the shoes of a sociopath: is it selfishly profitable, in expectation, to go around and commit a lot of fraud? Maybe if you can get away with it with certainty. But most of the time, people can’t be certain they’ll get away with it, and the consequences of getting caught are quite severe.
Is it selfishly profitable to reject the norm punishing fraudsters? Maybe if you can ensure this rejection won’t come back to hurt you. But I don’t think that’s something you can always ensure. The world seems much richer and better off, even from your perspective, if we have laws against theft, fraud, and murder.
It is true in a literal sense that selfish people have no incentive to tell the truth about things that nobody will ever find out about. But the world is a repeated game. Many important truths about our social world are things that will at some point be exposed. If you want to have power, there’s a lot of stuff you should not lie about, because it will hurt you, even selfishly, in the eyes of other (mostly selfish) people.
Selfishness doesn’t mean stupidity. Going around flagrantly violating norms, stealing from people, and violating everyone’s trust doesn’t actually increase your own selfish utility. It hurts you, because people will be less likely to want to deal with and trade with you in the future. It also usually helps you to uphold these norms, even as a selfish person, because like everyone else, you don’t want to be the victim of fraud either.
I agree that a large part of Joe’s post was about the idea that human values diverge in the limit. But I think if you take the thing he wrote about human selfishness seriously, then it is perfectly reasonable to talk about the ordinary cases of value divergence too, which I think are very common. Joe wrote,
I think you’re potentially mixing up two separate claims that have different implications. I’m not saying that people rarely act in such a way that makes random strangers better off. In addition to my belief in a small yet clearly real altruistic element to human nature, there’s the obvious fact that the world is not zero-sum, and people routinely engage in mutually beneficial actions that make both parties in the interaction better off.
I am claiming that people are mostly selfish, and that the majority of economic and political behavior seems to be the result of people acting in their own selfish interests, rather than mostly out of the kindness of their heart. (Although by selfishness I’m including concern for one’s family and friends; I’m simply excluding concern for total strangers.)
That is exactly Adam Smith’s point: it is literally in the self-interest of the baker for them to sell us dinner. Even if the baker were completely selfish, they’d still sell us food. Put yourself in their shoes. If you were entirely selfish and had no regard for the preferences of other people, wouldn’t you still try to obey the law and engage in trade with other people?
You said that, “if the butcher and brewer and everyone else were selfish to the point of being sociopathic, it would be a catastrophe.” But I live in the world where the average person donates very little to charity. I’m already living in a world like the one you are describing; it’s simply less extreme along this axis. In such a world, maybe charitable donations go from being 3% of GDP to being 0% of GDP, but presumably we’d still have to obey laws and trade with each other to get our dinner, because those things are mainly social mechanisms we use to coordinate our mostly selfish values.
Trust is also valuable selfishly, if you can earn it from others, or if you care about not being deceived by others yourself. Again, put yourself in the shoes of a sociopath: is it selfishly profitable, in expectation, to go around and commit a lot of fraud? Maybe if you can get away with it with certainty. But most of the time, people can’t be certain they’ll get away with it, and the consequences of getting caught are quite severe.
Is it selfishly profitable to reject the norm punishing fraudsters? Maybe if you can ensure this rejection won’t come back to hurt you. But I don’t think that’s something you can always ensure. The world seems much richer and better off, even from your perspective, if we have laws against theft, fraud, and murder.
It is true in a literal sense that selfish people have no incentive to tell the truth about things that nobody will ever find out about. But the world is a repeated game. Many important truths about our social world are things that will at some point be exposed. If you want to have power, there’s a lot of stuff you should not lie about, because it will hurt you, even selfishly, in the eyes of other (mostly selfish) people.
Selfishness doesn’t mean stupidity. Going around flagrantly violating norms, stealing from people, and violating everyone’s trust doesn’t actually increase your own selfish utility. It hurts you, because people will be less likely to want to deal with and trade with you in the future. It also usually helps you to uphold these norms, even as a selfish person, because like everyone else, you don’t want to be the victim of fraud either.