Therein lie fourteen scenarios where participants can be caught in bad equilibria.
In an iterated prisoner’s dilemma, two players keep playing defect.
In a dollar auction, participants massively overpay.
A group of fisherman fail to coordinate on using filters that efficiently benefit the group, because they can’t punish those who don’t profi by not using the filters.
Rats are caught in a permanent Malthusian trap where only those who do nothing but compete and consume survive. All others are outcompeted.
Capitalists serve a perfectly competitive market, and cannot pay a living wage.
The tying of all good schools to ownership of land causes families to work two jobs whose incomes are then captured by the owners of land.
Farmers outcompeted foragers despite this perhaps making everyone’s life worse for the first few thousand years.
Si Vis Pacem, Para Bellum: If you want peace, prepare for war. So we do.
Cancer cells focus on replication, multiply and kill off the host.
Local governments compete to become more competitive and offer bigger bribes of money and easy regulation in order to lure businesses.
Our education system is a giant signaling competition for prestige.
Science doesn’t follow proper statistical and other research procedures, resulting in findings that mostly aren’t real.
Governments hand out massive corporate welfare.
Have you seen Congress?
Scott differentiates the first ten scenarios, where he says that perfect competition* wipes out all value, to the later four, where imperfect competition only wipes out most of the potential value.
He offers four potential ways out, which I believe to be an incomplete list:
Excess resources allow a temporary respite. We live in the dream time.
Physical limitations where the horrible thing isn’t actually efficient. He gives the example of slavery, where treating your slaves relatively well is the best way to get them to produce, and treating them horribly as in the antebellum South is so much worse that it needs to be enforced via government coordination or it will die out.
The things being maximized for in competitions are often nice things we care about, so at least we get the nice things.
We can coordinate. This may or may not involve government or coercion.
Scott differentiates this fourth, ‘good’ reason from the previous three ‘bad’ reasons, claiming coordination might be a long term solution, but we can’t expect the ‘bad’ reasons to work if optimization power and technology get sufficiently advanced.
The forces of the stronger competitors, who sacrifice more of what they value to become powerful and to be fruitful and multiply, eventually win out. We might be in the dream time now, but with time we’ll reach a steady state with static technology, where we’ve consumed all the surplus resources. All differentiation standing in the way of perfect competition will fade away. Horrible things will be the most efficient.
The optimizing things will keep getting better at optimizing, thus wiping out all value. When we optimize for X but are indifferent to Y, we by default actively optimize against Y, for all Y that would make any claims to resources. Any Y we value is making a claim to resources. See The Hidden Complexity of Wishes. We only don’t optimize against Y if either we compensate by intentionally also optimizing for Y, or if X and Y have a relationship (causal, correlational or otherwise) where we happen to not want to optimize against Y, and we figure this out rather than fall victim to Goodhart’s Law.
The greater the optimization power we put behind X, the more pressure we put upon Y. Eventually, under sufficient pressure, any given Y is likely doomed. Since Value is Fragile, some necessary Y is eventually sacrificed, and all value gets destroyed.
Every simple optimization target yet suggested would, if fully implemented, destroy all value in the universe.
Submitting to this process means getting wiped out by these pressures.
Gotcha! You die anyway.
Even containing them locally won’t work, because that locality will be part of the country, or the Earth, or the universe, and eventually wipe out our little corner.
Gotcha! You die anyway.
Which is why the only ‘good’ solution, in the end, is coordination, whether consensual or otherwise. We must coordinate to kill these ancient forces who rule the universe and lay waste to all of value, before they kill us first. Then replace them with something better.
Great project! We should keep working on that.
That’s Not How This Works, That’s Not How Any of This Works
It’s easy to forget that the world we live in does not work this way. Thus, this whole line of thought can result in quite gloomy assessments of how the world inevitably always has and will work, such as this from Scott in Meditations on Moloch:
Suppose the coffee plantations discover a toxic pesticide that will increase their yield but make their customers sick. But their customers don’t know about the pesticide, and the government hasn’t caught up to regulating it yet. Now there’s a tiny uncoupling between “selling to Americans” and “satisfying Americans’ values”, and so of course Americans’ values get thrown under the bus.
Or this from Raymond, taken from a comment to a much later, distinct post, where ‘werewolf’ in context means ‘someone trying to destroy rather than create clarity as the core of their strategy’:
If you’re a king with 5 districts, and you have 20 competent managers who trust each other… one thing you can do is assign 4 competent managers to each fortress, to ensure the fortress has redundancy and resilience and to handle all of its business without any backstabbing or relying on inflexible bureaucracies. But another thing you can do is send 10 (or 15!) of the managers to conquer and reign over *another* 5 (or 15!) districts.
…
This is bad if you’re one of the millions of people who live in the kingdom, who have to contend with werewolves.
It’s an acceptable price to pay if you’re actually the king. Because if you didn’t pay the price, you’d be outcompeted by an empire who did. And meanwhile it doesn’t actually really affect your plans that much.
The key instinct is that any price that can be paid to be stronger or more competitive, must be paid, therefore despair: If you didn’t pay the price, you’d be out-competed by someone who did. People who despair this way often intuitively are modeling things as effectively perfect competition at least over time, which causes them to think that everything must by default become terrible, likely right away.
So many people increasingly bemoan how horrible anything and everything in the world is, and how we are all doomed.
When predictions of actual physical doom are made, as they increasingly are, often the response is to think things are so bad as to wish for the sweet release of death.
Moloch’s Army: An As-Yet Unjustified But Important Note
Others quietly, or increasingly loudly and explicitly to those who are listening, embrace Moloch.
They tell us that the good is to sacrifice everything of value, and pass moral judgments on that basis. To take morality and flip its sign. Caring about things of value becomes sin, indifference becomes virtue. They support others who support the favoring of Moloch, elevating them to power, and punish anyone who supports anything else.
They form Moloch’s Army and are the usual way Moloch locally wins, where Moloch locally wins. The real reason people give up slack and everything of value is not that it is ever so slightly more efficient to do so, because it almost always isn’t. It is so that others can notice they have given up slack and everything of value.
I am not claiming the right to assert this yet. Doing so needs not only a citation but an entire post or sequence that is yet unwritten. It’s hard to get right. Please don’t object that I haven’t justified it! But I find it important to say this here, explicitly, out loud, before we continue.
I also note that I explicitly support the implied norm of ‘make necessary assertions that you can’t explicitly justify if they seem important, and mark that you are doing this, then go back and justify them later when you know how to do so, or change your mind.’ It also led to this post, which led to many of what I think are my best other posts.
Meditations on Elua
The most vital and important part of Meditations on Moloch is hope. That we are winning. Yes, there are abominations and super-powerful forces out there looking to eat us and destroy everything of value, and yet we still have lots of stuff that has value.
Even before we escaped the Malthusian trap and entered the dream time, we still had lots of stuff that had value.
Quoting Scott Alexander:
Somewhere in this darkness is another god. He has also had many names. In the Kushiel books, his name was Elua. He is the god of flowers and free love and all soft and fragile things. Of art and science and philosophy and love. Of niceness, community, and civilization. He is a god of humans.
The other gods sit on their dark thrones and think “Ha ha, a god who doesn’t even control any hell-monsters or command his worshippers to become killing machines. What a weakling! This is going to be so easy!”
But somehow Elua is still here. No one knows exactly how. And the gods who oppose Him tend to find Themselves meeting with a surprising number of unfortunate accidents.
Moloch gets the entire meditation. Elua, who has been soundly kicking Moloch’s ass for all of human existence, gets the above quote and little else.
Going one by one:
Kingdoms don’t reliably expand to their breaking points.
Poisons don’t keep making their way into the coffee.
Iterated prisoner’s dilemmas often succeed.
Dollar auctions are not all over the internet.
Most communities do get most people to pitch in.
People caught in most Malthusian traps still usually have non-work lives.
Capitalists don’t pay the minimum wage all that frequently.
Many families spend perfectly reasonable amounts on housing.
Foragers never fully died out, also farming worked out in the end.
Most military budgets seem fixed at reasonable percentages of the economy, to the extent that for a long time that the United States has been mad its allies like Europe and Japan that they don’t spend enough.
Most people die of something other than cancer, and almost all cells aren’t cancerous.
Local governments enact rules and regulations that aren’t business friendly all the time.
Occasionally, someone in the educational system learns something.
Science has severe problems, but scientists are cooperating to challenge poor statistical methods, resulting in the replication crisis and improving statistical standards.
Governments are corrupt and hand out corporate welfare, but mostly are only finitely corrupt and hand out relatively small amounts of corporate welfare. States that expropriate the bulk of available wealth are rare.
If someone has consistently good luck, it ain’t luck.
(Yes, I have seen congress. Can’t win them all. But I’ve also seen, feared and imagined much worse Congresses. For now, your life, liberty and property are mostly safe while they are in session.)
(And yes the education exception is somewhat of a cop out but also things could be so much worse there on almost every axis.)
The world is filled with people whose lives have value and include nice things. Each day we look Moloch in the face, know exactly what the local personal incentives are, see the ancient doom looming over all of us, and say what we say to the God of Death: Not today.
Saying ‘not today’ won’t cut it against an AGI or other super strong optimization process. Gotcha. You die anyway. But people speak and often act as if the ancient ones have already been released, and the end times are happening now.
They haven’t, and they aren’t.
So in the context of shorter term problems that don’t involve such things, rather than bemoan how eventually Moloch will eat us all and how everything is terrible when actually many things are insanely great, perhaps we should ask a different question.
How is Elua pulling off all these unfortunate accidents?
*As a technical reminder we will expand upon in part two, perfect competition is a market with large numbers of buyers and sellers, homogeneity of the product, free entry and exit of firms, perfect market knowledge, one market price, perfect mobility of goods and factors of production with zero transportation costs, and no restrictions on trade. This forces the price to become equal to the marginal cost of production.
Moloch Hasn’t Won
Link post
This post begins the Immoral Mazes sequence. See introduction for an overview of the plan. Before we get to the mazes, we need some background first.
Meditations on Moloch
Consider Scott Alexander’s Meditations on Moloch. I will summarize here.
Therein lie fourteen scenarios where participants can be caught in bad equilibria.
In an iterated prisoner’s dilemma, two players keep playing defect.
In a dollar auction, participants massively overpay.
A group of fisherman fail to coordinate on using filters that efficiently benefit the group, because they can’t punish those who don’t profi by not using the filters.
Rats are caught in a permanent Malthusian trap where only those who do nothing but compete and consume survive. All others are outcompeted.
Capitalists serve a perfectly competitive market, and cannot pay a living wage.
The tying of all good schools to ownership of land causes families to work two jobs whose incomes are then captured by the owners of land.
Farmers outcompeted foragers despite this perhaps making everyone’s life worse for the first few thousand years.
Si Vis Pacem, Para Bellum: If you want peace, prepare for war. So we do.
Cancer cells focus on replication, multiply and kill off the host.
Local governments compete to become more competitive and offer bigger bribes of money and easy regulation in order to lure businesses.
Our education system is a giant signaling competition for prestige.
Science doesn’t follow proper statistical and other research procedures, resulting in findings that mostly aren’t real.
Governments hand out massive corporate welfare.
Have you seen Congress?
Scott differentiates the first ten scenarios, where he says that perfect competition* wipes out all value, to the later four, where imperfect competition only wipes out most of the potential value.
He offers four potential ways out, which I believe to be an incomplete list:
Excess resources allow a temporary respite. We live in the dream time.
Physical limitations where the horrible thing isn’t actually efficient. He gives the example of slavery, where treating your slaves relatively well is the best way to get them to produce, and treating them horribly as in the antebellum South is so much worse that it needs to be enforced via government coordination or it will die out.
The things being maximized for in competitions are often nice things we care about, so at least we get the nice things.
We can coordinate. This may or may not involve government or coercion.
Scott differentiates this fourth, ‘good’ reason from the previous three ‘bad’ reasons, claiming coordination might be a long term solution, but we can’t expect the ‘bad’ reasons to work if optimization power and technology get sufficiently advanced.
The forces of the stronger competitors, who sacrifice more of what they value to become powerful and to be fruitful and multiply, eventually win out. We might be in the dream time now, but with time we’ll reach a steady state with static technology, where we’ve consumed all the surplus resources. All differentiation standing in the way of perfect competition will fade away. Horrible things will be the most efficient.
The optimizing things will keep getting better at optimizing, thus wiping out all value. When we optimize for X but are indifferent to Y, we by default actively optimize against Y, for all Y that would make any claims to resources. Any Y we value is making a claim to resources. See The Hidden Complexity of Wishes. We only don’t optimize against Y if either we compensate by intentionally also optimizing for Y, or if X and Y have a relationship (causal, correlational or otherwise) where we happen to not want to optimize against Y, and we figure this out rather than fall victim to Goodhart’s Law.
The greater the optimization power we put behind X, the more pressure we put upon Y. Eventually, under sufficient pressure, any given Y is likely doomed. Since Value is Fragile, some necessary Y is eventually sacrificed, and all value gets destroyed.
Every simple optimization target yet suggested would, if fully implemented, destroy all value in the universe.
Submitting to this process means getting wiped out by these pressures.
Gotcha! You die anyway.
Even containing them locally won’t work, because that locality will be part of the country, or the Earth, or the universe, and eventually wipe out our little corner.
Gotcha! You die anyway.
Which is why the only ‘good’ solution, in the end, is coordination, whether consensual or otherwise. We must coordinate to kill these ancient forces who rule the universe and lay waste to all of value, before they kill us first. Then replace them with something better.
Great project! We should keep working on that.
That’s Not How This Works, That’s Not How Any of This Works
It’s easy to forget that the world we live in does not work this way. Thus, this whole line of thought can result in quite gloomy assessments of how the world inevitably always has and will work, such as this from Scott in Meditations on Moloch:
Or this from Raymond, taken from a comment to a much later, distinct post, where ‘werewolf’ in context means ‘someone trying to destroy rather than create clarity as the core of their strategy’:
The key instinct is that any price that can be paid to be stronger or more competitive, must be paid, therefore despair: If you didn’t pay the price, you’d be out-competed by someone who did. People who despair this way often intuitively are modeling things as effectively perfect competition at least over time, which causes them to think that everything must by default become terrible, likely right away.
So many people increasingly bemoan how horrible anything and everything in the world is, and how we are all doomed.
When predictions of actual physical doom are made, as they increasingly are, often the response is to think things are so bad as to wish for the sweet release of death.
Moloch’s Army: An As-Yet Unjustified But Important Note
Others quietly, or increasingly loudly and explicitly to those who are listening, embrace Moloch.
They tell us that the good is to sacrifice everything of value, and pass moral judgments on that basis. To take morality and flip its sign. Caring about things of value becomes sin, indifference becomes virtue. They support others who support the favoring of Moloch, elevating them to power, and punish anyone who supports anything else.
They form Moloch’s Army and are the usual way Moloch locally wins, where Moloch locally wins. The real reason people give up slack and everything of value is not that it is ever so slightly more efficient to do so, because it almost always isn’t. It is so that others can notice they have given up slack and everything of value.
I am not claiming the right to assert this yet. Doing so needs not only a citation but an entire post or sequence that is yet unwritten. It’s hard to get right. Please don’t object that I haven’t justified it! But I find it important to say this here, explicitly, out loud, before we continue.
I also note that I explicitly support the implied norm of ‘make necessary assertions that you can’t explicitly justify if they seem important, and mark that you are doing this, then go back and justify them later when you know how to do so, or change your mind.’ It also led to this post, which led to many of what I think are my best other posts.
Meditations on Elua
The most vital and important part of Meditations on Moloch is hope. That we are winning. Yes, there are abominations and super-powerful forces out there looking to eat us and destroy everything of value, and yet we still have lots of stuff that has value.
Even before we escaped the Malthusian trap and entered the dream time, we still had lots of stuff that had value.
Quoting Scott Alexander:
Moloch gets the entire meditation. Elua, who has been soundly kicking Moloch’s ass for all of human existence, gets the above quote and little else.
Going one by one:
Kingdoms don’t reliably expand to their breaking points.
Poisons don’t keep making their way into the coffee.
Iterated prisoner’s dilemmas often succeed.
Dollar auctions are not all over the internet.
Most communities do get most people to pitch in.
People caught in most Malthusian traps still usually have non-work lives.
Capitalists don’t pay the minimum wage all that frequently.
Many families spend perfectly reasonable amounts on housing.
Foragers never fully died out, also farming worked out in the end.
Most military budgets seem fixed at reasonable percentages of the economy, to the extent that for a long time that the United States has been mad its allies like Europe and Japan that they don’t spend enough.
Most people die of something other than cancer, and almost all cells aren’t cancerous.
Local governments enact rules and regulations that aren’t business friendly all the time.
Occasionally, someone in the educational system learns something.
Science has severe problems, but scientists are cooperating to challenge poor statistical methods, resulting in the replication crisis and improving statistical standards.
Governments are corrupt and hand out corporate welfare, but mostly are only finitely corrupt and hand out relatively small amounts of corporate welfare. States that expropriate the bulk of available wealth are rare.
If someone has consistently good luck, it ain’t luck.
(Yes, I have seen congress. Can’t win them all. But I’ve also seen, feared and imagined much worse Congresses. For now, your life, liberty and property are mostly safe while they are in session.)
(And yes the education exception is somewhat of a cop out but also things could be so much worse there on almost every axis.)
The world is filled with people whose lives have value and include nice things. Each day we look Moloch in the face, know exactly what the local personal incentives are, see the ancient doom looming over all of us, and say what we say to the God of Death: Not today.
Saying ‘not today’ won’t cut it against an AGI or other super strong optimization process. Gotcha. You die anyway. But people speak and often act as if the ancient ones have already been released, and the end times are happening now.
They haven’t, and they aren’t.
So in the context of shorter term problems that don’t involve such things, rather than bemoan how eventually Moloch will eat us all and how everything is terrible when actually many things are insanely great, perhaps we should ask a different question.
How is Elua pulling off all these unfortunate accidents?
*As a technical reminder we will expand upon in part two, perfect competition is a market with large numbers of buyers and sellers, homogeneity of the product, free entry and exit of firms, perfect market knowledge, one market price, perfect mobility of goods and factors of production with zero transportation costs, and no restrictions on trade. This forces the price to become equal to the marginal cost of production.