This is very impressive, and yet less impressive than it sounds. I used to watch a lot of StarCraft II (I stopped interacting with Blizzard recently because of how they rolled over for China), and over the summer there were many breakdowns of AlphaStar games once players figured out how to identify the accounts.
The impressive part is getting reinforcement learning to work at all in such a vast state space- that took breakthroughs beyond what was necessary to solve Go and beat Atari games. AlphaStar had to have a rich enough set of potential concepts (in the sense that e.g. a convolutional net ends up having concepts of different textures) that it could learn a concept like “construct building P” or “attack unit Q” or “stay out of the range of unit R” rather than just “select spot S and enter key T”. This is new and worth celebrating.
The overhyped part is that AlphaStar doesn’t really do the “strategy” part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don’t look creative or even especially reactive to opponent strategies.
That’s because there’s no representation of causal thinking—“if I did X then they could do Y, so I’d better do X’ instead”. Instead there are many agents evolving together, and if there’s an agent evolving to try Y then the agents doing X will be replaced with agents that do X’.
(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents’ units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn’t get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it destroyed one building and then immediately began rebuilding it before it could move the units out!)
This means that, first, AlphaStar just doesn’t have a decent response to strategies that it didn’t evolve, and secondly, it doesn’t do much in the way of a reactive decision tree of strategies (if I scout this, I do that). That kind of play is unfortunately very necessary for playing Zerg at a high level, so the internal meta has just collapsed into one where its Zerg agents predictably rush out early attacks that are easy to defend if expected. This has the flow-through effect that its Terran and Protoss are weaker against human Zerg than against other races, because they’ve never practiced against a solid Zerg that plays for the late game.
The end result cleaned up against weak players, performed well against good players, but practically never took a game against the top few players. I think that DeepMind realized they’d need another breakthrough to do what they did to Go, and decided to throw in the towel while making it look like they were claiming victory.
Finally, RL practitioners have known that genuine causal reasoning could never be achieved via known RL architectures- you’d only ever get something that could execute the same policy as an agent that had reasoned that way, via a very expensive process of evolving away from dominated strategies at each step down the tree of move and countermove. It’s the biggest known unknown on the way to AGI.
“I endorse endorsing X” is a sign of a really promising topic for therapy (or your preferred modality of psychological growth).
If I can simply say “X”, then I’m internally coherent enough on that point.
If I can only say “I endorse X”, then not-X is psychologically load-bearing for me, but often in a way that is opaque to my conscious reasoning, so working on that conflict can be slippery.
But if I can only say “I endorse endorsing X”, then not only is not-X load-bearing for me, but there’s a clear feeling of resistance to X that I can consciously hone in on, connect with, and learn about.
I’d understand this better (and perhaps even agree) if there were a few examples and a few counter-examples to find the boundaries of when this is effective.
For myself, without more words like “I endorse endorsing X under Y conditions because X is good for those who are hearing the endorsement and not necessarily for the endorser”, I don’t see how it works. The direct, unconditional form just makes me notice my dissonance and worry at it until I either endorse X or not-X (or neither—I’m allowed to be uncertain or ambivalent or just “context-dependent”).
Ah, I’m talking about introspection in a therapy context and not about exhorting others.
For example:
Internal coherence: “I forgive myself for doing that stupid thing”.
Load-bearing but opaque: “It makes sense to forgive myself, and I want to, but for some reason I just can’t”.
Load-bearing and clear resistance: “I want other people to forgive themselves for things like that, but when I think about forgiving myself, I get a big NOPE NOPE NOPE”.
P.S. Maybe forgiving oneself isn’t actually the right thing to do at the moment! But it will also be easier to learn that in the third case than in the second.
Has any serious AI Safety research org thought about situating themselves so that they could continue to function after a nuclear war?
Wait, hear me out.
A global thermonuclear war would set AI timelines back by at least a decade, for all of the obvious reasons. So an AI Safety org that survived would have additional precious years to work on the alignment problem, compared to orgs in the worlds where we avoid that war.
So it seems to me that at least one org with short timelines ought to move to New Zealand or at least move farther away from cities.
(Yes, I know MIRI was pondering leaving the Bay Area for underspecified reasons. I’d love to know what their thinking was regarding this effect, but I don’t expect they’d reveal it.)
[Cross-posted from Medium, written for a pretty general audience]
There are many words that could describe my political positions. But there’s one fundamental label for me: I am a consequentialist.
Consequentialism is a term from ethics; there, it means the position that consequences are what truly make an action right or wrong, rather than rules or virtues. What that means is that for me, the most essential questions about policy aren’t things like “what is fair” or “what rights do people have”, although these are good questions. For me, it all boils down to “how do we make people’s lives better?”
(There are some bits of nuance to the previous paragraph, which I’ve kept as a long endnote.)
“Make people’s lives better” isn’t a platitude- there’s a real difference here! To explain, I want to point out that there are both consequentialists and non-consequentialists within different political camps. Let’s consider socialists first and then libertarians second.
Many socialists believe both that (A) the world is headed for plutocratic disaster unless capitalism is overthrown, and that (B) labor markets and massive wealth disparities would be crimes even if they did not doom others to suffering. The difference is that some are more motivated by beliefs like (A), and could thus change their positions if convinced that e.g. the Nordic model was much better for future growth than a marketless society; while others are more motivated by beliefs like (B), and would continue to support pure socialism even if they were convinced it would mean catastrophe.
And many libertarians believe both that (A’) the only engine that can consistently create prosperity for all is a free market with no interference, and that (B’) taxation is a monstrous act of aggression and theft. The difference is that some are more motivated by beliefs like (A’), and thus could change their position if convinced that e.g. progressive taxation and redistribution would not destroy the incentives behind economic growth; while others are more motivated by beliefs like (B’), and would continue to support pure libertarianism even if they were convinced it would mean catastrophe.
I find it fruitful to talk with the first kind of socialist and the first kind of libertarian, but not the second kind of either. The second type just isn’t fundamentally interested in thinking about the consequences (except insofar as they can convince others by arguing for certain consequences). But among the first type, it’s possible to figure out the truth together by arguing about historical cases, studying natural experiments in policy, and articulating different theories.
I hope it’s been helpful to draw out this distinction; I’d encourage you to first find fellow consequentialists among your natural allies, and expand from there when and if you feel comfortable. There’s a lot that can be done to make the world a better place, and those of us who care most about making the world better can achieve more once we find each other!
P.S. The above focuses on the sort of political questions where most people’s influence is limited to voting and convincing others to vote with them. But there’s more ways to have an effect than that; I’d like to take one last moment to recommend the effective altruism movement, which investigates the best ways for people to have a big positive impact on the world.
---
Nuance section:
the position that consequences are what truly make an action right or wrong
There’s a naive version of this, which is that you should seize any good immediate outcome you can, even by doing horrific things. That’s… not a healthy version of consequentialism. The way to be less naive is to care about long-term consequences, and also to expect that you can’t get away with keeping your behavior secret from others in general. Here’s a good account of what non-naive consequentialism can look like.
the most essential questions about policy aren’t things like “what is fair” or “what rights do people have”, although these are good questions
In particular, fairness and rights are vital to making people’s lives better! We want more than just physical comforts; we want autonomy and achievement and meaning, we want to have trustworthy promises about what the world will ask of us tomorrow, and we want past injustices to be rectified. But these can be traded off, in extreme situations, against the other things that are important for people. In a massive emergency, I’d rather save lives in an unfair way and try to patch up the unfairness later, than let people die to preserve fairness.
how do we make people’s lives better?
This gets complicated and weird when you apply it to things like our distant descendants, but there are some aspects in the world today that seem fairly straightforward. Our world has built an engine of prosperity that makes food and goods available to many, beyond what was dreamt of in the past. But many people in the world are still living short and painful lives filled with disease and starvation. Another dollar of goods will do much more for one of them than for one of us. If we can improve their lives without destroying that engine, it is imperative to do that. (What consequentialists mostly disagree on is how the engine really works, how it could be destroyed, and how it could be improved!)
Virtue ethics seems less easily applicable to the domain of “what governmental policies to support” than to the domain of personal behavior, so I had a hard time thinking of examples. Can you?
On politics, virtue ethics might say: “try to have leaders that are good”*, “accepting bribes is wrong”, and perhaps “seek peace and shared ground rather than division and fear.” (Working towards peace seems more virtuous than fear mongering.)
*and if they’re not good, try and change that—gradual progress is better than no progress at all.
Is there already a concept handle for the notion of a Problem Where The Intuitive Solution Actually Makes It Worse But Makes You Want To Use Even More Dakka On It?
My most salient example is the way that political progressives in the Bay Area tried using restrictive zoning and rent control in order to prevent displacement… but this made for a housing shortage and made the existing housing stock skyrocket in value… which led to displacement happening by other (often cruel and/or backhanded) methods… which led to progressives concluding that their rules weren’t restrictive enough.
Another example is that treating a chunk of the population with contempt makes a good number of people in that chunk become even more opposed to you, which makes you want to show even more contempt for them, etc. (Which is not to say their ideas are correct or even worthy of serious consideration—but the people are always worthy of respect.)
That sort of dynamic is how you can get an absolutely fucked-up self-reinforcing situation, an inadequate quasi-equilibrium that’s not even a Nash equilibrium, but exists because at least one party is completely wrong about its incentives.
(And before you get cynical, of course there are disingenuous people whose preferences are perfectly well served in that quasi-equilibrium. But most activists do care about the outcomes, and would change their actions if they were genuinely convinced the outcomes would be different.)
More seriously, though, do you have any examples that aren’t based on the instinct-to-punish(reality, facts, people,...) that I ranted about in Curse of the Counterfactual? If they all fall in this category, one could call it an Argument With Reality, which is Byron Katie’s term for it. (You could also call it, “The Principle of the Thing”, an older and more colloquial term for people privileging the idea of a thing over the substance of the thing, usually to an irrational extent.)
When people are having an Argument With Reality, they:
Go for approaches that impose costs on some target(s), in preference to ones that are of benefit to anyone
Refuse to acknowledge other points of view except for how it proves those holding them to be the Bad Wrong Enemies
Double down as long as reality refuses to conform or insufficient Punishment has occurred (defined as the Bad Wrong Enemies surrendering and submitting or at least showing sufficiently-costly signals to that effect)
A lot of public policy is driven this way; Wars on Abstract Nouns are always more popular than rehabiliation, prevention, and other benefit-oriented policies, which will be denigrated as being too Soft On Abstract Nouns. (This also applies of course to non-governmental public policies, with much the same incentives for anybody in the public view to avoid becoming considered one of the Bad Wrong Enemies.)
In terms of naming / identifying this, do you think it would help to distinguish what makes you want to double down on the current solution? I can think of at least 3 reasons:
Not being aware that it’s making things worse
Knowing that it made things worse, but feeling like giving up on that tactic would make things get even worse instead of better
Being committed to the tactic more than to the outcome (what pjeby described as “The Principle of the Thing”) -- which could itself have multiple reasons, including emotionally-driven responses, duty-based reasoning, or explicitly believing that doubling down somehow leads to better outcomes in the long run.
Do these all fall within the phenomenon you’re trying to describe?
Eliezer wrote something about two types of definitions, one where you explain your criterion, and one where you point and say “things like that and that, but not that or that”. I thought it was called intensive vs extensive definition, but I can’t find the post I thought existed. Does anyone else remember this?
Decision-theoretic blackmail is when X gets Y to choose A over B, not via acting to make the consequences of A more appealing to Y, but by making the consequences of B less appealing to Y.
The exceptions to this definition are pretty massive, though, and I don’t know a principled emendation that excludes them.
1. There’s a contract / social contract / decision-theoretic equilibrium, and within that, B will be punished. (This may not be a true counterexample, because the true choice is whether to join the contract… though this is less clear for the social contract than for the other two.)
2. Precommitting not to give in to blackmail is not itself blackmail. Of course, in an ultimatum game both players can imagine themselves as doing this.
Can anyone think of more exceptions, or a redefinition that clearly excludes these?
In high-leverage situations, you should arguably either be playing tic-tac-toe (simple, legible, predictable responses) or playing 4-D chess to win. If you’re making really nonstandard and surprising moves (especially in PR), you have no excuse for winding up with a worse outcome than you would have if you’d acted in bog-standard normal ways.
(This doesn’t mean suspending your ethics! Those are part of winning! But if you can’t figure out how to win 4-D chess ethically, then you need to play an ethical tic-tac-toe strategy instead.)
DeepMind released their AlphaStar paper a few days ago, having reached Grandmaster level at the partial-information real-time strategy game StarCraft II over the summer.
This is very impressive, and yet less impressive than it sounds. I used to watch a lot of StarCraft II (I stopped interacting with Blizzard recently because of how they rolled over for China), and over the summer there were many breakdowns of AlphaStar games once players figured out how to identify the accounts.
The impressive part is getting reinforcement learning to work at all in such a vast state space- that took breakthroughs beyond what was necessary to solve Go and beat Atari games. AlphaStar had to have a rich enough set of potential concepts (in the sense that e.g. a convolutional net ends up having concepts of different textures) that it could learn a concept like “construct building P” or “attack unit Q” or “stay out of the range of unit R” rather than just “select spot S and enter key T”. This is new and worth celebrating.
The overhyped part is that AlphaStar doesn’t really do the “strategy” part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don’t look creative or even especially reactive to opponent strategies.
That’s because there’s no representation of causal thinking—“if I did X then they could do Y, so I’d better do X’ instead”. Instead there are many agents evolving together, and if there’s an agent evolving to try Y then the agents doing X will be replaced with agents that do X’.
(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents’ units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn’t get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it destroyed one building and then immediately began rebuilding it before it could move the units out!)
This means that, first, AlphaStar just doesn’t have a decent response to strategies that it didn’t evolve, and secondly, it doesn’t do much in the way of a reactive decision tree of strategies (if I scout this, I do that). That kind of play is unfortunately very necessary for playing Zerg at a high level, so the internal meta has just collapsed into one where its Zerg agents predictably rush out early attacks that are easy to defend if expected. This has the flow-through effect that its Terran and Protoss are weaker against human Zerg than against other races, because they’ve never practiced against a solid Zerg that plays for the late game.
The end result cleaned up against weak players, performed well against good players, but practically never took a game against the top few players. I think that DeepMind realized they’d need another breakthrough to do what they did to Go, and decided to throw in the towel while making it look like they were claiming victory.
Finally, RL practitioners have known that genuine causal reasoning could never be achieved via known RL architectures- you’d only ever get something that could execute the same policy as an agent that had reasoned that way, via a very expensive process of evolving away from dominated strategies at each step down the tree of move and countermove. It’s the biggest known unknown on the way to AGI.
This is the clearest and most insightful analysis of AlphaStar I’ve seen and IMO really should be a top-level post.
Thanks, will do.
“I endorse endorsing X” is a sign of a really promising topic for therapy (or your preferred modality of psychological growth).
If I can simply say “X”, then I’m internally coherent enough on that point.
If I can only say “I endorse X”, then not-X is psychologically load-bearing for me, but often in a way that is opaque to my conscious reasoning, so working on that conflict can be slippery.
But if I can only say “I endorse endorsing X”, then not only is not-X load-bearing for me, but there’s a clear feeling of resistance to X that I can consciously hone in on, connect with, and learn about.
I’d understand this better (and perhaps even agree) if there were a few examples and a few counter-examples to find the boundaries of when this is effective.
For myself, without more words like “I endorse endorsing X under Y conditions because X is good for those who are hearing the endorsement and not necessarily for the endorser”, I don’t see how it works. The direct, unconditional form just makes me notice my dissonance and worry at it until I either endorse X or not-X (or neither—I’m allowed to be uncertain or ambivalent or just “context-dependent”).
Ah, I’m talking about introspection in a therapy context and not about exhorting others.
For example:
Internal coherence: “I forgive myself for doing that stupid thing”.
Load-bearing but opaque: “It makes sense to forgive myself, and I want to, but for some reason I just can’t”.
Load-bearing and clear resistance: “I want other people to forgive themselves for things like that, but when I think about forgiving myself, I get a big NOPE NOPE NOPE”.
P.S. Maybe forgiving oneself isn’t actually the right thing to do at the moment! But it will also be easier to learn that in the third case than in the second.
Has any serious AI Safety research org thought about situating themselves so that they could continue to function after a nuclear war?
Wait, hear me out.
A global thermonuclear war would set AI timelines back by at least a decade, for all of the obvious reasons. So an AI Safety org that survived would have additional precious years to work on the alignment problem, compared to orgs in the worlds where we avoid that war.
So it seems to me that at least one org with short timelines ought to move to New Zealand or at least move farther away from cities.
(Yes, I know MIRI was pondering leaving the Bay Area for underspecified reasons. I’d love to know what their thinking was regarding this effect, but I don’t expect they’d reveal it.)
[Cross-posted from Medium, written for a pretty general audience]
There are many words that could describe my political positions. But there’s one fundamental label for me: I am a consequentialist.
Consequentialism is a term from ethics; there, it means the position that consequences are what truly make an action right or wrong, rather than rules or virtues. What that means is that for me, the most essential questions about policy aren’t things like “what is fair” or “what rights do people have”, although these are good questions. For me, it all boils down to “how do we make people’s lives better?”
(There are some bits of nuance to the previous paragraph, which I’ve kept as a long endnote.)
“Make people’s lives better” isn’t a platitude- there’s a real difference here! To explain, I want to point out that there are both consequentialists and non-consequentialists within different political camps. Let’s consider socialists first and then libertarians second.
Many socialists believe both that (A) the world is headed for plutocratic disaster unless capitalism is overthrown, and that (B) labor markets and massive wealth disparities would be crimes even if they did not doom others to suffering. The difference is that some are more motivated by beliefs like (A), and could thus change their positions if convinced that e.g. the Nordic model was much better for future growth than a marketless society; while others are more motivated by beliefs like (B), and would continue to support pure socialism even if they were convinced it would mean catastrophe.
And many libertarians believe both that (A’) the only engine that can consistently create prosperity for all is a free market with no interference, and that (B’) taxation is a monstrous act of aggression and theft. The difference is that some are more motivated by beliefs like (A’), and thus could change their position if convinced that e.g. progressive taxation and redistribution would not destroy the incentives behind economic growth; while others are more motivated by beliefs like (B’), and would continue to support pure libertarianism even if they were convinced it would mean catastrophe.
I find it fruitful to talk with the first kind of socialist and the first kind of libertarian, but not the second kind of either. The second type just isn’t fundamentally interested in thinking about the consequences (except insofar as they can convince others by arguing for certain consequences). But among the first type, it’s possible to figure out the truth together by arguing about historical cases, studying natural experiments in policy, and articulating different theories.
I hope it’s been helpful to draw out this distinction; I’d encourage you to first find fellow consequentialists among your natural allies, and expand from there when and if you feel comfortable. There’s a lot that can be done to make the world a better place, and those of us who care most about making the world better can achieve more once we find each other!
P.S. The above focuses on the sort of political questions where most people’s influence is limited to voting and convincing others to vote with them. But there’s more ways to have an effect than that; I’d like to take one last moment to recommend the effective altruism movement, which investigates the best ways for people to have a big positive impact on the world.
---
Nuance section:
There’s a naive version of this, which is that you should seize any good immediate outcome you can, even by doing horrific things. That’s… not a healthy version of consequentialism. The way to be less naive is to care about long-term consequences, and also to expect that you can’t get away with keeping your behavior secret from others in general. Here’s a good account of what non-naive consequentialism can look like.
In particular, fairness and rights are vital to making people’s lives better! We want more than just physical comforts; we want autonomy and achievement and meaning, we want to have trustworthy promises about what the world will ask of us tomorrow, and we want past injustices to be rectified. But these can be traded off, in extreme situations, against the other things that are important for people. In a massive emergency, I’d rather save lives in an unfair way and try to patch up the unfairness later, than let people die to preserve fairness.
This gets complicated and weird when you apply it to things like our distant descendants, but there are some aspects in the world today that seem fairly straightforward. Our world has built an engine of prosperity that makes food and goods available to many, beyond what was dreamt of in the past. But many people in the world are still living short and painful lives filled with disease and starvation. Another dollar of goods will do much more for one of them than for one of us. If we can improve their lives without destroying that engine, it is imperative to do that. (What consequentialists mostly disagree on is how the engine really works, how it could be destroyed, and how it could be improved!)
It seems to me that your examples of B are mostly deontological, so it would be nice to have some C which represented virtue ethics as well.
Virtue ethics seems less easily applicable to the domain of “what governmental policies to support” than to the domain of personal behavior, so I had a hard time thinking of examples. Can you?
On politics, virtue ethics might say: “try to have leaders that are good”*, “accepting bribes is wrong”, and perhaps “seek peace and shared ground rather than division and fear.” (Working towards peace seems more virtuous than fear mongering.)
*and if they’re not good, try and change that—gradual progress is better than no progress at all.
Is there already a concept handle for the notion of a Problem Where The Intuitive Solution Actually Makes It Worse But Makes You Want To Use Even More Dakka On It?
My most salient example is the way that political progressives in the Bay Area tried using restrictive zoning and rent control in order to prevent displacement… but this made for a housing shortage and made the existing housing stock skyrocket in value… which led to displacement happening by other (often cruel and/or backhanded) methods… which led to progressives concluding that their rules weren’t restrictive enough.
Another example is that treating a chunk of the population with contempt makes a good number of people in that chunk become even more opposed to you, which makes you want to show even more contempt for them, etc. (Which is not to say their ideas are correct or even worthy of serious consideration—but the people are always worthy of respect.)
That sort of dynamic is how you can get an absolutely fucked-up self-reinforcing situation, an inadequate quasi-equilibrium that’s not even a Nash equilibrium, but exists because at least one party is completely wrong about its incentives.
(And before you get cynical, of course there are disingenuous people whose preferences are perfectly well served in that quasi-equilibrium. But most activists do care about the outcomes, and would change their actions if they were genuinely convinced the outcomes would be different.)
“The Human Condition”? ;-)
More seriously, though, do you have any examples that aren’t based on the instinct-to-punish(reality, facts, people,...) that I ranted about in Curse of the Counterfactual? If they all fall in this category, one could call it an Argument With Reality, which is Byron Katie’s term for it. (You could also call it, “The Principle of the Thing”, an older and more colloquial term for people privileging the idea of a thing over the substance of the thing, usually to an irrational extent.)
When people are having an Argument With Reality, they:
Go for approaches that impose costs on some target(s), in preference to ones that are of benefit to anyone
Refuse to acknowledge other points of view except for how it proves those holding them to be the Bad Wrong Enemies
Double down as long as reality refuses to conform or insufficient Punishment has occurred (defined as the Bad Wrong Enemies surrendering and submitting or at least showing sufficiently-costly signals to that effect)
A lot of public policy is driven this way; Wars on Abstract Nouns are always more popular than rehabiliation, prevention, and other benefit-oriented policies, which will be denigrated as being too Soft On Abstract Nouns. (This also applies of course to non-governmental public policies, with much the same incentives for anybody in the public view to avoid becoming considered one of the Bad Wrong Enemies.)
In terms of naming / identifying this, do you think it would help to distinguish what makes you want to double down on the current solution? I can think of at least 3 reasons:
Not being aware that it’s making things worse
Knowing that it made things worse, but feeling like giving up on that tactic would make things get even worse instead of better
Being committed to the tactic more than to the outcome (what pjeby described as “The Principle of the Thing”) -- which could itself have multiple reasons, including emotionally-driven responses, duty-based reasoning, or explicitly believing that doubling down somehow leads to better outcomes in the long run.
Do these all fall within the phenomenon you’re trying to describe?
Thanks for drawing distinctions—I mean #1 only.
[EDIT: found it. Extensional vs intensional.]
Eliezer wrote something about two types of definitions, one where you explain your criterion, and one where you point and say “things like that and that, but not that or that”. I thought it was called intensive vs extensive definition, but I can’t find the post I thought existed. Does anyone else remember this?
Some authors use ostensive to mean the same thing as “extensional.”
Is there a word for problems where, as they get worse, the exactly wrong response becomes more intuitively appealing?
For example, I’m thinking of the following chain (sorry for a political example, this is typically a political phenomenon):
resistance to new construction (using the ability of local boards to block projects)
causes skyrocketing rent
which together mean that the rare properties allowed to be developed get bid up to where they can only become high-end housing
which leads to anger at rich developers for building “luxury housing”
which leads to further resistance to new construction
and so on until you get San Francisco
You probably already know that, but a subset of what you described is called “positive feedback loop”.
Decision-theoretic blackmail is when X gets Y to choose A over B, not via acting to make the consequences of A more appealing to Y, but by making the consequences of B less appealing to Y.
The exceptions to this definition are pretty massive, though, and I don’t know a principled emendation that excludes them.
1. There’s a contract / social contract / decision-theoretic equilibrium, and within that, B will be punished. (This may not be a true counterexample, because the true choice is whether to join the contract… though this is less clear for the social contract than for the other two.)
2. Precommitting not to give in to blackmail is not itself blackmail. Of course, in an ultimatum game both players can imagine themselves as doing this.
Can anyone think of more exceptions, or a redefinition that clearly excludes these?
In high-leverage situations, you should arguably either be playing tic-tac-toe (simple, legible, predictable responses) or playing 4-D chess to win. If you’re making really nonstandard and surprising moves (especially in PR), you have no excuse for winding up with a worse outcome than you would have if you’d acted in bog-standard normal ways.
(This doesn’t mean suspending your ethics! Those are part of winning! But if you can’t figure out how to win 4-D chess ethically, then you need to play an ethical tic-tac-toe strategy instead.)