On the personal level, to me this seems like a potential failure mode worth worrying about in an AGI context because that’s Impossible Mode for everything, but not in practical human mode, and most definitely not on the margin. I’m not claiming I have the best possible answer here, I’m claiming that what humans are current doing is some mix of absurdly stupid things and non-FDT proposals that exist seem way worse than FDT proposals that exist—also that actual human attempts to be FDT-style agents will be incomplete and not lead to degenerate outcomes, the same way mostly-basically-CDT-style human agents often don’t actually do the fully crazy things it implies when it implies fully crazy things.
On the global level, again I’m not saying I know the details of how we should respond, only that we shouldn’t lay down and let him get whatever he wants.
I think it is essentially epsilon chance that Putin would choose nuclear firestorm over a world where Russia doesn’t recreate the USSR / Russian Empire (and even where he dies tomorrow as well) if those were 100% the choices (I do think he might well be willing to risk a non-trivial chance of nuclear war to get it, but that is different), but let’s say that it is 10% (and notice that in those 10%, if he knew for a fact we’d respond with our nukes, he’d just say ‘oh that’s too bad’ and destroy the world in a fit of pique, which very much doesn’t seem right). In the other 90%, he backs down after various numbers of escalations (e.g. in some he tries the escalate-to-deescalate single nuke, others he tries leveling Kyiv, others he folds tomorrow and leaves) in some combination, then folds after he sees we won’t fold, but losses are ‘acceptable’ here.
In those 10% of worlds, what are we hoping for? I don’t think there is much of an ‘eventually’ here.
He takes Ukraine, we let him. He sees we let him do what he wants. Everyone else sees too. Every state starts a nuclear weapons program that can afford one, so Putin knows he’s on limited time. Ukraine starts an insurgency and Putin starts killing A LOT of people in response. We do nothing. Moldavia is next pretty much right away. That falls in days. He then goes for Kazakhstan. Within a year he has all the non-NATO USSR republics in hand.
Meanwhile, Xi launches an invasion of Taiwan. Putin makes it clear that if the USA interferes he’ll back China up. We fold. China takes it. TSMC is destroyed. We lose the majority of our chip capacity, either to China or the void. Everyone knows our commitments mean nothing. Every country with a score to settle acts now.
Now Putin takes part of Estonia, and fortifies it. This is, let’s say, December 2022, and Trump-backed Republicans just swept the midterms during an extreme recession. What do we do? Again, clearly, nothing.
NATO is dead. No one believes us at all, anywhere. Putin invades and takes the Baltics. Six months later, he’s in Warsaw and Bucharest, perhaps without firing a shot. North Korea marches south and points its ICBMs.
And then it gets worse.
Or, more likely, at some point in that story we DO confront him, and the nuclear war happens anyway—and there’s a lot of worlds where he didn’t want that, but by folding so often, we let him think he could do it, so he doesn’t know where to stop, and then we get into a nuclear war over Estonia or whatever and someone miscalculates.
And that scenario happens 50%+ of the time rather than 10%, because there’s a ton of worlds where Putin/Xi/etc will take advantage like that, but where they very much would have folded if challenged. The chances of nukes flying in the medium (10 year) term go up, not down.
Even the best case scenarios I can imagine in such places are ones I very much do not like.
(Also, I think that if Putin tried to start a nuclear war without any attacks on Russia itself, that there is a >50% (although of course not terribly reassuring) chance that the answer would be ‘no’ and a substantial chance Russia’s nuclear weapons mostly no longer work and are a bluff whether or not Putin knows they don’t work—I very much would not launch ours until post-impact in the hopes this was the case, given our second strike capabilities. And there’s the worlds in which Putin tries to launch one nuke, it’s a dud, or his people turn on him, and that’s the end of all of it.)
If we lived by the rules you’re suggesting in the past, also, we wouldn’t have gotten this far—we would have folded to the USSR as nation after nation turned red, and at best we’d be in a much poorer, less free world (assuming it’s nuke-specific, and we still fight WW2).
I’m not claiming I have the best possible answer here, I’m claiming that what humans are current doing is some mix of absurdly stupid things
What are some examples of these?
I think it is essentially epsilon chance that Putin would choose nuclear firestorm over a world where Russia doesn’t recreate the USSR / Russian Empire (and even where he dies tomorrow as well) if those were 100% the choices (I do think he might well be willing to risk a non-trivial chance of nuclear war to get it, but that is different), but let’s say that it is 10% (and notice that in those 10%, if he knew for a fact we’d respond with our nukes, he’d just say ‘oh that’s too bad’ and destroy the world in a fit of pique, which very much doesn’t seem right).
Yeah, I should have said 10% chance of escalating all the way to destroying the world (if the West doesn’t let Putin have his way), through all causes, not just Putin having that kind of values.
If we lived by the rules you’re suggesting in the past, also, we wouldn’t have gotten this far—we would have folded to the USSR as nation after nation turned red, and at best we’d be in a much poorer, less free world (assuming it’s nuke-specific, and we still fight WW2).
But if the USSR became the world government, at least we wouldn’t be repeatedly facing 10% chance of destroying the world. Is that an obvious tradeoff to you?
I’m not sure why this got voted down, but I don’t disagree. In parts of my comments that you didn’t quote, I indicated my own uncertainty about the tradeoff.
If there were a bunch of Putin-style people in charge of the world that doesn’t seem like a ‘safe’ world either. It seems like a world where these states engage in continuous brinksmanship that, if this kind of mindset is common, leads to a 10% chance of Armageddon as often or more often than the current one.
We may have very different models of what happens if we let the USSR take over, but yeah I think that world has destroyed most of its value assuming it didn’t go negative. And I don’t think you eliminate the risks—you have a bunch of repressive communist governments everywhere in a world where conditions are getting worse (because communism doesn’t work) and they start fighting over resources slash have nuclear civil wars.
If the model is ‘Putin escalates to nuclear war sometimes and maybe he miscalculates’ then ‘fold to him’ is letting him conquer the world, literally, because no he wouldn’t stop with Russia’s old borders if we let him get Warsaw and Helsinki. Why would he? Otherwise, folding more makes him escalate until the nukes fly.
And I don’t think you eliminate the risks—you have a bunch of repressive communist governments everywhere in a world where conditions are getting worse (because communism doesn’t work) and they start fighting over resources slash have nuclear civil wars.
I’m assuming that the USSR would not have let other communist governments develop their own nuclear weapons.
It seems like the only world which doesn’t face repeated 10% chances of Armageddon is one in which some state has a nuclear monopoly and enforces it by threatening to attack any other state that tries to develop nuclear weapons (escalating to nuclear attack if necessary). Ideally, this would have been the US, but failing that, maybe the USSR having a nuclear monopoly would be preferable to the current situation.
Also, communist governments don’t always get worse over time, monotonically. Sometimes they get better instead. It’s pretty unclear to me what would have happened to the USSR in the long run, in this alternate world in which they achieved a nuclear monopoly.
On the personal level, to me this seems like a potential failure mode worth worrying about in an AGI context because that’s Impossible Mode for everything, but not in practical human mode, and most definitely not on the margin. I’m not claiming I have the best possible answer here, I’m claiming that what humans are current doing is some mix of absurdly stupid things and non-FDT proposals that exist seem way worse than FDT proposals that exist—also that actual human attempts to be FDT-style agents will be incomplete and not lead to degenerate outcomes, the same way mostly-basically-CDT-style human agents often don’t actually do the fully crazy things it implies when it implies fully crazy things.
On the global level, again I’m not saying I know the details of how we should respond, only that we shouldn’t lay down and let him get whatever he wants.
I think it is essentially epsilon chance that Putin would choose nuclear firestorm over a world where Russia doesn’t recreate the USSR / Russian Empire (and even where he dies tomorrow as well) if those were 100% the choices (I do think he might well be willing to risk a non-trivial chance of nuclear war to get it, but that is different), but let’s say that it is 10% (and notice that in those 10%, if he knew for a fact we’d respond with our nukes, he’d just say ‘oh that’s too bad’ and destroy the world in a fit of pique, which very much doesn’t seem right). In the other 90%, he backs down after various numbers of escalations (e.g. in some he tries the escalate-to-deescalate single nuke, others he tries leveling Kyiv, others he folds tomorrow and leaves) in some combination, then folds after he sees we won’t fold, but losses are ‘acceptable’ here.
In those 10% of worlds, what are we hoping for? I don’t think there is much of an ‘eventually’ here.
He takes Ukraine, we let him. He sees we let him do what he wants. Everyone else sees too. Every state starts a nuclear weapons program that can afford one, so Putin knows he’s on limited time. Ukraine starts an insurgency and Putin starts killing A LOT of people in response. We do nothing. Moldavia is next pretty much right away. That falls in days. He then goes for Kazakhstan. Within a year he has all the non-NATO USSR republics in hand.
Meanwhile, Xi launches an invasion of Taiwan. Putin makes it clear that if the USA interferes he’ll back China up. We fold. China takes it. TSMC is destroyed. We lose the majority of our chip capacity, either to China or the void. Everyone knows our commitments mean nothing. Every country with a score to settle acts now.
Now Putin takes part of Estonia, and fortifies it. This is, let’s say, December 2022, and Trump-backed Republicans just swept the midterms during an extreme recession. What do we do? Again, clearly, nothing.
NATO is dead. No one believes us at all, anywhere. Putin invades and takes the Baltics. Six months later, he’s in Warsaw and Bucharest, perhaps without firing a shot. North Korea marches south and points its ICBMs.
And then it gets worse.
Or, more likely, at some point in that story we DO confront him, and the nuclear war happens anyway—and there’s a lot of worlds where he didn’t want that, but by folding so often, we let him think he could do it, so he doesn’t know where to stop, and then we get into a nuclear war over Estonia or whatever and someone miscalculates.
And that scenario happens 50%+ of the time rather than 10%, because there’s a ton of worlds where Putin/Xi/etc will take advantage like that, but where they very much would have folded if challenged. The chances of nukes flying in the medium (10 year) term go up, not down.
Even the best case scenarios I can imagine in such places are ones I very much do not like.
(Also, I think that if Putin tried to start a nuclear war without any attacks on Russia itself, that there is a >50% (although of course not terribly reassuring) chance that the answer would be ‘no’ and a substantial chance Russia’s nuclear weapons mostly no longer work and are a bluff whether or not Putin knows they don’t work—I very much would not launch ours until post-impact in the hopes this was the case, given our second strike capabilities. And there’s the worlds in which Putin tries to launch one nuke, it’s a dud, or his people turn on him, and that’s the end of all of it.)
If we lived by the rules you’re suggesting in the past, also, we wouldn’t have gotten this far—we would have folded to the USSR as nation after nation turned red, and at best we’d be in a much poorer, less free world (assuming it’s nuke-specific, and we still fight WW2).
What are some examples of these?
Yeah, I should have said 10% chance of escalating all the way to destroying the world (if the West doesn’t let Putin have his way), through all causes, not just Putin having that kind of values.
But if the USSR became the world government, at least we wouldn’t be repeatedly facing 10% chance of destroying the world. Is that an obvious tradeoff to you?
After reading Solzhenitsyn’s The Gulag Archipelago, I am not really sure which outcome is worse.
I’m not sure why this got voted down, but I don’t disagree. In parts of my comments that you didn’t quote, I indicated my own uncertainty about the tradeoff.
If there were a bunch of Putin-style people in charge of the world that doesn’t seem like a ‘safe’ world either. It seems like a world where these states engage in continuous brinksmanship that, if this kind of mindset is common, leads to a 10% chance of Armageddon as often or more often than the current one.
We may have very different models of what happens if we let the USSR take over, but yeah I think that world has destroyed most of its value assuming it didn’t go negative. And I don’t think you eliminate the risks—you have a bunch of repressive communist governments everywhere in a world where conditions are getting worse (because communism doesn’t work) and they start fighting over resources slash have nuclear civil wars.
If the model is ‘Putin escalates to nuclear war sometimes and maybe he miscalculates’ then ‘fold to him’ is letting him conquer the world, literally, because no he wouldn’t stop with Russia’s old borders if we let him get Warsaw and Helsinki. Why would he? Otherwise, folding more makes him escalate until the nukes fly.
I’m assuming that the USSR would not have let other communist governments develop their own nuclear weapons.
It seems like the only world which doesn’t face repeated 10% chances of Armageddon is one in which some state has a nuclear monopoly and enforces it by threatening to attack any other state that tries to develop nuclear weapons (escalating to nuclear attack if necessary). Ideally, this would have been the US, but failing that, maybe the USSR having a nuclear monopoly would be preferable to the current situation.
Also, communist governments don’t always get worse over time, monotonically. Sometimes they get better instead. It’s pretty unclear to me what would have happened to the USSR in the long run, in this alternate world in which they achieved a nuclear monopoly.
Zvi: It’s interesting that your argument above is phrased entirely in the framework of causal decision theory. Might there be a good reason for that?