Tyler Cowen and Bryan Caplan among others have both challenged so-called AI doomers to put money where their mouth is, bet on extinction. I’m not the first person to point out the big problems with this:
The naive version: a direct bet on extinction is incoherent because the doomer would expect to be dead.
The slightly more advanced version: the doomer gets paid up front and pays back double (Or whatever the betting odds are) with interest later if doom doesn’t happen. But this doesn’t quite make sense either. If doom does happen, the doomer has a brief and mostly useless window to spend the money, and the accelerationist has no reason to expect the doomer to save any money. And if the doomer does save it (plus enough extra to cover the doubled payback), they’ve effectively just locked up double the original capital until the end of the world. Neither party has a coherent incentive structure.
Here’s a version that perhaps actually works (under some assumptions and unless I’m overlooking something here): bet on protective policy outcomes that are correlated with survival or at least longer timelines.
Examples: Will the US enact a federal datacenter moratorium before 2030 with export controls? Will the US and China sign a meaningful bilateral agreement on frontier AI before 20XX? Will there be a federal AI safety law? (Edit: I changes the examples a bit)
These outcomes point to longer timelines and higher survival probability. The structure is much more sensible because the marginal utility of money is higher in the surviving-or-longer-timelines branch than in the short-timelines-to-doom branch.
Concretely: if the doomer bets $100 YES on a moratorium and it doesn’t pass, they’ve lost the money in a world where they’re going to be dead sooner. If the moratorium does pass, the $100 (at appropriate long odds) becomes a much larger payout in a world where the doomer actually expects to live, or at least live longer. So the doomer wants their payout in the surviving-or-longer-timelines branch, which means betting YES on protective policy at appropriate betting odds.
For the accelerationist side, it depends on flavor. If they’re an effective accelerationist who thinks faster AI development gets them to literally owning galaxies sooner (Leopold Aschenbrenner and Dwarkesh: galaxies may become purchasable for money soon after ASI), they’d have a clean reason to bet NO. If they just think AGI isn’t coming and therefore low P(doom), they shouldn’t really have a strong view on a datacenter moratorium, or it depends on their specific view.
Importantly though, it’s not necessary to bet with an effective accelerationist at all. You can just bet with people who are betting on the object-level reality: will a moratorium pass or not? You’d expect to have an edge: you have the insight that the money will be more valuable to you in the branch where the moratorium does happen, so you’re willing to take the side at odds others aren’t.
Or am I overlooking something here?
(I know this particular betting market is about a moratorium anywhere in the US but couldn’t find anything better)
(Couldn’t think of a better shorthand for doomer that made it clear who I am talking about)
This requires having more views about worlds where money is most valuable. There’s a U-curve where
excellent policies pass: we’re in the clear. low value of money
decent policies pass, or maybe they fail by a thin margin: more work required. high value of money
nothing is even close to passing: we’re doomed. low value of money
It’s complicated to determine where the peak of the U-curve is, or even which side of it you’re on. I think I’d have a hard time figuring out which side of a bet to take to maximize expected value.
For example, it’s not obvious to me that the doomer should prefer to bet YES on a datacenter moratorium getting passed.
If excellent policy passes that let’s say shuts down AI, money would be valuable in the selfish sense? Like you can buy something from it, this is the main sense I was thinking of when talking about value of money. Are you thinking of how useful is money in advancing safe AI?
One good scenario with low payoffability is one where there is ASI aligned to humanity as a whole, such that only a few prestige goods are meaningfully scarce and money matters much less to material quality of life.
This could be the default outcome of good policy if alignment is moderately hard—not so easy that we get it for free with capabilities, but not so hard that we can’t have a moonshot program or temporary pause that allows it to catch up. (We can throw in ”aligned to human or sentient interests as a whole rather than some corporate/political egomaniac” as part of “good.”)
Ofc no policy can guarantee that alignment is moderately easy!
accelerationist has no reason to expect the doomer to save any money. And if the doomer does save it (plus enough extra to cover the doubled payback), they’ve effectively just locked up double the original capital until the end of the world
Couldn’t the doomer and accelerationist just agree that the doomer doesn’t have to pay until (e.g.) one year after the bet resolves? Then the doomer could spend all the money in anticipation of doom. If the doomer loses the bet, they can use the year after resolution to earn money to pay the accelerationist back.
(Of course there are extra practical difficulties here, like e.g. it might be hard for humans to earn money in the future. But I’m just talking about theoretical barriers.)
The bet still resolves at the same time. The doomer just has one year after resolution to get their bank balance back up from $0 so they can pay the accelerationist back.
The shape of their argument is “see, ‘doomers’ don’t actually believe their statements about AI risk because they didn’t put money on the line!”. I think you’re suggesting an interesting mechanism for converting beliefs about AI risk to bets (I can understand wanting this separately from convincing them), but I reject the frame and premise of their argument. If you manufactured the perfect financial instrument and substantially invested in it, I don’t believe their opinions on AI-risk or the safety community would change.
I saw an interesting rebuttal on X: “Do you believe people concerned about nuclear war were wrong to be concerned? If they didn’t make the kind of extreme financial decisions you’re suggesting, were they hypocritical?”
This works as a means of publicly demonstrating a credence (in doom), but exists in tension with the original prediction market serving as a credence aggregator (about the passing of data centers.)
(Let’s say there are many such bets whose payoffability correlates, to varying degrees, with doom. Is there a way to collectively abstract from them a “true” crowd-level credence in doom and “true” crowd-level credences in e.g. data center moratoria? I’m not deep enough in this world to say. Perhaps if you believe prediction markets are otherwise highly efficient at capturing best available credence and, as doom-correlated bets pay off, you track just how systematically “biased” the market was betting in one direction.)
I agree– from the perspective of trying to maximize profit from this bet, you’d want the market to not factor in the conditional P(doom¦moratorium passes).
If an AI doomer bets $100 to forecast that a moratorium will pass, then they’ve created two incentives:
An incentive of $100 for themselves to make the moratorium pass.
A bounty of $100 for anybody else to block the moratorium from passing.
Notably, they’ve locked up their $100 of capital for the duration of the forecast, which is now unavailable to support their efforts.
These “put your money where your mouth is” arguments are turning into an unbelievably stupid form of rhetoric. Fortunately, they’re so outlandish that they only register in extremely online aging libertarian land.
AI slowdowns are popular and common sense policies among American voters.
Don’t get caught up in debating the astroturfing pundits! They are not worth your time.
I mean it harsh. At worst, it’s a cheap and surprisingly effective morale-degrading confusion tactic aimed directly at undermining people working on existential risk.
On betting on AI doom
Tyler Cowen and Bryan Caplan among others have both challenged so-called AI doomers to put money where their mouth is, bet on extinction. I’m not the first person to point out the big problems with this:
The naive version: a direct bet on extinction is incoherent because the doomer would expect to be dead.
The slightly more advanced version: the doomer gets paid up front and pays back double (Or whatever the betting odds are) with interest later if doom doesn’t happen. But this doesn’t quite make sense either. If doom does happen, the doomer has a brief and mostly useless window to spend the money, and the accelerationist has no reason to expect the doomer to save any money. And if the doomer does save it (plus enough extra to cover the doubled payback), they’ve effectively just locked up double the original capital until the end of the world. Neither party has a coherent incentive structure.
Here’s a version that perhaps actually works (under some assumptions and unless I’m overlooking something here): bet on protective policy outcomes that are correlated with survival or at least longer timelines.
Examples: Will the US enact a federal datacenter moratorium before 2030 with export controls? Will the US and China sign a meaningful bilateral agreement on frontier AI before 20XX? Will there be a federal AI safety law? (Edit: I changes the examples a bit)
These outcomes point to longer timelines and higher survival probability. The structure is much more sensible because the marginal utility of money is higher in the surviving-or-longer-timelines branch than in the short-timelines-to-doom branch.
Concretely: if the doomer bets $100 YES on a moratorium and it doesn’t pass, they’ve lost the money in a world where they’re going to be dead sooner. If the moratorium does pass, the $100 (at appropriate long odds) becomes a much larger payout in a world where the doomer actually expects to live, or at least live longer. So the doomer wants their payout in the surviving-or-longer-timelines branch, which means betting YES on protective policy at appropriate betting odds.
For the accelerationist side, it depends on flavor. If they’re an effective accelerationist who thinks faster AI development gets them to literally owning galaxies sooner (Leopold Aschenbrenner and Dwarkesh: galaxies may become purchasable for money soon after ASI), they’d have a clean reason to bet NO. If they just think AGI isn’t coming and therefore low P(doom), they shouldn’t really have a strong view on a datacenter moratorium, or it depends on their specific view.
Importantly though, it’s not necessary to bet with an effective accelerationist at all. You can just bet with people who are betting on the object-level reality: will a moratorium pass or not? You’d expect to have an edge: you have the insight that the money will be more valuable to you in the branch where the moratorium does happen, so you’re willing to take the side at odds others aren’t.
Or am I overlooking something here?
(I know this particular betting market is about a moratorium anywhere in the US but couldn’t find anything better)
(Couldn’t think of a better shorthand for doomer that made it clear who I am talking about)
This requires having more views about worlds where money is most valuable. There’s a U-curve where
excellent policies pass: we’re in the clear. low value of money
decent policies pass, or maybe they fail by a thin margin: more work required. high value of money
nothing is even close to passing: we’re doomed. low value of money
It’s complicated to determine where the peak of the U-curve is, or even which side of it you’re on. I think I’d have a hard time figuring out which side of a bet to take to maximize expected value.
For example, it’s not obvious to me that the doomer should prefer to bet YES on a datacenter moratorium getting passed.
If excellent policy passes that let’s say shuts down AI, money would be valuable in the selfish sense? Like you can buy something from it, this is the main sense I was thinking of when talking about value of money. Are you thinking of how useful is money in advancing safe AI?
Yeah I was thinking of the value of money for reducing x-risk.
One good scenario with low payoffability is one where there is ASI aligned to humanity as a whole, such that only a few prestige goods are meaningfully scarce and money matters much less to material quality of life.
This could be the default outcome of good policy if alignment is moderately hard—not so easy that we get it for free with capabilities, but not so hard that we can’t have a moonshot program or temporary pause that allows it to catch up. (We can throw in ”aligned to human or sentient interests as a whole rather than some corporate/political egomaniac” as part of “good.”)
Ofc no policy can guarantee that alignment is moderately easy!
Couldn’t the doomer and accelerationist just agree that the doomer doesn’t have to pay until (e.g.) one year after the bet resolves? Then the doomer could spend all the money in anticipation of doom. If the doomer loses the bet, they can use the year after resolution to earn money to pay the accelerationist back.
(Of course there are extra practical difficulties here, like e.g. it might be hard for humans to earn money in the future. But I’m just talking about theoretical barriers.)
I don’t see how this is different from just pushing the date of the bet back by one year?
The bet still resolves at the same time. The doomer just has one year after resolution to get their bank balance back up from $0 so they can pay the accelerationist back.
The shape of their argument is “see, ‘doomers’ don’t actually believe their statements about AI risk because they didn’t put money on the line!”. I think you’re suggesting an interesting mechanism for converting beliefs about AI risk to bets (I can understand wanting this separately from convincing them), but I reject the frame and premise of their argument. If you manufactured the perfect financial instrument and substantially invested in it, I don’t believe their opinions on AI-risk or the safety community would change.
I saw an interesting rebuttal on X: “Do you believe people concerned about nuclear war were wrong to be concerned? If they didn’t make the kind of extreme financial decisions you’re suggesting, were they hypocritical?”
I think if we relax the constraint on bets between hardline doomers and hardline accelerationists we can get more interesting and productive dynamics.
Instead of full on extinction we could bet on extinction proxies:
Rouge Independent AI by 2030? AI governed cults by 2035? Deaths by AI per 100,000 people exceeds 0.001 by 2050
These are bets I’m willing to actually take someone up on, since I think each of these will come true.
This works as a means of publicly demonstrating a credence (in doom), but exists in tension with the original prediction market serving as a credence aggregator (about the passing of data centers.)
(Let’s say there are many such bets whose payoffability correlates, to varying degrees, with doom. Is there a way to collectively abstract from them a “true” crowd-level credence in doom and “true” crowd-level credences in e.g. data center moratoria? I’m not deep enough in this world to say. Perhaps if you believe prediction markets are otherwise highly efficient at capturing best available credence and, as doom-correlated bets pay off, you track just how systematically “biased” the market was betting in one direction.)
I agree– from the perspective of trying to maximize profit from this bet, you’d want the market to not factor in the conditional P(doom¦moratorium passes).
If an AI doomer bets $100 to forecast that a moratorium will pass, then they’ve created two incentives:
An incentive of $100 for themselves to make the moratorium pass.
A bounty of $100 for anybody else to block the moratorium from passing.
Notably, they’ve locked up their $100 of capital for the duration of the forecast, which is now unavailable to support their efforts.
These “put your money where your mouth is” arguments are turning into an unbelievably stupid form of rhetoric. Fortunately, they’re so outlandish that they only register in extremely online aging libertarian land.
AI slowdowns are popular and common sense policies among American voters.
Don’t get caught up in debating the astroturfing pundits! They are not worth your time.
Seems harsh. At worst they are a misguided in a way that’s subtle enough that intelligent people still get the wrong answer.
I mean it harsh. At worst, it’s a cheap and surprisingly effective morale-degrading confusion tactic aimed directly at undermining people working on existential risk.