The curve of AI death
Motivation
On Manifold, you’ll see lots of markets like “Will AI wipe out humanity by 2040?”. These prices aren’t very informative, and they trade more like opinion polls. The anthropic bias ruins it, because YES holders can’t win mana without being dead. Though even for NO holders, the timeframe and opportunity cost make it unappealing (we can get higher annualized returns elsewhere). I can’t do much about that obstacle yet.
But I wanted to try capturing AI risk from a more falsifiable angle.
A strong prior: the power law curve
Other disasters share a familiar power law curve. Take for example the worst novel plagues, which are disproportionately worse than mid-sized ones, and so on. We see this in lots of domains, happens in financial disasters too. Also true for deaths via human conflicts.
Though “power law curve” doesn’t necessarily mean “small numbers”. And it’s plausible it could be “spikier” than a vanilla power law curve, as I suspect things start happening much faster. But whatever your prior is for AI death events, I suggest some sort of power law curve.
How much can doom-arguments update this curve?
Some people say the upcoming AI transformations will be so violent, that this “curve” is extremely spikey, almost approaching being binary. So maybe a few people die in self-driving cars, maybe a bunch of people get hacked and lose money. And then in a few short years, it skips to “8 billion+ fall over dead during the same second, in a coordinated global nanotech strike.”
Do they have enough information to justify updating so far away from the prior? That doesn’t seem credible. I think their pre-paradigmatic thought experiments and fuzzy terms are much weaker than some of them assert.
Existing instruments are poor proxies for this
I’ve heard Tyler Cowan say he doesn’t find doomers sincere. He claims they ought to take out debt, etc. I think I might’ve even once heard him say credit card debt [pending enough attention to find the clip].
This seems false. In general, the existing instruments are very contaminated “measures” of AI risk. Maybe there’s an argument for holding semiconductors (I think some of them do). But taking out massive debt based on this belief is generally ill-advised, even for short-timeliners. This sounds Kelly-suboptimal, and is long-term very disadvantaged whenever the position isn’t removed by observer selection effects. [1]
Ideally, there would be efficient markets on this. In the ideal world, the slope of AI death would be kind of like the bond market yield curve, but for AI death. And we wouldn’t say to people “See? Bond market says no AI death anytime soon”. We’d have instruments directly about the thing we mean. But people more influential than me would rather “wing it” on getting reliable estimates about lots of things, than to let compulsive people bankrupt themselves gambling.
Manifold it is.
Testing something more testable
I am offering markets to illustrate this curve. They have much less anthropic bias than extinction-based ones. The slope of the curve seems worth trying to understand, so here’s my contribution.
No free lunch
There are gray areas I haven’t disambiguated yet. E.g. Jonathan Ray asks:
If WIV uses AI to design its next bioweapon, and that starts another pandemic like covid-19 killing >10M people, does that count?
I’m afraid I don’t have a fleshed-out taxonomy yet. But here I tried to show my current attitude, for future improvement.
Great question, I haven’t actually decided this yet. Will partly depend on what we find out about the event. Some dimensions I’m considering:
An increase in how easily we can point to a causal blunder by the human workers/managers → a decrease in how much I’d count it as an AI disaster.
A decrease in a lab’s general track record of safety → a decrease in how much I’d attribute it to the AI.
An increase in how much we think the infectiousness and virulence are “explained by” the AI’s contribution to its development → an increase in counting it.
I know this doesn’t answer the question, it’s a hard one! Open to suggestions.
Some of these gray areas are just difficult to taxonomize. Still, I feel it is shameful for a market maker to have big ambiguities (I think synthetic plagues are a massive risk). For some context, my capacity to work on this stuff is limited and distracted. But I’ll try to refine these over time, roughly along the lines of the above framework.
- ^
Trust me bro
Wanted: has anyone on LessWrong said they moved money off an exchange, after seeing my markets? I made a meta-market asking if literally anyone would be influenced to move money off any exchange mentioned in any of my crypto-related markets.
If you’ve been influenced by any of my markets on the relative exchange risks, please let me know so I can reward predictors.