I want markets for x-risks, basically. Suppose someone wants to train an AI and they’re pretty sure it’ll be fine, but only pretty sure. How do we aggregate estimates to figure out whether or not the AI should be trained? Seems like we should be able to have a risk market. [so8res proposes apocalypse insurance here with “premiums dependent on their behavior”, but what’s the dependence? Is it just set by the regulator?]
But the standard problems appear; on the “we all die” risk, the market doesn’t exist and so people who bet on risk never get paid out.
You could imagine instead using a cap-and-trade system, where, say, only 3 AIs can be trained per year and companies bid for one of the permits, but it seems like this is still tilted towards “who thinks they can make the most money from success?” and not “who does the group think is least likely to fail?”. You could have instead an explicit veto/permission system, where maybe you have 11 votes on whether or not to go thru with an AI training run and you need to buy at least 6 ’yes’es, but this doesn’t transfer resources from worse predictors to better predictors, just from projects that look worse to projects that look better.
And so I think we end up with, as johnswentworth suggests over here, needing to use regular liability / regular insurance, where people are betting on short-term questions that we expect to resolve (“how will this lawsuit by rightsholders against generative AI companies go?”) instead of the unresolvable questions that are most existentially relevant.
I want markets for x-risks, basically. Suppose someone wants to train an AI and they’re pretty sure it’ll be fine, but only pretty sure. How do we aggregate estimates to figure out whether or not the AI should be trained? Seems like we should be able to have a risk market. [so8res proposes apocalypse insurance here with “premiums dependent on their behavior”, but what’s the dependence? Is it just set by the regulator?]
But the standard problems appear; on the “we all die” risk, the market doesn’t exist and so people who bet on risk never get paid out.
You could imagine instead using a cap-and-trade system, where, say, only 3 AIs can be trained per year and companies bid for one of the permits, but it seems like this is still tilted towards “who thinks they can make the most money from success?” and not “who does the group think is least likely to fail?”. You could have instead an explicit veto/permission system, where maybe you have 11 votes on whether or not to go thru with an AI training run and you need to buy at least 6 ’yes’es, but this doesn’t transfer resources from worse predictors to better predictors, just from projects that look worse to projects that look better.
And so I think we end up with, as johnswentworth suggests over here, needing to use regular liability / regular insurance, where people are betting on short-term questions that we expect to resolve (“how will this lawsuit by rightsholders against generative AI companies go?”) instead of the unresolvable questions that are most existentially relevant.