I largely agree, I think the main benefit of actually using a real-money market is when you need to acquire information that’s difficult to access and the market becomes a public “bounty” of-sorts for insiders. If the answer doesn’t hinge on some proprietary information, then I don’t see how paying 10 experts wouldn’t be better.
alexjaniak
Karma: 28
Definitely agree. The “will” does a lot of heavy lifting here and also the foot-note is a bit vague. I recognize that it’s a big assumption but tried to keep it simple since I think it’s worthy of a separate post. I meant mostly that, ideally, the AI agents will have those properties.
Hopefully we are able to align the AI so that it behaves and doesn’t gamble your money away if you ask it to bet on some belief. Not too familiar with the mechanics and details of “alignment” so take that with a grain-of-salt.
Agreed. I found this point to be especially insightful: “If the final advice looks the same but the process of getting there involved structured reasoning and the exposure of hidden risks, that is a victory.”
https://forum.effectivealtruism.org/posts/WpofddF8waFhxakNB/why-forecasting-fails-decision-makers
In a world where these markets were extremely calibrated, I could potentially see decision makers blindly trusting them, but that’s unlikely for markets that give actionable advice for the reasons outlined in the post.
I think you’re measuring the right thing (decisions changed) but blaming the wrong cause. I think the field underperformed because:
The questions are at the wrong altitude. “P(AGI by 2027)” is fun to trade but hard to act on. The decision-relevant questions (e.g., will this research direction work, will this eval saturate first, will this intervention move its metric) rarely get asked because they’re too narrow and poorly funded to attract pro-forecasters. Moreover, such narrow questions usually rely on internal information that is difficult to attain for the forecasters.
Good forecasts aren’t reaching decision-makers. There’s no apparent pipeline from these forecasting platforms to decisions at a large enough scale to appear noticeable. I’d argue forecasting is still “niche” amongst the general population.
AI forecasters fix (1) directly. Calibrated answers to arbitrary narrow questions means you can finally ask the questions that bind to actual decisions, with the internal information and predictive power actually needed to forecast on these questions correctly.
If you had an expert AI forecasting on your daily decisions would you not listen?I don’t think the right update from the last decade is “stop funding”. I think it’s “stop funding platforms and tournaments, start funding question design, decision integration, and automated forecasting.”
Fair pushback. I actually dug into this claim before I posted, and I think Wikipedia is a bit vague on the details.
Specific betting markets are pretty old, but from my research, prediction markets weren’t formalized as general-purpose mechanisms until the late 80s.
Hayek, Mises, and, oddly unmentioned in the wiki, Kenneth Arrow, made the economic theory underpinnings. Of course there were others but I think those are the most prominent.
Hanson is credited here for the idea as well. Hanson himself says “I generated this idea in the fall of 1988. See our article describing some history of the web game and of my involvement with the idea. I’ve found several prior publications where others had similar ideas. I think I’ve thought the idea through more though.” in his Idea Futures page.
My read is that the theoretical underpinnings and the contemporaneous specific implementations (IEM for elections, betting markets generally) didn’t add up to a general-purpose formulation until Hanson explicitly proposed one in 1988-90. Curious if you’d point to someone earlier who made that jump?