Yes, but in a highly dynamic, rapidly evolving situation where the classified data is extremely asymmetric with respect to open sources? A CIA analyst will have daily or even hourly satellite photo updates, real-time access to Russian and Ukrainian communications and transmitter positions, the non-propaganda numbers for functioning military equipment and losses, not to mention all sorts of human intel on the ground. Someone outside of that world and not on the ground in Ukraine has very limited access to information—mostly propaganda from both sides. It’s really hard to believe you can extract a better signal from that.
I can believe that prediction markets can outperform a handful of analysts with access to classified data in certain contexts. But this situation appears to me to be maximally weighted in favor of the analysts.
After screwing up predictions about Iraqi WMDs the CIA believed that they have a problem. That lead to IARPA making a tournament for different approaches to making predictions. Tedlock’s Good Judgement project won that tournament.
I think you can make an argument that Metaculus community predictions aren’t really reflecting superforcasting predictions as there are plenty of people who are making predictions without that skillset but a well-functioning prediction market wouldn’t have that problem.
Yes, but in a highly dynamic, rapidly evolving situation where the classified data is extremely asymmetric with respect to open sources? A CIA analyst will have daily or even hourly satellite photo updates, real-time access to Russian and Ukrainian communications and transmitter positions, the non-propaganda numbers for functioning military equipment and losses, not to mention all sorts of human intel on the ground. Someone outside of that world and not on the ground in Ukraine has very limited access to information—mostly propaganda from both sides. It’s really hard to believe you can extract a better signal from that.
I can believe that prediction markets can outperform a handful of analysts with access to classified data in certain contexts. But this situation appears to me to be maximally weighted in favor of the analysts.
After screwing up predictions about Iraqi WMDs the CIA believed that they have a problem. That lead to IARPA making a tournament for different approaches to making predictions. Tedlock’s Good Judgement project won that tournament.
I think you can make an argument that Metaculus community predictions aren’t really reflecting superforcasting predictions as there are plenty of people who are making predictions without that skillset but a well-functioning prediction market wouldn’t have that problem.