I haven’t heard much about machine learning used for forecast aggregation. It would seem to me like many, many factors could be useful in aggregating forecasts. For instance, some elements of one’s social media profile may be indicative of their forecasting ability. Perhaps information about the educational differences between multiple individuals could provide insight on how correlated their knowledge is.
I think people are looking in to it: The Good Judgment Project team used simple machine learning algorithms as part of their submission to IARPA during the ACE Tournament. One of the PhD students involved in the project wrote his dissertation on a framework for aggregating probability judgments. In the Good Judgment team at least, people are also in using ML for other aspects of prediction—for example, predicting if a given comment will change another person’s forecasts—but I don’t think there’s been much success.
I think a real problem is that there’s a real paucity of data for ML-based prediction aggregation compared to most machine learning projects—a good prediction tournament gets a couple hundred forecasts resolving in a year, at most.
Probability density inputs would also require additional understanding from users. While this could definitely be a challenge, many prediction markets already are quite complicated, and existing users of these tools are quite sophisticated.
I think this is a bigger hurdle than you’d expect if you’re implementing these for prediction tournaments, though it might be possible to do for prediction markets. (However, I’m curious how you’re going to implement the market mechanism in this case.) Anecdotally speaking many of the people involved in GJ Open are not particularly math or tech savvy, even amongst the people who are good at prediction.
I think people are looking in to it: The Good Judgment Project team used simple machine learning algorithms as part of their submission to IARPA during the ACE Tournament. One of the PhD students involved in the project wrote his dissertation on a framework for aggregating probability judgments. In the Good Judgment team at least, people are also in using ML for other aspects of prediction—for example, predicting if a given comment will change another person’s forecasts—but I don’t think there’s been much success.
I think a real problem is that there’s a real paucity of data for ML-based prediction aggregation compared to most machine learning projects—a good prediction tournament gets a couple hundred forecasts resolving in a year, at most.
I think this is a bigger hurdle than you’d expect if you’re implementing these for prediction tournaments, though it might be possible to do for prediction markets. (However, I’m curious how you’re going to implement the market mechanism in this case.) Anecdotally speaking many of the people involved in GJ Open are not particularly math or tech savvy, even amongst the people who are good at prediction.