I wonder if it’d be possible to write a scraper script that gathers all of the empirical predictions casually made by the top ~100 AI influencers over the past couple of years, and then evaluates them in an automated fashion. Even just limiting it to tweets predicting benchmark numbers and having an LLM jsonify them so they could be fed into an API that checks and aggregates them seems like it’d produce a very telling output.
For instance:
Person A's Predictions:
- Prediction accuracy: 60%
- Error breakdown: 90% too fast, 10% too slow
- ELO (determined by disagreeing with others' resolved predictions): 1400
Not the solution you asked for, but tighter integration of prediction markets into social media could enable this. For example, making the tweet → prediction market UI/UX flow really good (and also surfacing relevant prediction markets next to the content) would make it easier 1. setup a market based on someone’s predictions and 2. keep track of those predictions.
I wonder if it’d be possible to write a scraper script that gathers all of the empirical predictions casually made by the top ~100 AI influencers over the past couple of years, and then evaluates them in an automated fashion. Even just limiting it to tweets predicting benchmark numbers and having an LLM jsonify them so they could be fed into an API that checks and aggregates them seems like it’d produce a very telling output.
For instance:
Not the solution you asked for, but tighter integration of prediction markets into social media could enable this. For example, making the tweet → prediction market UI/UX flow really good (and also surfacing relevant prediction markets next to the content) would make it easier 1. setup a market based on someone’s predictions and 2. keep track of those predictions.