I am intrigued by the ‘solution seeking a problem’ framing. Unlike blockchain, there are a number of domains and decisions that would analytically benefit from more accurate predictions about the future (e.g. issue prioritisation, campaigning decisions in politics, domestic policy decisions, geopolitical actions) even if they have practically not adopted forecasting widely. Better forecasts can prevent worse counterfactual worlds, e.g. by preventing planning mistakes leading to rapid inflation, encouraging better resilience planning, etc.
The claim that ‘Forecasting cannot be useful’ seems to have the plausible alternative claim ‘Forecasting can be useful with better distribution’, i.e. we need to find better ways to get policy- and decision-makers access to low-friction, high-quality, defensible forecasts that they can use. This would still mean that existing funding that is focused on improving the resolution and quality of forecasts needs to re-prioritise and focus on the distribution more, but would not go so far as to say we should stop funding it altogether.
The pattern of ‘analytically useful, but practically unadopted’ is not unprecedented. Seatbelts and childproof medical caps, for example, were both analytically beneficial but did not see widespread adoption till concentrated efforts were made on the distribution front. Clear, measurable benefits followed.
On the solution seeking a problem, I think proponents of blockchain say similar things about how useful it would be if our financial system were predominantly on the blockchain and how much benefit there would be, in particular to the unbanked. They do, in fact, say this.
I never said that forecasting cannot be useful. I think it hasn’t been useful to the tune of the money thrown at it, and I’m skeptical that further investment is going to pay off in usefulness. I was somewhat expecting the defenses of forecasting to be better than they have been. The best one was probably this one by @habryka or another reply by Eva Vivalt on the EA forum.
It’s possible we haven’t been funding the right thing within forecasting. Maybe this weekend I will tally up every grant and investment made into forecasting, but I think it is ~$100m. That said, I think basically any poor funding decision will have people saying something similar to this:
This would still mean that existing funding that is focused on improving the resolution and quality of forecasts needs to re-prioritise and focus on the distribution more, but would not go so far as to say we should stop funding it altogether.
When a startup says that their product hasn’t taken off and all they really need is better marketing to do so, you should run. I think we should say the same thing here, too. If you need to focus on distribution and try to force your product into being used, it probably isn’t that useful. When something is very useful, it gets adopted at extreme speed. The AI companies aren’t spending a lot on marketing. For example, people adopted LLMs, chatbots, and other products extremely quickly, and they were more or less just announced.
Furthermore, prediction markets have been adopted, so it’s hard to say it’s an issue of marketing. Almost everyone important has heard of Polymarket and Kalshi at this point and know how PMs work. And none of them has decided that they would be very useful. It just so happens that they are useful for sports gambling primarily and not for making better decisions.
On the usefulness of blockchain, I think the analytical case for our financial system being on blockchain is significantly weaker than the analytical case for better forecasts being useful for policymaking.
With my original comment, I was implicitly drawing a distinction between ‘cannot be useful’ and ‘has not been useful’:
If forecasting simply has not been useful, then it leaves us with two possibilities: (1) it cannot be and we should stop funding it; (2) it can be and we should redirect funding to efforts that figure out and solve why it hasn’t been (e.g. distribution).
I feel like the analytical case for at least attempting (2) is pretty strong. I will admit that this may validate skepticism to the tune of, ‘We should reduce funding and seriously consider our theory of change’.
If the argument is that no amount of additional funding will justify the ~$100m we’ve already spent, I would argue that it doesn’t need to. Any additional funding only needs to be useful enough to justify itself. Even if the overall forecasting program ends up being overfunded and unjustified, we should treat future funding as independent of any potential bad decisions in the past.
For example, if the program is currently giving us $10m of value (for $100m of funding), and spending an additional $10m would increase that to $30m of value, then we should spend it. Even if the overall program remains a failure, our $10m has given us a 2x return.
Wrt the startup analogy and LLMs, I am not sure it is reasonable to claim that all useful technologies get adopted at this extreme speed, re: seatbelts, childproof medical caps, vaccinations, drunk driving laws, helmets, indoor smoking bans, etc. which all required significant distribution efforts.
To clarify ‘distribution’, I think there’s a difference between ‘everyone has heard of prediction markets’ and ‘we have the right tools to allow policy/decision-makers to adopt prediction markets in their decision-making processes’.
I am intrigued by the ‘solution seeking a problem’ framing. Unlike blockchain, there are a number of domains and decisions that would analytically benefit from more accurate predictions about the future (e.g. issue prioritisation, campaigning decisions in politics, domestic policy decisions, geopolitical actions) even if they have practically not adopted forecasting widely. Better forecasts can prevent worse counterfactual worlds, e.g. by preventing planning mistakes leading to rapid inflation, encouraging better resilience planning, etc.
The claim that ‘Forecasting cannot be useful’ seems to have the plausible alternative claim ‘Forecasting can be useful with better distribution’, i.e. we need to find better ways to get policy- and decision-makers access to low-friction, high-quality, defensible forecasts that they can use. This would still mean that existing funding that is focused on improving the resolution and quality of forecasts needs to re-prioritise and focus on the distribution more, but would not go so far as to say we should stop funding it altogether.
The pattern of ‘analytically useful, but practically unadopted’ is not unprecedented. Seatbelts and childproof medical caps, for example, were both analytically beneficial but did not see widespread adoption till concentrated efforts were made on the distribution front. Clear, measurable benefits followed.
On the solution seeking a problem, I think proponents of blockchain say similar things about how useful it would be if our financial system were predominantly on the blockchain and how much benefit there would be, in particular to the unbanked. They do, in fact, say this.
I never said that forecasting cannot be useful. I think it hasn’t been useful to the tune of the money thrown at it, and I’m skeptical that further investment is going to pay off in usefulness. I was somewhat expecting the defenses of forecasting to be better than they have been. The best one was probably this one by @habryka or another reply by Eva Vivalt on the EA forum.
It’s possible we haven’t been funding the right thing within forecasting. Maybe this weekend I will tally up every grant and investment made into forecasting, but I think it is ~$100m. That said, I think basically any poor funding decision will have people saying something similar to this:
When a startup says that their product hasn’t taken off and all they really need is better marketing to do so, you should run. I think we should say the same thing here, too. If you need to focus on distribution and try to force your product into being used, it probably isn’t that useful. When something is very useful, it gets adopted at extreme speed. The AI companies aren’t spending a lot on marketing. For example, people adopted LLMs, chatbots, and other products extremely quickly, and they were more or less just announced.
Furthermore, prediction markets have been adopted, so it’s hard to say it’s an issue of marketing. Almost everyone important has heard of Polymarket and Kalshi at this point and know how PMs work. And none of them has decided that they would be very useful. It just so happens that they are useful for sports gambling primarily and not for making better decisions.
But… aren’t prediction markets one of the fastest growing industries in the world right now?
On the usefulness of blockchain, I think the analytical case for our financial system being on blockchain is significantly weaker than the analytical case for better forecasts being useful for policymaking.
With my original comment, I was implicitly drawing a distinction between ‘cannot be useful’ and ‘has not been useful’:
If forecasting simply has not been useful, then it leaves us with two possibilities: (1) it cannot be and we should stop funding it; (2) it can be and we should redirect funding to efforts that figure out and solve why it hasn’t been (e.g. distribution).
I feel like the analytical case for at least attempting (2) is pretty strong. I will admit that this may validate skepticism to the tune of, ‘We should reduce funding and seriously consider our theory of change’.
If the argument is that no amount of additional funding will justify the ~$100m we’ve already spent, I would argue that it doesn’t need to. Any additional funding only needs to be useful enough to justify itself. Even if the overall forecasting program ends up being overfunded and unjustified, we should treat future funding as independent of any potential bad decisions in the past.
For example, if the program is currently giving us $10m of value (for $100m of funding), and spending an additional $10m would increase that to $30m of value, then we should spend it. Even if the overall program remains a failure, our $10m has given us a 2x return.
Wrt the startup analogy and LLMs, I am not sure it is reasonable to claim that all useful technologies get adopted at this extreme speed, re: seatbelts, childproof medical caps, vaccinations, drunk driving laws, helmets, indoor smoking bans, etc. which all required significant distribution efforts.
To clarify ‘distribution’, I think there’s a difference between ‘everyone has heard of prediction markets’ and ‘we have the right tools to allow policy/decision-makers to adopt prediction markets in their decision-making processes’.