Suppose there is a consensus belief, and suppose that it’s totally correct. If funders, and more generally anyone who can make stuff happen (e.g. builders and thinkers), use this totally correct consensus belief to make local decisions about where to allocate resources, and they don’t check the global margin, then they will in aggregrate follow a portfolio of strategies that is incorrect. The make-stuff-happeners will each make happen the top few things on their list, and leave the rest undone. The top few things will be what the consensus says is most important——in our case, projects that help if AGI comes within 10 years. If a project helps in 30 years, but not 10 years, then it doesn’t get any funding at all. This is not the right global portfolio; it oversaturates fast interventions and leaves slow interventions undone.
Obvious enough but worth saying explicitly: It’s not just impacts on epistemology, but also collective behavior. From https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#My_views_on_strategy :