I guess the moral is “Don’t trust anyone but a mathematician”?
Safety in numbers? ;)
Perhaps it’s useful to distinguish between the frontier of science vs. established science. One should expect the frontier to be rather shaky and full of disagreements, before the winning theories have had time to be thoroughly tested and become part of our scientific bedrock. There was a time after all when it was rational for a layperson to remain rather neutral with respect to Einstein’s views on space and time. The heuristic of “is this science established / uncontroversial amongst experts?” is perhaps so boring we forget it, but it’s one of the most useful ones we have.
I have an image of Eliezer queued up in a coffee shop, guiltily eyeing up the assortment of immodestly priced sugary treats. The reptilian parts of his brain have commandeered the more recently evolved parts of his brain into fervently computing the hedonic calculus of an action that other, more foolish types, might misclassify as a sordid instance of discretionary spending. Caught staring into the glaze of a particularly sinful muffin, he now faces a crucial choice. A cognitive bias, thought to have been eradicated from his brain before the SIAI was founded, seizes its moment. “I’ll take the triple chocolate muffin thank you” Eliezer blurts out. “Are you sure?” the barista asks. “Well I can’t be 100% sure. But the future of intergalactic civilizations may very well depend on it!”