Zealous evaluation also serves to entrench the power of existing academic hierarchies. As a case study, it’s instructive to consider how the academic ML community oriented towards the concept of AGI overall. In some ways, AGI is a very simple concept: AIs that can generalize to a comparable extent as humans. Yet what we saw when the AI alignment community interacted with the academic ML community was something akin to an immune system response. Most scoffed—like Andrew Ng, who claimed that worrying about AGI risk was “like worrying about overpopulation on Mars”.
The reaction of the ML community is a textbook example of Strevens’ “iron law”, no? Until recently, there was no empirical validation of the concept of AGI, and so it was rejected by people socialized into the “iron law” norm of science. (The iron law as conceived by Strevens only admits empirical tests, not “rigorous proofs” as you include in your definition of the core scientific norm here.)
You seem to think that what is misguided is the way scientific institutions implement the iron law (requiring self-contained/justified incremental paper-sized units), whereas I (with Strevens, in my reading) think the iron law is itself poorly suited for governing short-run practical discernment (as opposed to growing the coral reef of empirical observations for long-run benefit).
An instructive recent case of the iron law hamstringing practical discernment was the bizarrely slow acceptance of COVID-19′s airborne transmission. The short version of why that happened is that scientists are not trained to step back and ask how likely a given hypothesis is, all things considered. They’re trained to dismiss deviations from the status quo default hypothesis unless there is decisive new evidence against it. (The reason for that norm, as Strevens explains, is how well it incentivizes painstaking experiments and other empirical observation.)
The reason I’m trying to sharpen this point is that I think it essential context for decisionmakers trying to sort through the he-said-she-said of disputes between rationalists, say, and academic scientists. Those refereeing these disputes should expect academics to dismiss unorthodox ideas absent decisive empirical tests. Such “iron law” enforcement serves a long-term scientific purpose but should not unduly muddle the discernment we need now.
The reaction of the ML community is a textbook example of Strevens’ “iron law”, no? Until recently, there was no empirical validation of the concept of AGI, and so it was rejected by people socialized into the “iron law” norm of science. (The iron law as conceived by Strevens only admits empirical tests, not “rigorous proofs” as you include in your definition of the core scientific norm here.)
You seem to think that what is misguided is the way scientific institutions implement the iron law (requiring self-contained/justified incremental paper-sized units), whereas I (with Strevens, in my reading) think the iron law is itself poorly suited for governing short-run practical discernment (as opposed to growing the coral reef of empirical observations for long-run benefit).
An instructive recent case of the iron law hamstringing practical discernment was the bizarrely slow acceptance of COVID-19′s airborne transmission. The short version of why that happened is that scientists are not trained to step back and ask how likely a given hypothesis is, all things considered. They’re trained to dismiss deviations from the status quo default hypothesis unless there is decisive new evidence against it. (The reason for that norm, as Strevens explains, is how well it incentivizes painstaking experiments and other empirical observation.)
The reason I’m trying to sharpen this point is that I think it essential context for decisionmakers trying to sort through the he-said-she-said of disputes between rationalists, say, and academic scientists. Those refereeing these disputes should expect academics to dismiss unorthodox ideas absent decisive empirical tests. Such “iron law” enforcement serves a long-term scientific purpose but should not unduly muddle the discernment we need now.