Of course, there is a huge problem with expert surveys—at the meta-level they have a very poor predictive track record. There is the famous example that Stuart Russell likes to cite, where rutherford said “anyone who looked for a source of power in the transformation of the atoms was talking moonshine”—a day before leo szilard created a successful fission chain reaction. There is also the similar example of the Wright Brothers—some unknown guys without credentials claim to have cracked aviation when recognized experts like Langley have just failed in a major way and respected scientists such as Lord Kelvin claim the whole thing is impossible. The wright brothers then report their first successful manned flight and no newspaper will even publish it.
Of course, there is a huge problem with expert surveys—at the meta-level they have a very poor predictive track record. There is the famous example that Stuart Russell likes to cite, where rutherford said “anyone who looked for a source of power in the transformation of the atoms was talking moonshine”—a day before leo szilard created a successful fission chain reaction. There is also the similar example of the Wright Brothers—some unknown guys without credentials claim to have cracked aviation when recognized experts like Langley have just failed in a major way and respected scientists such as Lord Kelvin claim the whole thing is impossible. The wright brothers then report their first successful manned flight and no newspaper will even publish it.