My immediate critique would be step 7: insofar as people are updating today on experiments which are bullshit, that is likely to slow us down during early crunch, not speed us up. Or, worse, result in outright failure to notice fatal problems. Rather than going in with no idea what’s going on, people will go in with too-confident wrong ideas of what’s going on.
To a perfect Bayesian, a bullshit experiment would be small value, but never negative. Humans are not perfect Bayesians, and a bullshit experiment can very much be negative value to us.
Yep, I’ll bite the bullet here. This is a real problem and partly my motivation for writing the perspective explicitly.
I think people who are “in the know” are good at not over-updating on the quantitative results. And they’re good at explaining that the experiments are weak proxies which should be interpreted qualitatively at best. But people “out of the know” (e.g. junior ai safety researches) tend to overupdate and probably read the senior researchers as professing generic humility.
I would guess that even the “in the know” people are over-updating, because they usually are Not Measuring What They Think They Are Measuring even qualitatively. Like, the proxies are so weak that the hypothesis “this result will qualitatively generalize to <whatever they actually want to know about>” shouldn’t have been privileged in the first place, and the right thing for a human to do is ignore it completely.
My immediate critique would be step 7: insofar as people are updating today on experiments which are bullshit, that is likely to slow us down during early crunch, not speed us up. Or, worse, result in outright failure to notice fatal problems. Rather than going in with no idea what’s going on, people will go in with too-confident wrong ideas of what’s going on.
To a perfect Bayesian, a bullshit experiment would be small value, but never negative. Humans are not perfect Bayesians, and a bullshit experiment can very much be negative value to us.
Yep, I’ll bite the bullet here. This is a real problem and partly my motivation for writing the perspective explicitly.
I think people who are “in the know” are good at not over-updating on the quantitative results. And they’re good at explaining that the experiments are weak proxies which should be interpreted qualitatively at best. But people “out of the know” (e.g. junior ai safety researches) tend to overupdate and probably read the senior researchers as professing generic humility.
I would guess that even the “in the know” people are over-updating, because they usually are Not Measuring What They Think They Are Measuring even qualitatively. Like, the proxies are so weak that the hypothesis “this result will qualitatively generalize to <whatever they actually want to know about>” shouldn’t have been privileged in the first place, and the right thing for a human to do is ignore it completely.