Now imagine that knowing how to read a study is, in fact, a very important rationality skill. [...] I think it’s very possible that hypothetical Deworming CFAR would flinch away from the idea “maybe we should teach people to read studies”. Or they’d run a pilot program, notice that teaching people to read studies tends to make them less enthusiastic about deworming, and say “well, guess that learning how to read studies actually makes people more irrational.”
My impression is that CFAR specifically intends to focus on rationality skills relevant to evaluating AI risk. Obviously this puts it at elevated risk of the second failure mode you mention (mistaking correct updating away from AI risk for AI risk not being a thing), but it seems to me like if CFAR can’t pass that elementary a test of rationality, it fails even without this sort of conflict of interest adjacent problem.
My impression is that CFAR specifically intends to focus on rationality skills relevant to evaluating AI risk. Obviously this puts it at elevated risk of the second failure mode you mention (mistaking correct updating away from AI risk for AI risk not being a thing), but it seems to me like if CFAR can’t pass that elementary a test of rationality, it fails even without this sort of conflict of interest adjacent problem.