III) If lots of (smart/senior) people seem to dismiss an idea, assume there’s something wrong with it [even if most of the smart/senior people are doing local work that makes it locally disincentivized for them to seem not to dismiss that idea, eg because it would annoy Ai companies].
I do something like: Model the incentives placed on those smart/senior people, including by looking at them differentially (how is the average smart person at a lab thinking differently from an average smart person who is independent/at MIRI/etc), then also adjust for the memetic tug from a bunch of the smart people running thought cycles partly tethered to a conclusion for non-truth-tracking reasons?
Seems somewhat risky, as it’s not hard to end up biased about the effect size and dismiss things you shouldn’t, but combined with “and check the object level reasoning of some of the people who think it’s a bad idea” this is the best patch I have so far.
I do something like: Model the incentives placed on those smart/senior people, including by looking at them differentially (how is the average smart person at a lab thinking differently from an average smart person who is independent/at MIRI/etc), then also adjust for the memetic tug from a bunch of the smart people running thought cycles partly tethered to a conclusion for non-truth-tracking reasons?
Seems somewhat risky, as it’s not hard to end up biased about the effect size and dismiss things you shouldn’t, but combined with “and check the object level reasoning of some of the people who think it’s a bad idea” this is the best patch I have so far.