The heuristic of averaging beliefs is clearly poor. We all accept that in principle we should agree on the same truth (subject only to differences in irreducible priors) after we share our evidence and update on the union; if we just average the beliefs of some people (expert or otherwise), we under- or over-count some evidence. But good luck finding a willing expert partner for such an exercise—one both capable of really doing it, and generous enough to go to the expense.
You seem to suggest a slightly more subtle heuristic for combining conflicting experts’ views, but it’s still imperfect.
For example, suppose there are contested facts X and Y. Suppose all 3 experts agree that X->Q and Y->Q, but 1 holds (X and not Y), 1 (Y and not X), and the other (not X and not Y). I claim that this is effectively a 2⁄3 vote for Q, even though there’s a 2⁄3 vote against both X and Y, although of course as a practical matter such wild disagreement amongst “experts” makes me suspicious of their credentials :) I think this is acceptable even if X is christianity, Y is islam, and Q is “some sort of afterlife”. I just wouldn’t make the mistake of doing no evaluation of the evidence for X and Y myself.
Here’s where your heuristic would work: 1⁄3 expert holds “A and A->G”, 1⁄3 holds “B and B->G”, and 1⁄3 denies all 4 statements. This should be interpreted at best as a 1⁄3 vote for G (maybe you think it’s no evidence at all?)
I would say it requires “A and A->G and not B” and “B and B->G and not A”
such wild disagreement amongst “experts” makes me suspicious of their credentials
I think that’s part of what I’m trying to quantify here. when there’s little direct evidence(or we don’t understand it ourselves), and a lot of thinking, experts are pretty much defined by the opinions of other experts. If we want to guess at the reliability of their conclusions, the only track record we have is how often other experts agree with them.
The heuristic of averaging beliefs is clearly poor. We all accept that in principle we should agree on the same truth (subject only to differences in irreducible priors) after we share our evidence and update on the union; if we just average the beliefs of some people (expert or otherwise), we under- or over-count some evidence. But good luck finding a willing expert partner for such an exercise—one both capable of really doing it, and generous enough to go to the expense.
You seem to suggest a slightly more subtle heuristic for combining conflicting experts’ views, but it’s still imperfect.
For example, suppose there are contested facts X and Y. Suppose all 3 experts agree that X->Q and Y->Q, but 1 holds (X and not Y), 1 (Y and not X), and the other (not X and not Y). I claim that this is effectively a 2⁄3 vote for Q, even though there’s a 2⁄3 vote against both X and Y, although of course as a practical matter such wild disagreement amongst “experts” makes me suspicious of their credentials :) I think this is acceptable even if X is christianity, Y is islam, and Q is “some sort of afterlife”. I just wouldn’t make the mistake of doing no evaluation of the evidence for X and Y myself.
Here’s where your heuristic would work: 1⁄3 expert holds “A and A->G”, 1⁄3 holds “B and B->G”, and 1⁄3 denies all 4 statements. This should be interpreted at best as a 1⁄3 vote for G (maybe you think it’s no evidence at all?)
I do like your heuristic.
I would say it requires “A and A->G and not B” and “B and B->G and not A”
I think that’s part of what I’m trying to quantify here. when there’s little direct evidence(or we don’t understand it ourselves), and a lot of thinking, experts are pretty much defined by the opinions of other experts. If we want to guess at the reliability of their conclusions, the only track record we have is how often other experts agree with them.