Perhaps there does exist a route towards resolving this situation.
Suppose Eliezer has a coin for one week, during which he flips it from time to time. He doesn’t write down the results, record how many times he flips it, or even keeps a running mental tally. Instead, at the end of the week, relying purely upon his direct memory of particular flips he can remember, he makes an estimate: “Hmm, I think I can remember about 20 of those flips fairly accurately and, of those 20 flips, I have 90% confidence that 15 of them came up heads.”
The coin is then passed to Robin, who does the same exercise the following week. At the end of that week, Robin thinks to himself “I think I can remember doing about 40 flips, and I have 80% confidence that 10 of them came up heads.”
They then meet up and have the following conversation:
Eliezer: 75% chance of a head
Robin: 25% chance of a head, not taking your data into account yet, just mine.
Eliezer: Ok, so first level of complexity is we could just average that to get 50%. But can we improve upon that?
Robin: My sample size was 40
Eliezer: My sample size was 20 so, second level of complexity, we could add them together to get 25 heads of out 60 flips, giving 42% chance of a head
Robin: Third level of complexity, how confident are you about your numbers? I’m 80% confident of mine
Eliezer: I’m 90% confident of mine. So using that as a weighting would give us (0.9x15+0.8x10)/(0.9x20+0.8x40) which is 21.5 out of 50 which is 43% chance of a head.
Robin: But Eliezer, you always overestimate how confident you are about your memory, whereas I’m conservative. I don’t think your memory is any better than mine. I think 42% is the right answer.
Eliezer: Ok, let’s go to level 4. Can we find some objective evidence? Did you do any of your flips in the presence of a third party? I can remember 5 incidents where someone else saw the flip I did. We could take a random sampling of my shared flips and then go ask the relevant third parties for confirmation, then do the same for a random sample of your shared flips, and see if your theory about our memories is bourne out.
In the end, as long as you can trace back at least some (a random sampling) of the facts people are basing their estimates upon to things that can be checked against reality, you should have some basis to move forwards.
Perhaps there does exist a route towards resolving this situation.
Suppose Eliezer has a coin for one week, during which he flips it from time to time. He doesn’t write down the results, record how many times he flips it, or even keeps a running mental tally. Instead, at the end of the week, relying purely upon his direct memory of particular flips he can remember, he makes an estimate: “Hmm, I think I can remember about 20 of those flips fairly accurately and, of those 20 flips, I have 90% confidence that 15 of them came up heads.”
The coin is then passed to Robin, who does the same exercise the following week. At the end of that week, Robin thinks to himself “I think I can remember doing about 40 flips, and I have 80% confidence that 10 of them came up heads.”
They then meet up and have the following conversation:
Eliezer: 75% chance of a head
Robin: 25% chance of a head, not taking your data into account yet, just mine.
Eliezer: Ok, so first level of complexity is we could just average that to get 50%. But can we improve upon that?
Robin: My sample size was 40
Eliezer: My sample size was 20 so, second level of complexity, we could add them together to get 25 heads of out 60 flips, giving 42% chance of a head
Robin: Third level of complexity, how confident are you about your numbers? I’m 80% confident of mine
Eliezer: I’m 90% confident of mine. So using that as a weighting would give us (0.9x15+0.8x10)/(0.9x20+0.8x40) which is 21.5 out of 50 which is 43% chance of a head.
Robin: But Eliezer, you always overestimate how confident you are about your memory, whereas I’m conservative. I don’t think your memory is any better than mine. I think 42% is the right answer.
Eliezer: Ok, let’s go to level 4. Can we find some objective evidence? Did you do any of your flips in the presence of a third party? I can remember 5 incidents where someone else saw the flip I did. We could take a random sampling of my shared flips and then go ask the relevant third parties for confirmation, then do the same for a random sample of your shared flips, and see if your theory about our memories is bourne out.
In the end, as long as you can trace back at least some (a random sampling) of the facts people are basing their estimates upon to things that can be checked against reality, you should have some basis to move forwards.