I need help: Device of imaginary results by I J Good
In the chapter 5 of the Probability Theory: Logic of Science you can read about so-called device of imaginary results which seems to go back to the book of I J Good named Probability and the Weighing of Evidence.
The idea is simple and fascinating:
1) You want to estimate your probability of something, and you know that this probability is very, very far from 0.5. For the sake of simplicity, let’s assume that it’s some hypothesis A and P(A|X) << 0.5
2) You imagine the situation where the A and some well-posed alternative ~A are the only possibilities.
(For example, A = “Mr Smith has extrasensory perception and can guess the number you’ve written down” and ~A = “Mr Smith can guess your number purely by luck”. Maybe Omega told you that the room where the experiment is located makes it’s impossible for Smith to secretly look at your paper, and you are totally safe from every other form of deception.)
3) You imagine the evidence which would convince you otherwise: P(E|A,X) ~ 1 and P(E|~A,X) is small (you should select E and ~A that way that it’s possible to evaluate P(E|~A,X) )
4) After a while, you feel that you are truly in doubt about A: P(A|E1,E2,..., X) ~ 0.5
5) And now you can backtrack everything back to your prior P(A|X) since you know every P(E|A) and P(E|~A).
After this explanation with the example about Mr Smith’s telepathic powers, Jaynes gives reader the following exercise:
Exercise 5.1. By applying the device of imaginary results, find your own strength of
belief in any three of the following propositions: (1) Julius Caesar is a real historical
person (i.e. not a myth invented by later writers); (2) Achilles is a real historical person;
(3) the Earth is more than a million years old; (4) dinosaurs did not die out; they are
still living in remote places; (5) owls can see in total darkness; (6) the configuration of
the planets influences our destiny; (7) automobile seat belts do more harm than good;
(8) high interest rates combat inflation; (9) high interest rates cause inflation.
I have trouble tackling the first two propositions and would be glad to hear your thoughts about another seven. Anybody care to help me?
(I decided not to share details of my attempt to solve this exercise unless asked. I don’t think that my perspective is so valuable and anchoring would be bad.)
UPD: here is my attempt to solve the Julius Caesar problem.
(As promised, I post description of my attempts here)
Case 1: Julius Caesar. The existence of Caesar seems very likely for me. Therefore, I will think about evidence that would convince me that Caesar is a myth.
I decided that the following three pieces of evidence will be enough for me to start doubting existence of Julius Caesar:
1) I don’t know anything about the process of burial of the Roman emperors. Hence it wouldn’t be inconsistent to assume that there is an official “emperor’s tomb”, a luxurious necropolis where the crypts of every emperor are located. Having assumed that this true, I imagine a discovery saying that the crypt of Julius Caesar is missing, or his tomb is empty, or the body inside couldn’t belong to Julius, or something along those lines.
2) Similarly, the existence of a reliable independent Arabian historian similar to Herodotus wouldn’t contradict my worldview. So I can assume that there indeed was such a historian. If his book about Roman Empire had failed to mention Julius Caesar at all, that would have been an evidence pointing to Julius being a myth.
3) Why can’t I imagine the second Arabian history book failing to mention Caesar?
Having imagined all that, I decided that it would be enough for me to start doubting Julius Caesar’s existence.
Let’s turn to the probabilities.
1) P(no crypt|no Julius Caesar) ~ 1. P(no crypt|Julius Caesar) = ?
It doesn’t look like there’s some way to easily estimate that quality. I notice that I’m confused, but let’s try anyway. It’s possible to convert this probability into the relative frequency: there were cases of some monarch’s bodies having been removed from their graves. I know two such cases: False Dmitriy I of Russian Tsars and Akhenaten of the Eighteenth dynasty of Egypt. Counting the other people of their lineage gives the approximate figure around 2⁄350 which I consider kind of satisfying, though it appears to be somewhat higher than desired. However, I had to consult Wikipedia to get this estimate, and that kind of violates the “no gathering other relevant real-world data” rule.
2) and 3) P(no mention of Caesar in the Arabic book|no Julius Caesar) ~ 1 P(no mention of Caesar in the Arabic book|Julius Caesar) = ?
I have trouble estimating that at all. I have no slightest idea how to estimate this without a lot of imaginary betting, and imaginary betting kind of defeats the whole point. Why shouldn’t I estimate P(Julius Caesar existed) via betting?
Well, betting makes sense only when my utility function is linear in money, and that holds only when the probabilities are sufficiently close to 0.5. Maybe I should break my prior into several parts via Good’s device and then estimate the parts via betting.
It seems plausible. However, I think that if Jaynes had meant his exercise to be done in that way, he would have explicitly told so.
I notice that I’m confused. Let’s try to find my mistake:
1) The alternative hypothesis “Julius Caesar existed” is too vague, and hence it is difficult to come up with the conditional probabilities. It seems likely, but I can’t come up with something better.
2) The experiment is bad. I should think about something else: for example, I draw a random portrait of Roman emperor from the urn containing portraits of every Roman emperor, how many non-Caesar’s portraits are enough? Coming up with something like that seems very difficult for me. I’m very bad at selling non-apples.
This is a reason I wanted to find a new perspective and don’t wanted to spoil anyone by anchoring him/her to my vision.
3) The evidence is bad. I should come up with some new evidence such that I can calculate the probability P(E|Julius Caesar)
4) There is a good way to estimate conditional probabilities, I just missed it.
I haven’t worked much with the other cases, but it seems that it would be difficult to calculate their relevant probabilities as well (how can I calculate anything about geology in (3), for example?). So I think that I’ve misunderstood something, and maybe someone there can describe the correct way of doing this exercise to me.
A couple points.
Jaynes would emphasize background information I that provides the contextual information that allows a meaningful estimate. Without identifying and specifying that contextual information as something specific, the mind spins round and round on hypotheticals, and you will “notice that you are confused”.
I’ve found it helpful to specify that background knowledge, even if I’m not committed to it. Then do the math for alternatives scenarios as well. You can’t estimate until you narrow down reality enough to have some meaningful scenario where you can make causal inferences.
Another point Jaynes makes. You have to be very clear on what the facts are. Many “facts” are reports about facts by others, continuing a chains of artifacts, reports, and communications. Is the evidence the book you see in front of you, or the assumption that the reported historical and translational lineage of the the book in front of you is accurate back to the original author?
For example, you say
P(no crypt|no Julius Caesar, All known reports on Caesar) is a very different animal than P(no crypt|no Julius Caesar, No known reports of a Julius Caesar exist). If you have a zillion reports on Caesar, and he didn’t exist, it would seem that someone did a good job pretending that he did exist. Wouldn’t they want make a pretend crypt too? Maybe not, but on balance, probably. This can be more causally circumscribed by I = All reported contemporaneous reports occurred.
It seems to me that it would be more effective to work from evidence that you have encountered personally or in the case of hypothetical evidence, could have hypothetically encountered. In the case of historical figures, unless you happen to be an archaeologist yourself, the majority of the evidence you have is through secondary and tertiary sources. For example, if a publication alleged that Julius was a title, not a name, and was used by many Caesars, and thus many acts attributed to the person Julius Caesar were in fact performed by separate individuals, you would probably have little reason to believe this. If a great number of publications, especially from respected organizations and individuals within the archaeological field posited the same thing, it might be sufficient to give you pause (it would for me in any case).
It seems to me that the intent here is to evaluate a prior based on a great quantity of weak evidence. Both weak evidence directly to the contrary in sufficient quantity, or evidence that discredits the sources used to generate your prior should sufficiently alter the probability to create doubt.
I have the same reservation regarding the probability regarding propositions 1) and 2) as army1987. In particular, I find that the probability that all writings regarding the aforementioned people are true is exceedingly low for both, but the probability that some person existed bearing that name, who performed at least one action or bore one trait that was subsequently recorded is rather high. Considering that this is meant to be an exercise on evaluating one’s priors (or at least that is how it appears to me), I would consider choosing one interpretation or the other and work from that. If you feel the need, simply try both interpretations or find a middle ground that you feel comfortable with. If this is not your issue with the propositions, then I would require more information on your attempts to solve the exercise in order to provide meaningful feedback.
Here are the details
I think it would be good. It narrows down the issues required for the response. And it demonstrates effort to solve the problem on your part.
To be brief—“explain this to me, but I won’t show any evidence of making the effort myself or attempting to make it easier for you to respond to me”—doesn’t feel so respectful of the effort your asking of others, and it’s so broad that I don’t expect great bang for the buck with the effort required.
If you’ve got a question, make the issue as clear as you can make it, instead of treating us like guinea pigs in your experiment where you’re trying to avoid bias.
I don’t mean this in a big negative way (I don’t do the emotion free writing tone so popular here). I see that you’ve spent effort detailing your question, are asking an honest one, and have a reasonable reason besides laziness not to elaborate further (avoid bias).
But asking for input is asking for a favor, and I tend to expect people asking for favors to make my fulfillment of their needs as easy and productive for me as possible.
Another way to express it, is that I’d be writing expressly in the dark of what the real issue is. Maybe it’s a control thing. “Do this so I can analyze the results” doesn’t have a lot of appeal.
I guess I wasn’t brief. It’s an interesting question, but as posed, it left me with a shrug.
Your point of view makes sense. Hence, I’ve written about my partial results: http://lesswrong.com/lw/h54/i_need_help_device_of_imaginary_results_by_i_j/8pow
In my defence, I will say that I decided not to share my approach not because of “we should reduce bias no matter what” reasoning, but because I truly think that my approach is (a) wrong and (b) attractive, and (c) the problem is difficult; hence, the information about can be dangerous.
But it makes sense to post this sort of information in comments anyway, so I did.
In (1) and (2), how you count a character if it was inspired to a real person of the same name but a sizeable fraction of the stories about him are false?
(4) sounds like a shibboleth to test whether you’ve read the title text of http://xkcd.com/867/.
(5) is logically impossible, unless “see” means something other than ‘detect light’ and/or “darkness” means something other than ‘lack of light’.
(7), (8) and (9) in principle might be correct in certain circumstances and wrong in others.
I think that your concerns are valid, but we should focus on the problems, assuming that we are in the least convenient possible world if needed.
(1) and (2): I believe, we should consider him to be real. I think that Jaynes meant “person is real and most stories about him are true”, however.
(4): I think Jaynes talked not about “dinosaurs by definition” but about actual dinosaurs. Loch Ness Monster is a surviving plesiosaur or something.
(5): Why “see” should mean “detect light”? If I taboo word “see”, I will get something like that: “Owls are able to avoid obstacles and find food in the total darkness”, not “Owls are able to detect light in the total darkness”.
Slight quibble about the fifth problem: I see what you’re doing, and I mostly agree with it, but I think you’re stretching the taboo thing a bit. By those criteria, eyeless people, cats and bats can see in total darkness.
(I’m using “eyeless” to avoid trouble with what “blind” means, and also to get around the fact that “total darkness” isn’t a well-defined concept; taken literally it would mean “no variation in the electromagnetic field along the time axis”, which even if it were physically possible would probably require a very life-threatening environment.)
Depending on the environment, one can use touch, smell, sound, taste (even electricity if you’re a shark) to avoid obstacles and find food, and “see” can be colloquially used to describe this, especially where the particular sense is sufficiently acute and/or predominant to resemble the usual importance of sight for humans (as it’s often the case with bats and cetaceans), but it doesn’t seem that’s what the exercise meant. (BTW, IIRC owls can in fact hunt mice using only sound, though I suspect they can’t navigate much around silent obstacles. But the feat was demonstrated by filming the action with IR—having determined first that owls can’t see IR despite their very good low-light vision—which brings us back to that “total darkness” problem.)
If you define “can see” as “able to avoid obstacles and find food”, then you’re pretty much forced to conclude every motile living being can see.
How are birds not actual dinosaurs? :-)
I agree that he probably was thinking of non-avian dinosaurs (and possibly was intending “die out” more broadly than ‘die without descendants’), but…
A problem with this is that humans often exhibit base rate neglect, and so may overestimate small probabilities in this sort of exercise .
I am not sure I understood you correctly.
Do you mean that since humans tend to overvalue weight of the evidence and undervalue how the hypothesis was likely in the first place, I could overestimate P(A|X) by not requiring enough evidence and deciding that P(A|E,X)=0.5 earlier than justified?
Exactly.
What evidence would convince you that there is a very small chance that Julius Cesear was a real person? What evidence would convince you that Achillies was a real person? (with whatever certainty you call “certain”)
This exercise is not about “certain” (P ~ 1), it’s about “doubt” (P ~ 1⁄2).
Concerning your question, I can say following:
1) Birth records/absence of such.
2) Place of burial/absence of such.
3) Mention of Achilles in the independent sources/lack of such for Julius Caesar (if some reliable Egyptian historian describes the century between 100 BC and 0 BC as the “epoch of a perfect stability of Roman Empire”, it would speak against both coup-d’etat by Julius and coup-d’etat by Brutus, which is part of the legend of Julius)
4) Evidence about reliability of Homer (can legend of Odyssey be some sort of political satire in disguise?)/Evidence about unreliability of historians describing Julius Caesar.
5) Social evidence: some person respected by me starts to believe in it.
I believe I would start taking existence of Achilles/non-existence of Julius Caesar seriously after those things: P ~ 1⁄2.
About “certainty”: I think large amount of independent Arabian/Egyptian/whatever books will do the trick.
Would it take all of those to cause you to doubt your current belief? The purpose is to figure out how strong of evidence you need to doubt your current belief, and use that as a measuring tool for how strongly you believe...