But, of course, the mathematics of probability theory don’t work that way. A hypothesis, such as that the apparent burglary in Filomena Romanelli’s room was staged—doesn’t get points for its ability to explain the data unless it does so better than its negation. And, in the absence of the assumption that Knox and Sollecito are guilty—if we’re presuming them to be innocent, as the law requires, or assigning a tiny prior probability to their guilt, as epistemic rationality requires—this contest is rigged. The standards for “explaining well” that the fake-burglary hypothesis has to meet in order to be taken seriously are much higher than those that its negation has to meet, because of the dependence relation that exists between the fake-burglary question and the murder question.
This isn’t quite true. If the prior probability of being a murderer is 1 in 10^6, and I can find 30 things that are explained twice as well by the murder hypothesis as the non-murder hypothesis, then the posterior probability of being a murderer is 99.9%, in the absence of mitigating factors (since 2^30/10^6 is about 1000.) So, many pieces of weak evidence for an unlikely proposition can still establish that proposition.
Probably superfluous nitpicking: you can build a strong case even with partially-interdependent pieces of evidence, you will just need more of them since you have to work off their conditional probabilities (which is mathematically equivalent to “splitting” them into independent pieces of evidence).
So, many pieces of weak evidence for an unlikely proposition can still establish that proposition.
This doesn’t contradict anything in the paragraph you quoted. (If you don’t mind, tell me where you thought the contradiction was, so that I can explain further.)
The standards for “explaining well” that the fake-burglary hypothesis has to meet in order to be taken seriously are much higher than those that its negation has to meet, because of the dependence relation that exists between the fake-burglary question and the murder question.
My impression was that you were claiming that, since the fake burglary hypothesis would imply murder, evidence must be extremely strong to be counted in favor of fake burglary. But I may have misunderstood you. At any rate, you elsewhere state that
Of their 427-page report, Massei and Cristiani devote approximately 20 pages (mainly pp. 27-49) to their argument that the burglary was staged by Knox and Sollecito [...] if they were really able to demonstrate this, they would scarcely have needed to bother writing the remaining 400-odd pages of the report!
If you agree with my point, then I don’t see how you can find it odd that they would feel obliged to include more in their report than just the claim that the burglary was faked. Like you said, even if the evidence is fairly strong in favor of this assertion, far more evidence would be needed to convict those two of murder, which is presumably the point of the remaining 400 pages.
My impression was that you were claiming that, since the fake burglary hypothesis would imply murder, evidence must be extremely strong to be counted in favor of fake burglary
My claim is that to reach a desired level of certainty about the burglary being faked, you would need evidence of approximately the same strength required to reach the same level of certainty about murder. (In other words, that the prior probability of fake burglary is roughly the prior probability of murder.)
Like you said, even if the evidence is fairly strong in favor of [fake burglary], far more evidence would be needed to convict those two of murder
This is the very opposite of what I said! What I said was that if you knew with high confidence that the burglary was fake, then you would need almost no additional evidence to convict of murder.
Okay, so we seem to be in complete agreement about how the math works out. If so, then I’m confused as to why you object so strongly to the prosecution’s argument on purely mathematical grounds; I haven’t read their argument myself, so it’s entirely possible that the argument itself is weak in some way, but I think that right now we’re just talking about the math.
If we ignore their specific language, the plan of coming up with ~20 pieces of moderate evidence is a perfectly reasonable strategy for correctly establishing guilt, assuming that there is absolutely no mitigating evidence. Your complaint seems to be that they use different language/notation than you and I do to talk about evidence, which seems hardly fair.
Although I would also note that since humans are bad at intuitively distinguishing between moderate evidence for and moderate evidence against a hypothesis, trying to find many pieces of weak evidence is probably not a good strategy if the goal is to get humans to correctly decide the accuracy of an assertion.
ETA: By the way, I’ve been working under the assumption, based on the tone of the original post, that you think there are serious mathematical flaws in the prosecutions argument. If that’s not the case, and you just wanted to use this case as a point of illustration, then I apologize for the confusion.
What I gather is that the prosecution concludes, after the first twenty pages of the brief that discuss the break-in exclusively that the break in was almost certainly staged by Knox and Sallecito. But if they really thought that they would have already more or less made the case the Knox and Sallecito are guilty and the remaining 380 pages would be unnecessary. So the prosecution can’t be weighing the evidence correctly.
Okay, so we seem to be in complete agreement about how the math works out. If so, then I’m confused as to why you object so strongly to the prosecution’s argument on purely mathematical grounds; I haven’t read their argument myself, so it’s entirely possible that the argument itself is weak in some way, but I think that right now we’re just talking about the math.
If I may presume to diagnose your confusion, it seems that you’re compartmentalizing between “mathematical” aspects of an argument and “other” aspects. But I’m not. I’m taking it for granted that “the math” is the argument. Probability theory is a mathematical formalization of the process of argument and inference. It isn’t just a cool gadget that one throws in on special occasions.
So, I don’t object to Massei and Cristiani’s argument on “purely mathematical grounds”. I simply object to it, period—and in this post I have used mathematical language to describe, in precise terms, what my objection is.
(And I expected readers to assume, given my previous writing on the case, that this particular point was far from my only objection to Massei and Cristiani’s 427-page argument that Knox and Sollecito killed Kercher; hence I was not expecting replies of the form “well, but they might have other good evidence that Knox and Sollecito are guilty”. They don’t; we’ve already covered that.)
If we ignore their specific language, the plan of coming up with ~20 pieces of moderate evidence is a perfectly reasonable strategy for correctly establishing guilt, assuming that there is absolutely no mitigating evidence. Your complaint seems to be that they use different language/notation than you and I do to talk about evidence, which seems hardly fair.
I honestly have no idea where you’re getting this from. I don’t know of any passage in the post where I complained about Massei and Cristiani’s choice of language; and nor did I attempt to argue (as several people seem to have thought I did) against a strategy of proving one’s case by adducing a large amount of weak evidence in one’s favor (although as a matter of fact I do believe that is the wrong type of argument to expect for a proposition of this sort, and that people have probably been misled by detective stories and the like into thinking it a reasonable strategy, when it would actually be very difficult to make work in practice—that however would be the topic of a separate post, and isn’t addressed in this one).
My criticism of Massei and Cristiani in this post is really quite simple, or so I thought: the type of evidence that they cite to prove that the burglary was faked suggests that they did not realize how high the burden of proof for this proposition was—that, just to prove the burglary was faked, they needed evidence of the same level of strength as would be required to directly prove Knox and Sollecito guilty of murder.
Quite frankly, I’m baffled at how this point seems to have gotten lost, because I thought I was emphatic and indeed repetitious about it in the post.
If we ignore their specific language, the plan of coming up with ~20 pieces of moderate evidence is a perfectly reasonable strategy for correctly establishing guilt, assuming that there is absolutely no mitigating evidence. Your complaint seems to be that they use different language/notation than you and I do to talk about evidence, which seems hardly fair.
I think the assertion is that they appear to be coming up with ~20 pieces of evidence and then trying to say that each piece is very strong—or at least, they have done so for the burglary hypothesis, so they might be doing so for the other pieces of evidence too. Naturally, their methods of making each piece look very strong are flawed.
You almost pinpointed the reason why this happening here:
trying to find many pieces of weak evidence is probably not a good strategy if the goal is to get humans to correctly decide the accuracy of an assertion.
Humans are bad at intuitively handling evidence in general. There is a possibility that this case suffers from a serious malady: presiding judge Massei has decided the correct, accurate decision in this matter is that Knox and Sollecito are guilty, and has strategically prepared the judge’s report to get people to decide this way. This hypothesis explains why the judge has produced such a weighty document when 20 pages of it would have sufficed.
“my claim is that to reach a desired level of certainty about the burglary being faked, you would need evidence of approximately the same strength required to reach the same level of certainty about murder.”
This assumes that the burglary being faked is the only piece of evidence. If we have three sets of evidence, and each one suggests a 90% chance of guilt, and each is independent of the other, then we have probability (10:1) x (10:1) x (10:1) = (1000:1). No one set of evidence needs to have a (1000:1) probability of guilty in order to reach a final conclusion that the odds are (1000:1). Arguing via modus tollens about a single piece of evidence tells us only that that evidence, in and of itself, is insufficient proof. It tells us nothing about how that evidence may act cumulatively with other pieces of evidence.
This assumes that the burglary being faked is the only piece of evidence
No; I fully grant that other evidence that Knox and Sollecito are guilty, if it exists, would be evidence of the burglary being fake, which would lower the burden of proof on that hypothesis.
However, that isn’t how Massei and Cristiani reason. They don’t say, in the section on the burglary (which is at the beginning of the report), “and since we know from all the other evidence that Knox and Sollecito are guilty, we can therefore easily use these arguments about glass patterns to confirm that they did in fact stage the burglary, in case you were wondering about that”. And it’s easy to see why they don’t say that: there wouldn’t be much point, because if they’ve already shown that Knox and Sollecito are guilty, their work is done! (*)
Instead, what they say is “these arguments about glass patterns etc. prove that the burglary was staged. Now, having established that piece of evidence against them (i,e. the staging of the burglary), let us now consider the other evidence, which, in combination with the burglary, will show how really guilty they are.”
( ) Technically, staging a burglary is itself an offense, so there may actually have been reason for them to proceed this way. But in that case the burglary issue would have come at the end* of the report, not the beginning.
“these arguments about glass patterns etc. prove that the burglary was staged. Now, having established that piece of evidence against them (i,e. the staging of the burglary), let us now consider the other evidence, which, in combination with the burglary, will give us an accurate probability on whether they are guilty”
I’ve bolded a single change to your quote. With that change made, do you feel this is a reasonable assertion?
P(Someone faked the burglary) != P(Amanda Knox faked the burglary). The report asserts the first, not the second, from my reading.
Given that “someone faked” is true, I think assigning an approximately 100% chance that Amanda Knox is guilty is rather seriously unfounded. What am I missing?
That “burglary was faked” is shorthand for “burglary was faked by Knox and Sollecito” throughout this post and discussion. The latter is what Massei and Cristiani argue, and is what would most strongly imply that Knox and Sollecito are guilty of murder.
The evidence you quoted merely suggests the burglary was faked. I’d assume there are more people with a motive to do that than just Knox and Sollecito? Why would we assume, with high enough certainty to convict, that it was certainly them and not a roommate, or someone who knew them?
Look, I’m not saying Massei and Cristiani’s argument that Knox and Sollecito staged the burglary is convincing, by any means!
That said, their argument that if the burglary was staged, the staging was done by Knox and Sollecito is probably the most convincing part of it. At the very least, they would have a highish prior, since they had access to the house and were “available” that night to do the staging if they wanted to.
I figured this out but it threw me when I got to this part of the post. I’m not sure the convenience of the shorthand justifies throwing your readers off.
Exactly. The problem for Knox and Sollecito is that there is so much evidence that even if it was all weak (and it isn’t) just the number of items is sufficent to arrive at a high certainty of guilt because they are all independent events.
That is a lot of evidence. Some items are so strong that they in isolation would be sufficent to reach the level of certainty required to convict and others are strong evidence but not enough. I count 24 items.
This isn’t quite true. If the prior probability of being a murderer is 1 in 10^6, and I can find 30 things that are explained twice as well by the murder hypothesis as the non-murder hypothesis, then the posterior probability of being a murderer is 99.9%, in the absence of mitigating factors (since 2^30/10^6 is about 1000.) So, many pieces of weak evidence for an unlikely proposition can still establish that proposition.
You’d also need those 30 things to be independent.
Probably superfluous nitpicking: you can build a strong case even with partially-interdependent pieces of evidence, you will just need more of them since you have to work off their conditional probabilities (which is mathematically equivalent to “splitting” them into independent pieces of evidence).
This doesn’t contradict anything in the paragraph you quoted. (If you don’t mind, tell me where you thought the contradiction was, so that I can explain further.)
The sentence in particular that I objected to was
My impression was that you were claiming that, since the fake burglary hypothesis would imply murder, evidence must be extremely strong to be counted in favor of fake burglary. But I may have misunderstood you. At any rate, you elsewhere state that
If you agree with my point, then I don’t see how you can find it odd that they would feel obliged to include more in their report than just the claim that the burglary was faked. Like you said, even if the evidence is fairly strong in favor of this assertion, far more evidence would be needed to convict those two of murder, which is presumably the point of the remaining 400 pages.
My claim is that to reach a desired level of certainty about the burglary being faked, you would need evidence of approximately the same strength required to reach the same level of certainty about murder. (In other words, that the prior probability of fake burglary is roughly the prior probability of murder.)
This is the very opposite of what I said! What I said was that if you knew with high confidence that the burglary was fake, then you would need almost no additional evidence to convict of murder.
Okay, so we seem to be in complete agreement about how the math works out. If so, then I’m confused as to why you object so strongly to the prosecution’s argument on purely mathematical grounds; I haven’t read their argument myself, so it’s entirely possible that the argument itself is weak in some way, but I think that right now we’re just talking about the math.
If we ignore their specific language, the plan of coming up with ~20 pieces of moderate evidence is a perfectly reasonable strategy for correctly establishing guilt, assuming that there is absolutely no mitigating evidence. Your complaint seems to be that they use different language/notation than you and I do to talk about evidence, which seems hardly fair.
Although I would also note that since humans are bad at intuitively distinguishing between moderate evidence for and moderate evidence against a hypothesis, trying to find many pieces of weak evidence is probably not a good strategy if the goal is to get humans to correctly decide the accuracy of an assertion.
ETA: By the way, I’ve been working under the assumption, based on the tone of the original post, that you think there are serious mathematical flaws in the prosecutions argument. If that’s not the case, and you just wanted to use this case as a point of illustration, then I apologize for the confusion.
What I gather is that the prosecution concludes, after the first twenty pages of the brief that discuss the break-in exclusively that the break in was almost certainly staged by Knox and Sallecito. But if they really thought that they would have already more or less made the case the Knox and Sallecito are guilty and the remaining 380 pages would be unnecessary. So the prosecution can’t be weighing the evidence correctly.
If I may presume to diagnose your confusion, it seems that you’re compartmentalizing between “mathematical” aspects of an argument and “other” aspects. But I’m not. I’m taking it for granted that “the math” is the argument. Probability theory is a mathematical formalization of the process of argument and inference. It isn’t just a cool gadget that one throws in on special occasions.
So, I don’t object to Massei and Cristiani’s argument on “purely mathematical grounds”. I simply object to it, period—and in this post I have used mathematical language to describe, in precise terms, what my objection is.
(And I expected readers to assume, given my previous writing on the case, that this particular point was far from my only objection to Massei and Cristiani’s 427-page argument that Knox and Sollecito killed Kercher; hence I was not expecting replies of the form “well, but they might have other good evidence that Knox and Sollecito are guilty”. They don’t; we’ve already covered that.)
I honestly have no idea where you’re getting this from. I don’t know of any passage in the post where I complained about Massei and Cristiani’s choice of language; and nor did I attempt to argue (as several people seem to have thought I did) against a strategy of proving one’s case by adducing a large amount of weak evidence in one’s favor (although as a matter of fact I do believe that is the wrong type of argument to expect for a proposition of this sort, and that people have probably been misled by detective stories and the like into thinking it a reasonable strategy, when it would actually be very difficult to make work in practice—that however would be the topic of a separate post, and isn’t addressed in this one).
My criticism of Massei and Cristiani in this post is really quite simple, or so I thought: the type of evidence that they cite to prove that the burglary was faked suggests that they did not realize how high the burden of proof for this proposition was—that, just to prove the burglary was faked, they needed evidence of the same level of strength as would be required to directly prove Knox and Sollecito guilty of murder.
Quite frankly, I’m baffled at how this point seems to have gotten lost, because I thought I was emphatic and indeed repetitious about it in the post.
I think the assertion is that they appear to be coming up with ~20 pieces of evidence and then trying to say that each piece is very strong—or at least, they have done so for the burglary hypothesis, so they might be doing so for the other pieces of evidence too. Naturally, their methods of making each piece look very strong are flawed.
You almost pinpointed the reason why this happening here:
Humans are bad at intuitively handling evidence in general. There is a possibility that this case suffers from a serious malady: presiding judge Massei has decided the correct, accurate decision in this matter is that Knox and Sollecito are guilty, and has strategically prepared the judge’s report to get people to decide this way. This hypothesis explains why the judge has produced such a weighty document when 20 pages of it would have sufficed.
“my claim is that to reach a desired level of certainty about the burglary being faked, you would need evidence of approximately the same strength required to reach the same level of certainty about murder.”
This assumes that the burglary being faked is the only piece of evidence. If we have three sets of evidence, and each one suggests a 90% chance of guilt, and each is independent of the other, then we have probability (10:1) x (10:1) x (10:1) = (1000:1). No one set of evidence needs to have a (1000:1) probability of guilty in order to reach a final conclusion that the odds are (1000:1). Arguing via modus tollens about a single piece of evidence tells us only that that evidence, in and of itself, is insufficient proof. It tells us nothing about how that evidence may act cumulatively with other pieces of evidence.
No; I fully grant that other evidence that Knox and Sollecito are guilty, if it exists, would be evidence of the burglary being fake, which would lower the burden of proof on that hypothesis.
However, that isn’t how Massei and Cristiani reason. They don’t say, in the section on the burglary (which is at the beginning of the report), “and since we know from all the other evidence that Knox and Sollecito are guilty, we can therefore easily use these arguments about glass patterns to confirm that they did in fact stage the burglary, in case you were wondering about that”. And it’s easy to see why they don’t say that: there wouldn’t be much point, because if they’ve already shown that Knox and Sollecito are guilty, their work is done! (*)
Instead, what they say is “these arguments about glass patterns etc. prove that the burglary was staged. Now, having established that piece of evidence against them (i,e. the staging of the burglary), let us now consider the other evidence, which, in combination with the burglary, will show how really guilty they are.”
( ) Technically, staging a burglary is itself an offense, so there may actually have been reason for them to proceed this way. But in that case the burglary issue would have come at the end* of the report, not the beginning.
“these arguments about glass patterns etc. prove that the burglary was staged. Now, having established that piece of evidence against them (i,e. the staging of the burglary), let us now consider the other evidence, which, in combination with the burglary, will give us an accurate probability on whether they are guilty”
I’ve bolded a single change to your quote. With that change made, do you feel this is a reasonable assertion?
No. The error is in the first sentence
They only (conceivably) prove the burglary was staged if you’re already taking into account the rest of the evidence of murder.
That’s only true if you assume p(A=>B) is 1
...or approximately 1.
(And by P(A=>B), I think you meant P(B|A), didn’t you?)
P(Someone faked the burglary) != P(Amanda Knox faked the burglary). The report asserts the first, not the second, from my reading.
Given that “someone faked” is true, I think assigning an approximately 100% chance that Amanda Knox is guilty is rather seriously unfounded. What am I missing?
That “burglary was faked” is shorthand for “burglary was faked by Knox and Sollecito” throughout this post and discussion. The latter is what Massei and Cristiani argue, and is what would most strongly imply that Knox and Sollecito are guilty of murder.
The evidence you quoted merely suggests the burglary was faked. I’d assume there are more people with a motive to do that than just Knox and Sollecito? Why would we assume, with high enough certainty to convict, that it was certainly them and not a roommate, or someone who knew them?
Look, I’m not saying Massei and Cristiani’s argument that Knox and Sollecito staged the burglary is convincing, by any means!
That said, their argument that if the burglary was staged, the staging was done by Knox and Sollecito is probably the most convincing part of it. At the very least, they would have a highish prior, since they had access to the house and were “available” that night to do the staging if they wanted to.
I figured this out but it threw me when I got to this part of the post. I’m not sure the convenience of the shorthand justifies throwing your readers off.
Exactly. The problem for Knox and Sollecito is that there is so much evidence that even if it was all weak (and it isn’t) just the number of items is sufficent to arrive at a high certainty of guilt because they are all independent events.
http://themurderofmeredithkercher.com/The_Evidence
That is a lot of evidence. Some items are so strong that they in isolation would be sufficent to reach the level of certainty required to convict and others are strong evidence but not enough. I count 24 items.
Only works if each piece of evidence is independent. They’re clearly not.