“I really doubt the garbage disposal is really broken, we just had two other major things replaced, what are the odds that another thing would break so quickly?”
You have two hypotheses: the appliances breaking are not connected (independent); and the appliance breaking are connected (dependent).
In the first case you are saying the equivalent of “I tossed the coin twice and it came up heads both times, it’s really unlikely it will come up heads the third time as well” which should be obviously wrong.
In the second case you should discard your model of independence alongside with your original prior and consider that the breakages are connected.
I think the moral of the story is that life is complicated and simple models are often too simple to be useful. You should discard them faster when they show signs of not working.
And, of course, if you are wondering whether your garbage disposal is really broken, you should go look at your garbage disposal unit and not engage in pondering theoretical considerations.
See my response to ChristianKl below for my clarification on my reasoning about “consecutive coin flips” which could still be wrong but is hopefully less wrong than my original wording.
I agree that I should have discarded my model more quickly, but I don’t quite see how to generalize that observation. Sometimes the alternative hypothesis (e.g. the breakages are connected) is not apparent or obvious without more data—and the process of collecting data really just means continuing to make bad predictions as you go through life until something clicks and you notice the underlying structure.
My wife seems to think that making explicit model-based predictions in the first place is the problem. I have a lot of respect for System 1 and am sympathetic to this view. But System 2 really shouldn’t actively lead me astray.
Yes, and note that this part—“that I have to start considering that the die is loaded”—is key.
but I don’t quite see how to generalize that observation
Um, directly? All models which you are considering are much simpler than the real world. The relevant maxim is “All models are wrong, but some are useful”.
I think you got caught in the trap of “but I can’t change my prior because priors are not supposed to be changed”. That’s not exactly true. You can and (given sufficient evidence) should be willing to discard your entire model and the prior with it. Priors only make sense within a specified set of hypotheses. If your set of hypotheses changes, the old prior goes out of the window.
The naive Bayes approach sweeps a lot of complexity under the rug (e.g. hypotheses selection) which will bite you in the ass given the slightest opportunity.
Sometimes the alternative hypothesis (e.g. the breakages are connected) is not apparent or obvious
Yeah, well, welcome to the real world :-/
My wife seems to think that making explicit model-based predictions in the first place is the problem.
She is correct if your models are wrong. Getting right models is hard and you should not assume that the first model you came up with is going to be sufficiently correct to be useful.
System 2 really shouldn’t actively lead me astray.
I see absolutely no basis for this belief. To misquote someone from memory: “Logic is just a way of making errors with confidence” :-P
You have two hypotheses: the appliances breaking are not connected (independent); and the appliance breaking are connected (dependent).
In the first case you are saying the equivalent of “I tossed the coin twice and it came up heads both times, it’s really unlikely it will come up heads the third time as well” which should be obviously wrong.
In the second case you should discard your model of independence alongside with your original prior and consider that the breakages are connected.
I think the moral of the story is that life is complicated and simple models are often too simple to be useful. You should discard them faster when they show signs of not working.
And, of course, if you are wondering whether your garbage disposal is really broken, you should go look at your garbage disposal unit and not engage in pondering theoretical considerations.
See my response to ChristianKl below for my clarification on my reasoning about “consecutive coin flips” which could still be wrong but is hopefully less wrong than my original wording.
I agree that I should have discarded my model more quickly, but I don’t quite see how to generalize that observation. Sometimes the alternative hypothesis (e.g. the breakages are connected) is not apparent or obvious without more data—and the process of collecting data really just means continuing to make bad predictions as you go through life until something clicks and you notice the underlying structure.
My wife seems to think that making explicit model-based predictions in the first place is the problem. I have a lot of respect for System 1 and am sympathetic to this view. But System 2 really shouldn’t actively lead me astray.
Yes, and note that this part—“that I have to start considering that the die is loaded”—is key.
Um, directly? All models which you are considering are much simpler than the real world. The relevant maxim is “All models are wrong, but some are useful”.
I think you got caught in the trap of “but I can’t change my prior because priors are not supposed to be changed”. That’s not exactly true. You can and (given sufficient evidence) should be willing to discard your entire model and the prior with it. Priors only make sense within a specified set of hypotheses. If your set of hypotheses changes, the old prior goes out of the window.
The naive Bayes approach sweeps a lot of complexity under the rug (e.g. hypotheses selection) which will bite you in the ass given the slightest opportunity.
Yeah, well, welcome to the real world :-/
She is correct if your models are wrong. Getting right models is hard and you should not assume that the first model you came up with is going to be sufficiently correct to be useful.
I see absolutely no basis for this belief. To misquote someone from memory: “Logic is just a way of making errors with confidence” :-P