Let me dramatize your sniper example a bit with this hypothetical scenario.
You read about homeopathy, you realize that the purported theoretical underpinnings sound nonsensical, you learn that all properly conducted clinical trials show that homeopathy is no better than placebo and so estimate that the odds of it being anything but a sham are, charitably, 1 in a million. Then your mother, who is not nearly as rational, tries to convince you to take Sabadil for your allergies, because it has good product reviews. You reluctantly agree to humor her (after all, homeopathy is safe, if useless).
Lo, your hay fever subsides… and now you are stuck trying to recalculate your estimate of homeopathy being more than a sham in the face of the shear impossibility of the sugar pills with not a single active molecule remaining after multiple dilutions having any more effect than a regular sugar pill.
Lo, your hay fever subsides… and now you are stuck trying to recalculate your estimate of homeopathy being more than a sham in the face of the shear impossibility of the sugar pills with not a single active molecule remaining after multiple dilutions having any more effect than a regular sugar pill.
What would you do?
Test regular sugar pills, obviously. Apparently my hayfever responds to placebo, and that’s awesome. High fives all around!
Oooh! That would be so interesting. You could do blind trials with sugar pills vs other sugar pills. You could try to make other tests which could distinguish between homeopathic sugar and non-homeopathic sugar. Then if you found some you could try to make better placebos that pass the tests. You might end up finding an ingredient in the dye in the packaging that cures your hayfever. And if not something like that, you might end up rooting out something deep!
I suspect that if you were modelling an ideal mind rather than my rubbish one, the actual priors might not shift very much, but the expected value of investigating might be enough to prompt some experimenting. Especially if the ideal mind’s hayfever was bad and the homeopathic sugar was expensive.
Increase the probability that homeopathy works for me- but also increase the probability of all other competing hypotheses except the “sugar placebo” one. The next most plausible hypothesis at that point is probably that the effort is related to my mother’s recommendation. Unless I expect the knowledge gained here to extend to something else, at some point I’ll probably stop trying to explain it.
At some point, though, this hypothetical veers into “I wouldn’t explain X because X wouldn’t happen” territory. Hypothetical results don’t have to be generated in the same fashion as real results, and so may have unreal explanations.
I would follow the good old principle of least assumptions: It is extremely likely that something else, the nature of which is unknown and difficult to identify within the confines of the stated problem, caused the fever to subside. This merely assumes that your prior is more reliable and that you do not know all the truths of the universe, the latter assumption being de-facto in very much all bayesian reasoning, to my understanding.
From there, you can either be “stubborn” and reject this piece of evidence due to the lack of an unknown amount of unknown data on what other possible causes there could be, or you could be strictly bayesian and update your belief accordingly. Even if you update your beliefs, since the priors of Homeopathy working is already very low, and the prior of you being cured by this specific action is also extremely low when compared to everything else that could possibly have cured you… and since you’re strictly bayesian, you also update other beliefs, such as placebo curing minor ailments, “telepathic-like” thought-curing, self-healing through symbolic action, etc.
All in all, this paints a pretty solid picture of your beliefs towards homeopathy being unchanged, all things considered.
Following the OP’s sniper example, suppose you take the same pill the next time you have a flare-up, and it helps again. Would you still stick to your prewritten bottom line of “it ought to be something else”?
To better frame my earlier point, suppose that you flare up six times during a season, and the six times it gets better after having applied “X” method which was statistically shown to be no better than placebo, and after rational analysis you find out that each time you applied method X you were also circumstantially applying method Y at the same time (that is, you’re also applying a placebo, since unless in your entire life you have never been relieved of some ailment by taking a pill, your body/brain will remember the principle and synchronize, just like any other form of positive reinforcement training).
In other words, both P(X-works|cured) and P(Y-works|cured) are raised, but only by half, since statistically they’ve been shown to have the same effect, and thus your priors are that they are equally as likely of being the cure-cause, and while both could be a cure-cause, the cure-cause could also be to have applied both of them. Since those two latter possibilities end up evening out, you divide the posteriors in the two, from my understanding. I might be totally off though, since I haven’t been learning about Bayes’ theorem all that long and I’m very much still a novice in bayesian rationality and probability.
To make a long story short, yes, I would stick to it, because within the presented context there are potentially thousands of X, Y and Z possible cure-causes, so while the likeliness that one of them is a cure is going up really fast each time I cure myself under the same circumstances, only careful framing of said circumstances will allow anyone to really rationally establish which factors become more likely to be truly causal and which are circumstantial (or correlated in another, non-directly-causal fashion).
Since homeopathy almost invariably involves hundreds of other factors, many of which are unknown and some of which we might be completely unaware, it becomes extremely difficult to reliably test for its effectiveness in some circumstances. This is why we assign greater trust in the large-scale double-blind studies, because our own analysis is of lower comparative confidence. At least within the context of this particular sniper example.
The homoeopathy claims interaction of water molecules with each other leads to water memory… it can even claim that all the physics is the same (hypothesis size per Solomonoff induction is same) yet the physics works out to the stuff happening. And we haven’t really ran sufficiently detailed simulation to rule it out, just arguments of uncertain validity. There’s no way to estimate it’s probability.
We do something different than estimating probability of homoeopathy being true. It’s actually very beautiful method, very elegant solution. We say, well, let’s strategically take the risk of one-in-a-million that we discard a true curing method homoeopathy with such and such clinical effect. Then we run the clinical trials, and find evidence that there’s less than one-in-a-million chance that those trials happened as they were, if the homoeopathy was true. We still do not know the probability of homoeopathy, but we strategically discard it, and we know the probability of wrongfully having discarded it, we can even find upper bound on how much that strategy lost in terms of expected utility, by discarding. That’s how science works. The cut off strategy can be set from utility maximization considerations (with great care to avoid pascal’s wager).
So, suppose that Sabadil cured your allergies 10 times out of 10, you will not take again unless forced to, because “There’s no way to estimate it’s (sic) probability.”? Maybe you need to reread chapter 1 of HPMOR, and brush up on how to actually change your mind.
If it did cure allergies 10 times out of 10, and that ALL other possible cure-causes had been eliminated as causal beforehand (including the placebo effect which is inherent to most acts of taking a homoeopathic pill, even when the patient doesn’t believe it’ll work, simply out of subconscious memory of being cured by taking a pill), then yes, the posterior belief in its effectiveness would shoot up.
However, “the body curing itself by wanting to and being willing to even try things we know probably won’t work based on what-ifs alone” is itself a major factor, one that has also been documented.
Par contre, if it did work 10 times out of 10, then I almost definitely would take it again, since it has now been shown to be, at worst, statistically correlated with whatever actually does cure me of my symptoms, whether that’s the homoeopathic treatment or not. While doing that, I would keep attempting to rationally identify the proper causal links between events.
The point is that there is a decision method that allows me to decide without anyone having to make a prior.
Say, the cost of trial is a, the cost (utility loss) of missing valid cure to strategy failure is b, you do the N trials , N such that a N < (the probability of trials given assumption of validity of cure) b , then you proclaim cure not working. Then you can do more trials if the cost of trial falls. You don’t know the probability and you still decide in an utility-maximizing manner (on choice of strategy), because you have the estimate on the utility loss that the strategy will incur in general.
edit: clearer. Also I am not claiming it is the best possible method, it isn’t, but it’s a practical solution that works. You can know the probability that you will end up going uncured if the cure actually works.
let’s strategically take the risk of one-in-a-million that we discard a true curing method homoeopathy with such and such clinical effect
Where does your choice of “such and such clinical effect” come from? Keeping your one-in-a-million chance of being wrong fixed, the scale of the clinical trials required depends on the effect size of homeopathy. If homeopathy is a guaranteed cure, it’s enough to dose one incurably sick person. If it helps half of the patients, you might need to dose on the order of twenty. And so on for smaller effect sizes. The homeopathy claim is not just a single hypothesis but a compound hypothesis consisting of all these hypotheses. Choosing which of these hypotheses to entertain is a probabilistic judgment; it can’t be escaped by just picking one of the hypotheses, since that’s just concentrating the prior mass at one point.
(Pardon the goofy notation. Don’t want to deal with the LaTeX engine.)
The compound hypothesis is well-defined. Suppose that the baseline cure probability for a placebo is θ ∈ [0,1). Then hypotheses take the form H ⊂ [0,1], which have the interpretation that the cure rate for homeopathy is in H. The standing null hypothesis in this case is Hθ = { θ }. The alternative hypothesis that homeopathy works is H>θ = (θ,1] = { x : x > θ }. For any θ′ ∈ H>θ, we can construct a “one-in-a-million chance of being wrong” test for the simple hypothesis Hθ′ that homeopathy is effective with effect size exactly θ′. It is convenient that such tests work just as well for the hypothesis H≥θ′. However, we can’t construct a test for H>θ.
Bringing in falsifiability only confuses the issue. No clinical data exist that will strictly falsify any of the hypotheses considered above. On the other hand, rejecting Hθ′ seems like it should provide weak support for rejecting H>θ. My take on this is that since such a research program seems to work in practice, falsifiability doesn’t fully describe how science works in this case (see Popper vs. Kuhn, Lakatos, Feyerabend, etc.).
Clinical data still exists that would allow a strategy to stop doing more tests at specific cut off point as the payoff from the hypothesis being right is dependent to the size of the effect and there will be clinical data at some point where the integral of payoff over lost clinical effects is small enough. It just gets fairly annoying to calculate. . Taking the strategy will be similar to gambling decision.
I do agree that there is a place for occam’s razor here but there exist no formalism that actually lets you quantify this weak support. There’s the Solomonoff induction, which is un-computable and awesome for work like putting an upper bound on how good induction can (or rather, can’t) ever be.
No, I mean I would be equally confused, because that’s exactly what happened. Homeopathy is literally water, so I would be just as confused if I had drank water and my hay fever subsided.
Also, completely separately, even if you took some drug that wasn’t just water, your case is anecdotal evidence. It should hold the same weight as any single case in any of the clinical trials. That means that it adds pretty much no data, and it’s correct to not update much.
even if you took some drug that wasn’t just water, your case is anecdotal evidence. It should hold the same weight as any single case in any of the clinical trials.
Ah, but that’s not quite true. If a remedy helps you personally, it is less than a “single case” for clinical trial purposes, because it was not set up properly for that. You would do well to ignore it completely if you are calculating your priors for the remedy to work for you.
However, it is much more than nothing for you personally, once you have seen it working for you once. Now you have to update on the evidence. It’s a different step in the Bayesian reasoning: calculating new probabilities given that a certain event (in this case one very unlikely apriori—that the remedy works) actually happened.
OK, so you take it to your chem lab and they confirm that the composition is pure sugar, as far as they can tell. How many alternatives would you keep inventing before you update your probability of it actually working?
In other words, when do you update your probability that there is a sniper out there, as opposed to “there is a regular soldier close by”?
OK, I suppose this makes sense. Let me phrase it differently. What personal experience (not published peer-reviewed placebo-controlled randomized studies) would cause you to be convinced that what is essentially magically prepared water is as valid a remedy as, say, Claritin?
Well, I hate to say this for obvious reasons, but if the magic sugar water cured my hayfever just once, I’d try it again, and if it worked again, I’d try it again. And once it had worked a few times, I’d probably keep trying it even if it occasionally failed.
If it consistently worked reliably I’d start looking for better explanations. If no-one could offer one I’d probably start believing in magic.
I guess not believing in magic is something to do with not expecting this sort of thing to happen.
Well, I hate to say this for obvious reasons, but if the magic sugar water cured my hayfever just once, I’d try it again, and if it worked again, I’d try it again.
(This tripped my positive bias sense: only testing the outcome in the presence of an intervention doesn’t establish that it’s doing anything. It’s wrong to try again and again after something seemed to work, one should also try not doing it and see if it stops working. Scattering anti-tiger pills around town also “works”: if one does that every day, there will be no tigers in the neighborhood.)
Scattering anti-tiger pills around town also “works”: if one does that every day, there will be no tigers in the neighborhood.
That’s a bad analogy. If “anti-tiger pills” repeatedly got rid of a previously observed real tiger, you would be well advised to give the issue some thought.
That’s a bad analogy. If “anti-tiger pills” repeatedly got rid of a previously observed real tiger, you would be well advised to give the issue some thought.
What’s that line about how, if you treat a cold, you can get rid of it in seven days, but otherwise it lasts a week?
You would still want to check to see whether tigers disappear even when no “anti-tiger pills” are administered.
You would still want to check to see whether tigers disappear even when no “anti-tiger pills” are administered.
Depending on how likely the tiger is to eat people if it didn’t disappear, and your probability of the pills successfully repelling the tiger given that it’s always got rid of the tiger, and the cost of the pill and how many people the tiger is expected to eat if it doesn’t disappear? Not always.
A medical example of this is the lack of evidence for the efficacy of antihistamine against anaphylaxis. When I asked my sister (currently going through clinical school) about why, she said “because if you do a study, people in the control group will die if these things work, and we have good reason to believe they do”
and your probability of the pills successfully repelling the tiger given that it’s always got rid of the tiger
Yes. But the point is that this number should be negligible if you haven’t seen how the tiger behaves in the absence of the pills. (All of this assumes that you do not have any causal model linking pill-presence to tiger-absence.)
This case differs from the use of antihistamine against anaphylaxis for two reasons:
There is some theoretical reason to anticipate that antihistamine would help against anaphylaxis, even if the connection hasn’t been nailed down with double-blind experiments.
We have cases where people with anaphylaxis did not receive antihistamine, so we can compare cases with and without antihistamine. The observations might not have met the rigorous conditions of a scientific experiment, but that is not necessary for the evidence to be rational and to justify action.
Absolutely. The precise thing that matters is the probability tigers happen if you don’t use the pills. So, say, I wouldn’t recommend doing the experiment if you live in areas with high densities of tigers (which you do if there’s one showing up every day!) and you weren’t sure what was going into the pills (tiger poison?), but would recommend doing the experiment if you lived in London and knew that the pills were just sugar.
Similarly, I’m more likely to just go for a herbal remedy that hasn’t had scientific testing, but has lots of anecdotal evidence for lack of side-effects, than a homeopathic remedy with the same amount of recommendation.
It is positive bias (in that this isn’t the best way to acquire knowledge), but there’s a secondary effect: the value of knowing whether or not the magic sugar water cures his hayfever is being traded off against the value of not having hayfever.
Depending on how frequently he gets hayfever, and how long it took to go away without magic sugar water, and how bothersome it is, and how costly the magic sugar water is, it may be better to have an unexplained ritual for that portion of his life than to do informative experiments.
(And, given that the placebo effect is real, if he thinks the magic sugar water is placebo, that’s reason enough to drink it without superior alternatives.)
Agree with this. Knowing the truth has a value and a cost (doing the experiment).
I recently heard something along the lines of: “We don’t have proof that antihistamines work to treat anaphylaxis, because we haven’t done the study. But the reason we haven’t done the study is because we’re pretty sure the control group would die.”
I agree, I’d try not taking it too! I had hayfever as a child, and it was bloody awful. I used to put onionjuice in my eyes because it was the only thing that would provide relief. But even as a child I was curious enough to try it both ways.
Maybe, but it also explains why any other thing will cure my hayfever. And shouldn’t it go away if I realize it’s a placebo? And if I say ‘I have this one thing that cures my hayfever reliably, and no other thing does, but it has no mechanism except for the placebo effect’, is that very different from ‘I have this magic thing?’.
I’m not keen on explanations which don’t tell me what to anticipate. But maybe I misunderstand the placebo effect. How would I tell the difference between it and magic?
And if I say ‘I have this one thing that cures my hayfever reliably, and no other thing does, but it has no mechanism except for the placebo effect’, is that very different from ‘I have this magic thing?’.
No, not very. Also, if it turns out that only this one thing works, and no other thing works, then (correcting for the usual expectation effects) that is relatively strong evidence that something more than the placebo effect is going on. Conversely, if it is the placebo effect, I would expect that a variety of substances could replace the sugar pills without changing the effect much.
Another way of putting this is, if I believe that the placebo effect is curing my hayfever, that ultimately means the power to cure my hayfever resides inside my brain and the question is how to arrange things so that that power gets applied properly. If I believe that this pill cures my hayfever (whether via “the placebo effect” or via “magic” or via “science” or whatever other dimly understood label I tack onto the process), that means the power resides outside my brain and the question is how to secure a steady supply of the pill.
Apparently not. The effect might be less, I don’t think the study checked. But once you know it’s a placebo and the placebo works, then you’re no longer taking a sugar pill expecting nothing, you’re taking a sugar pill expecting to get better.
You could tell the difference between the placebo effect and magic by doing a double blind trial on yourself. e.g. Get someone to assign either “magic” pill or identical sugar pill (or solution) with a random number generator for a period where you’ll be taking the drug, prepare them and put them in order for you to take on successive days, and write down the order to check later. Then don’t talk to them for the period of the experiment. (If you want to talk to them you can apply your own shuffle and write down how to reverse it)
Exactly. You write down your observations for each day and then compare them to the list to see if you felt better on days when you were taking the actual pill.
Only if it’s not too costly to check, of course, and sometimes it is.
Edit: I think gwern’s done a number of self-trials, though I haven’t looked at his exact methodology.
Edit again: In case I haven’t been clear enough, I’m proposing a method to distinguish between “sugar pills that are magic” and “regular sugar pills”.
If you have a selection of ‘magic’ sugar pills, and you want to test them for being magic vs placebo effect, you do a study comparing their efficacy to that of ‘non-magic’ sugar pills.
If they are magic, then you aren’t comparing identical things, because only some of them have the ‘magic’ property
Well, you need it to work better than without the magic sugar water.
My approach is: I believe that the strategy of “if the magic sugar water worked with only 1 in a million probability of ‘worked’ being obtained by chance without any sugar water, and if only a small number of alternative cures were also tried, then adopt the belief that magic sugar water works” is a strategy that has only small risk of trusting in a non-working cure, but is very robust against unknown unknowns. It works even if you are living in a simulator where the beings-above mess with the internals doing all sorts of weird stuff that shouldn’t happen and for which you might be tempted to set very low prior.
Meanwhile the strategy of “make up a very low prior, then update it in vaguely Bayesian manner” has past history of screwing up big time leading up to significant preventable death, e.g. when antiseptic practices invented by this guy were rejected on the grounds of ‘sounds implausible’, and has pretty much no robustness against unknown unknowns, and as such is grossly irrational (in the conventional sense of ‘rational’) even though in the magic water example it sounds like awesomely good idea.
How many alternatives would you keep inventing before you update your probability of it actually working?
Precisely the hypotheses that are more likely than homeopathy. Once I’ve falsified those the probability starts pouring into homeopathy. Jaynes’ “Probability Theory: The Logic Of Science”, explains this really well in Chapter 4 and the “telepathy” example of Chapter 5. In particular I learnt a lot by staring at Figure 4.1.
Let me dramatize your sniper example a bit with this hypothetical scenario.
You read about homeopathy, you realize that the purported theoretical underpinnings sound nonsensical, you learn that all properly conducted clinical trials show that homeopathy is no better than placebo and so estimate that the odds of it being anything but a sham are, charitably, 1 in a million. Then your mother, who is not nearly as rational, tries to convince you to take Sabadil for your allergies, because it has good product reviews. You reluctantly agree to humor her (after all, homeopathy is safe, if useless).
Lo, your hay fever subsides… and now you are stuck trying to recalculate your estimate of homeopathy being more than a sham in the face of the shear impossibility of the sugar pills with not a single active molecule remaining after multiple dilutions having any more effect than a regular sugar pill.
What would you do?
Test regular sugar pills, obviously. Apparently my hayfever responds to placebo, and that’s awesome. High fives all around!
and if it does not?
Oooh! That would be so interesting. You could do blind trials with sugar pills vs other sugar pills. You could try to make other tests which could distinguish between homeopathic sugar and non-homeopathic sugar. Then if you found some you could try to make better placebos that pass the tests. You might end up finding an ingredient in the dye in the packaging that cures your hayfever. And if not something like that, you might end up rooting out something deep!
I suspect that if you were modelling an ideal mind rather than my rubbish one, the actual priors might not shift very much, but the expected value of investigating might be enough to prompt some experimenting. Especially if the ideal mind’s hayfever was bad and the homeopathic sugar was expensive.
Increase the probability that homeopathy works for me- but also increase the probability of all other competing hypotheses except the “sugar placebo” one. The next most plausible hypothesis at that point is probably that the effort is related to my mother’s recommendation. Unless I expect the knowledge gained here to extend to something else, at some point I’ll probably stop trying to explain it.
At some point, though, this hypothetical veers into “I wouldn’t explain X because X wouldn’t happen” territory. Hypothetical results don’t have to be generated in the same fashion as real results, and so may have unreal explanations.
Do a bayesian update.
Additional to the prior, you need:
(A) The probability that you will heal anyway
(B) The probability that you will heal if sabadil works
I have done the math for A=0,5 ; B=0,75 : The result is an update from 1/10^6 to 1,5 * 1/10^6
For A=1/1000 and B=1 the result is 0.000999
The math is explained here: http://lesswrong.com/lw/2b0/bayes_theorem_illustrated_my_way
I would follow the good old principle of least assumptions: It is extremely likely that something else, the nature of which is unknown and difficult to identify within the confines of the stated problem, caused the fever to subside. This merely assumes that your prior is more reliable and that you do not know all the truths of the universe, the latter assumption being de-facto in very much all bayesian reasoning, to my understanding.
From there, you can either be “stubborn” and reject this piece of evidence due to the lack of an unknown amount of unknown data on what other possible causes there could be, or you could be strictly bayesian and update your belief accordingly. Even if you update your beliefs, since the priors of Homeopathy working is already very low, and the prior of you being cured by this specific action is also extremely low when compared to everything else that could possibly have cured you… and since you’re strictly bayesian, you also update other beliefs, such as placebo curing minor ailments, “telepathic-like” thought-curing, self-healing through symbolic action, etc.
All in all, this paints a pretty solid picture of your beliefs towards homeopathy being unchanged, all things considered.
Following the OP’s sniper example, suppose you take the same pill the next time you have a flare-up, and it helps again. Would you still stick to your prewritten bottom line of “it ought to be something else”?
To better frame my earlier point, suppose that you flare up six times during a season, and the six times it gets better after having applied “X” method which was statistically shown to be no better than placebo, and after rational analysis you find out that each time you applied method X you were also circumstantially applying method Y at the same time (that is, you’re also applying a placebo, since unless in your entire life you have never been relieved of some ailment by taking a pill, your body/brain will remember the principle and synchronize, just like any other form of positive reinforcement training).
In other words, both P(X-works|cured) and P(Y-works|cured) are raised, but only by half, since statistically they’ve been shown to have the same effect, and thus your priors are that they are equally as likely of being the cure-cause, and while both could be a cure-cause, the cure-cause could also be to have applied both of them. Since those two latter possibilities end up evening out, you divide the posteriors in the two, from my understanding. I might be totally off though, since I haven’t been learning about Bayes’ theorem all that long and I’m very much still a novice in bayesian rationality and probability.
To make a long story short, yes, I would stick to it, because within the presented context there are potentially thousands of X, Y and Z possible cure-causes, so while the likeliness that one of them is a cure is going up really fast each time I cure myself under the same circumstances, only careful framing of said circumstances will allow anyone to really rationally establish which factors become more likely to be truly causal and which are circumstantial (or correlated in another, non-directly-causal fashion).
Since homeopathy almost invariably involves hundreds of other factors, many of which are unknown and some of which we might be completely unaware, it becomes extremely difficult to reliably test for its effectiveness in some circumstances. This is why we assign greater trust in the large-scale double-blind studies, because our own analysis is of lower comparative confidence. At least within the context of this particular sniper example.
The homoeopathy claims interaction of water molecules with each other leads to water memory… it can even claim that all the physics is the same (hypothesis size per Solomonoff induction is same) yet the physics works out to the stuff happening. And we haven’t really ran sufficiently detailed simulation to rule it out, just arguments of uncertain validity. There’s no way to estimate it’s probability.
We do something different than estimating probability of homoeopathy being true. It’s actually very beautiful method, very elegant solution. We say, well, let’s strategically take the risk of one-in-a-million that we discard a true curing method homoeopathy with such and such clinical effect. Then we run the clinical trials, and find evidence that there’s less than one-in-a-million chance that those trials happened as they were, if the homoeopathy was true. We still do not know the probability of homoeopathy, but we strategically discard it, and we know the probability of wrongfully having discarded it, we can even find upper bound on how much that strategy lost in terms of expected utility, by discarding. That’s how science works. The cut off strategy can be set from utility maximization considerations (with great care to avoid pascal’s wager).
So, suppose that Sabadil cured your allergies 10 times out of 10, you will not take again unless forced to, because “There’s no way to estimate it’s (sic) probability.”? Maybe you need to reread chapter 1 of HPMOR, and brush up on how to actually change your mind.
If it did cure allergies 10 times out of 10, and that ALL other possible cure-causes had been eliminated as causal beforehand (including the placebo effect which is inherent to most acts of taking a homoeopathic pill, even when the patient doesn’t believe it’ll work, simply out of subconscious memory of being cured by taking a pill), then yes, the posterior belief in its effectiveness would shoot up.
However, “the body curing itself by wanting to and being willing to even try things we know probably won’t work based on what-ifs alone” is itself a major factor, one that has also been documented.
Par contre, if it did work 10 times out of 10, then I almost definitely would take it again, since it has now been shown to be, at worst, statistically correlated with whatever actually does cure me of my symptoms, whether that’s the homoeopathic treatment or not. While doing that, I would keep attempting to rationally identify the proper causal links between events.
The point is that there is a decision method that allows me to decide without anyone having to make a prior.
Say, the cost of trial is a, the cost (utility loss) of missing valid cure to strategy failure is b, you do the N trials , N such that a N < (the probability of trials given assumption of validity of cure) b , then you proclaim cure not working. Then you can do more trials if the cost of trial falls. You don’t know the probability and you still decide in an utility-maximizing manner (on choice of strategy), because you have the estimate on the utility loss that the strategy will incur in general.
edit: clearer. Also I am not claiming it is the best possible method, it isn’t, but it’s a practical solution that works. You can know the probability that you will end up going uncured if the cure actually works.
Where does your choice of “such and such clinical effect” come from? Keeping your one-in-a-million chance of being wrong fixed, the scale of the clinical trials required depends on the effect size of homeopathy. If homeopathy is a guaranteed cure, it’s enough to dose one incurably sick person. If it helps half of the patients, you might need to dose on the order of twenty. And so on for smaller effect sizes. The homeopathy claim is not just a single hypothesis but a compound hypothesis consisting of all these hypotheses. Choosing which of these hypotheses to entertain is a probabilistic judgment; it can’t be escaped by just picking one of the hypotheses, since that’s just concentrating the prior mass at one point.
It’s part of the hypothesis, without it the idea is not a defined hypothesis. See falsifiability.
(Pardon the goofy notation. Don’t want to deal with the LaTeX engine.)
The compound hypothesis is well-defined. Suppose that the baseline cure probability for a placebo is θ ∈ [0,1). Then hypotheses take the form H ⊂ [0,1], which have the interpretation that the cure rate for homeopathy is in H. The standing null hypothesis in this case is Hθ = { θ }. The alternative hypothesis that homeopathy works is H>θ = (θ,1] = { x : x > θ }. For any θ′ ∈ H>θ, we can construct a “one-in-a-million chance of being wrong” test for the simple hypothesis Hθ′ that homeopathy is effective with effect size exactly θ′. It is convenient that such tests work just as well for the hypothesis H≥θ′. However, we can’t construct a test for H>θ.
Bringing in falsifiability only confuses the issue. No clinical data exist that will strictly falsify any of the hypotheses considered above. On the other hand, rejecting Hθ′ seems like it should provide weak support for rejecting H>θ. My take on this is that since such a research program seems to work in practice, falsifiability doesn’t fully describe how science works in this case (see Popper vs. Kuhn, Lakatos, Feyerabend, etc.).
Clinical data still exists that would allow a strategy to stop doing more tests at specific cut off point as the payoff from the hypothesis being right is dependent to the size of the effect and there will be clinical data at some point where the integral of payoff over lost clinical effects is small enough. It just gets fairly annoying to calculate. . Taking the strategy will be similar to gambling decision.
I do agree that there is a place for occam’s razor here but there exist no formalism that actually lets you quantify this weak support. There’s the Solomonoff induction, which is un-computable and awesome for work like putting an upper bound on how good induction can (or rather, can’t) ever be.
I would do the same thing I would do if my hay fever had gone away with no treatment.
And what would that be? Refuse to account for new evidence?
No, I mean I would be equally confused, because that’s exactly what happened. Homeopathy is literally water, so I would be just as confused if I had drank water and my hay fever subsided.
Also, completely separately, even if you took some drug that wasn’t just water, your case is anecdotal evidence. It should hold the same weight as any single case in any of the clinical trials. That means that it adds pretty much no data, and it’s correct to not update much.
Ah, but that’s not quite true. If a remedy helps you personally, it is less than a “single case” for clinical trial purposes, because it was not set up properly for that. You would do well to ignore it completely if you are calculating your priors for the remedy to work for you.
However, it is much more than nothing for you personally, once you have seen it working for you once. Now you have to update on the evidence. It’s a different step in the Bayesian reasoning: calculating new probabilities given that a certain event (in this case one very unlikely apriori—that the remedy works) actually happened.
hypothesize that they’re incompetent/dishonest at diluting and left some active agent in there.
OK, so you take it to your chem lab and they confirm that the composition is pure sugar, as far as they can tell. How many alternatives would you keep inventing before you update your probability of it actually working?
In other words, when do you update your probability that there is a sniper out there, as opposed to “there is a regular soldier close by”?
Before I update my probability of it working? The alternatives are rising to the surface because I’m updating the probability of it working.
OK, I suppose this makes sense. Let me phrase it differently. What personal experience (not published peer-reviewed placebo-controlled randomized studies) would cause you to be convinced that what is essentially magically prepared water is as valid a remedy as, say, Claritin?
Well, I hate to say this for obvious reasons, but if the magic sugar water cured my hayfever just once, I’d try it again, and if it worked again, I’d try it again. And once it had worked a few times, I’d probably keep trying it even if it occasionally failed.
If it consistently worked reliably I’d start looking for better explanations. If no-one could offer one I’d probably start believing in magic.
I guess not believing in magic is something to do with not expecting this sort of thing to happen.
(This tripped my positive bias sense: only testing the outcome in the presence of an intervention doesn’t establish that it’s doing anything. It’s wrong to try again and again after something seemed to work, one should also try not doing it and see if it stops working. Scattering anti-tiger pills around town also “works”: if one does that every day, there will be no tigers in the neighborhood.)
That’s a bad analogy. If “anti-tiger pills” repeatedly got rid of a previously observed real tiger, you would be well advised to give the issue some thought.
What’s that line about how, if you treat a cold, you can get rid of it in seven days, but otherwise it lasts a week?
You would still want to check to see whether tigers disappear even when no “anti-tiger pills” are administered.
Depending on how likely the tiger is to eat people if it didn’t disappear, and your probability of the pills successfully repelling the tiger given that it’s always got rid of the tiger, and the cost of the pill and how many people the tiger is expected to eat if it doesn’t disappear? Not always.
A medical example of this is the lack of evidence for the efficacy of antihistamine against anaphylaxis. When I asked my sister (currently going through clinical school) about why, she said “because if you do a study, people in the control group will die if these things work, and we have good reason to believe they do”
EDIT: I got beaten to posting this by the only other person I told about it
Yes. But the point is that this number should be negligible if you haven’t seen how the tiger behaves in the absence of the pills. (All of this assumes that you do not have any causal model linking pill-presence to tiger-absence.)
This case differs from the use of antihistamine against anaphylaxis for two reasons:
There is some theoretical reason to anticipate that antihistamine would help against anaphylaxis, even if the connection hasn’t been nailed down with double-blind experiments.
We have cases where people with anaphylaxis did not receive antihistamine, so we can compare cases with and without antihistamine. The observations might not have met the rigorous conditions of a scientific experiment, but that is not necessary for the evidence to be rational and to justify action.
Absolutely. The precise thing that matters is the probability tigers happen if you don’t use the pills. So, say, I wouldn’t recommend doing the experiment if you live in areas with high densities of tigers (which you do if there’s one showing up every day!) and you weren’t sure what was going into the pills (tiger poison?), but would recommend doing the experiment if you lived in London and knew that the pills were just sugar.
Similarly, I’m more likely to just go for a herbal remedy that hasn’t had scientific testing, but has lots of anecdotal evidence for lack of side-effects, than a homeopathic remedy with the same amount of recommendation.
It is positive bias (in that this isn’t the best way to acquire knowledge), but there’s a secondary effect: the value of knowing whether or not the magic sugar water cures his hayfever is being traded off against the value of not having hayfever.
Depending on how frequently he gets hayfever, and how long it took to go away without magic sugar water, and how bothersome it is, and how costly the magic sugar water is, it may be better to have an unexplained ritual for that portion of his life than to do informative experiments.
(And, given that the placebo effect is real, if he thinks the magic sugar water is placebo, that’s reason enough to drink it without superior alternatives.)
Agree with this. Knowing the truth has a value and a cost (doing the experiment).
I recently heard something along the lines of: “We don’t have proof that antihistamines work to treat anaphylaxis, because we haven’t done the study. But the reason we haven’t done the study is because we’re pretty sure the control group would die.”
I agree, I’d try not taking it too! I had hayfever as a child, and it was bloody awful. I used to put onion juice in my eyes because it was the only thing that would provide relief. But even as a child I was curious enough to try it both ways.
The placebo effect strikes me as a decent enough explanation.
Maybe, but it also explains why any other thing will cure my hayfever. And shouldn’t it go away if I realize it’s a placebo? And if I say ‘I have this one thing that cures my hayfever reliably, and no other thing does, but it has no mechanism except for the placebo effect’, is that very different from ‘I have this magic thing?’.
I’m not keen on explanations which don’t tell me what to anticipate. But maybe I misunderstand the placebo effect. How would I tell the difference between it and magic?
No, not very. Also, if it turns out that only this one thing works, and no other thing works, then (correcting for the usual expectation effects) that is relatively strong evidence that something more than the placebo effect is going on. Conversely, if it is the placebo effect, I would expect that a variety of substances could replace the sugar pills without changing the effect much.
Another way of putting this is, if I believe that the placebo effect is curing my hayfever, that ultimately means the power to cure my hayfever resides inside my brain and the question is how to arrange things so that that power gets applied properly. If I believe that this pill cures my hayfever (whether via “the placebo effect” or via “magic” or via “science” or whatever other dimly understood label I tack onto the process), that means the power resides outside my brain and the question is how to secure a steady supply of the pill.
Those two conditions seem pretty different to me.
Apparently not. The effect might be less, I don’t think the study checked. But once you know it’s a placebo and the placebo works, then you’re no longer taking a sugar pill expecting nothing, you’re taking a sugar pill expecting to get better.
You could tell the difference between the placebo effect and magic by doing a double blind trial on yourself. e.g. Get someone to assign either “magic” pill or identical sugar pill (or solution) with a random number generator for a period where you’ll be taking the drug, prepare them and put them in order for you to take on successive days, and write down the order to check later. Then don’t talk to them for the period of the experiment. (If you want to talk to them you can apply your own shuffle and write down how to reverse it)
Wait, what? I’m taking a stack of identical things and whether they work or not depends on a randomly generated list I’ve never seen?
Exactly. You write down your observations for each day and then compare them to the list to see if you felt better on days when you were taking the actual pill.
Only if it’s not too costly to check, of course, and sometimes it is.
Edit: I think gwern’s done a number of self-trials, though I haven’t looked at his exact methodology.
Edit again: In case I haven’t been clear enough, I’m proposing a method to distinguish between “sugar pills that are magic” and “regular sugar pills”.
Edit3: ninja’d
If you have a selection of ‘magic’ sugar pills, and you want to test them for being magic vs placebo effect, you do a study comparing their efficacy to that of ‘non-magic’ sugar pills.
If they are magic, then you aren’t comparing identical things, because only some of them have the ‘magic’ property
Well, you need it to work better than without the magic sugar water.
My approach is: I believe that the strategy of “if the magic sugar water worked with only 1 in a million probability of ‘worked’ being obtained by chance without any sugar water, and if only a small number of alternative cures were also tried, then adopt the belief that magic sugar water works” is a strategy that has only small risk of trusting in a non-working cure, but is very robust against unknown unknowns. It works even if you are living in a simulator where the beings-above mess with the internals doing all sorts of weird stuff that shouldn’t happen and for which you might be tempted to set very low prior.
Meanwhile the strategy of “make up a very low prior, then update it in vaguely Bayesian manner” has past history of screwing up big time leading up to significant preventable death, e.g. when antiseptic practices invented by this guy were rejected on the grounds of ‘sounds implausible’, and has pretty much no robustness against unknown unknowns, and as such is grossly irrational (in the conventional sense of ‘rational’) even though in the magic water example it sounds like awesomely good idea.
Precisely the hypotheses that are more likely than homeopathy. Once I’ve falsified those the probability starts pouring into homeopathy. Jaynes’ “Probability Theory: The Logic Of Science”, explains this really well in Chapter 4 and the “telepathy” example of Chapter 5. In particular I learnt a lot by staring at Figure 4.1.