Since when are Bayesians enamored of the results of trials against a null hypothesis that throws away almost all the relevant information about you, and leaves the decision about treatment for you up to the results of a 95% confidence interval on people who are not you, for whom you’ve also thrown away (or never collected) most of the relevant data?
You say that like it isn’t evidence, rather than simply being less powerful than it could have been. There is only a 5% chance of getting a false positive with a 95% confidence interval. Ignoring additional evidence will not change that.
but the best indicator of whether someone works for you is a trial on you.
It’s hard to tell if something’s working if you use it on one person with no control group. You can’t tell how much of what symptoms are caused by the drug.
Maybe it’s just because I’m a little older than the mean here, but I don’t think I have decades to wait for decades of longitudinal studies on life extension strategies
You certainly can’t find life-extension stuff that way. By the time you know how well it works, it’s too late to decide whether or not to use it.
You say that like it isn’t evidence, rather than simply being less powerful than it could have been. There is only a 5% chance of getting a false positive with a 95% confidence interval. Ignoring additional evidence will not change that.
But that does not tell you that how likely a given positive result is a false positive, you’d also need to know what fraction of (implicitly) tested hypotheses is true.
If it’s on the border of the confidence interval, the probability of false positive is 50%. If it’s twice as far as that, it’s 5%. That should give an okay idea of where the range is for that. I’d much prefer something with a 99% confidence interval, but a 95% one is still pretty good. If the effect is just barely statistically significant, the odds ratio is still 10:1
I’m not sure how likely it is for their hypothesis to be true, but it’s likely enough for them to risk spending money checking.
You say that like it isn’t evidence, rather than simply being less powerful than it could have been. There is only a 5% chance of getting a false positive with a 95% confidence interval. Ignoring additional evidence will not change that.
There are so many things wrong with this paragraph, but I’ll limit my comments.
Where did I state or imply that the results of a clinical trial are not evidence?
What epistemic claims do you think you’re justified in making about an individual based on a failure to reject a null hypothesis for a sample statistic for a population for a particular treatment regiment?
I would note, that if you’re a Frequentist, failure to reject an hypothesis is technically, literally, no evidence of anything. But I”m not a frequentist, so I’m free to actually use the data collected in this “failure” for my inferences, but generally, failures to reject at 95% proves little.
Ignoring additional evidence will not change that.
And ignoring the evidence that something works for you will not change whether it work for you either.
It’s hard to tell if something’s working if you use it on one person with no control group.
Really? I can’t sleep. I take a pill, and I fall asleep. I don’t take a pill, and I don’t fall asleep. How many trials of that do you need to be convinced?
You certainly can’t find life-extension stuff that way. By the time you know how well it works, it’s too late to decide whether or not to use it.
You can’t find a lot of solutions to medical problems that way, because there is an ocean of problems that no one has the financial interest to spend millions of dollars for the clinical trial.
You admit that you can’t find life extension solutions that way. Maybe you’ll equally admit that the same applies to a host of other solutions. Whatever you include in the class of “not amenable to solution by clinical trial”, what is your solution in those cases? Do nothing?
Where did I state or imply that the results of a clinical trial are not evidence?
You certainly didn’t state it, but I got the impression that you don’t think you should pay much attention to clinical trials. Perhaps I inferred what wasn’t there.
but generally, failures to reject at 95% proves little.
If failure to reject the null hypothesis proves little, then you’re unlikely to reject it even if it’s false, which raises the question of why you did the study in the first place. You clearly should have made a big enough study to actually notice things.
Really? I can’t sleep. I take a pill, and I fall asleep. I don’t take a pill, and I don’t fall asleep. How many trials of that do you need to be convinced?
I guess in that case (and many others, such as pain relief) it would work. The example in the original comment was life extension, where it clearly would not. Even so, you could rely on studies to tell you how likely it is to work, and how well it will work if it does. You also can’t just rely on testing them yourself, given that only a tiny fraction of things help you sleep, but the same goes for studies.
You admit that you can’t find life extension solutions that way.
I meant that you can’t find them by testing them on yourself, since you can only do it once, unlike something to help you sleep. You can find them by clinical trials. You just saw an example of that.
You say that like it isn’t evidence, rather than simply being less powerful than it could have been. There is only a 5% chance of getting a false positive with a 95% confidence interval. Ignoring additional evidence will not change that.
It’s hard to tell if something’s working if you use it on one person with no control group. You can’t tell how much of what symptoms are caused by the drug.
You certainly can’t find life-extension stuff that way. By the time you know how well it works, it’s too late to decide whether or not to use it.
But that does not tell you that how likely a given positive result is a false positive, you’d also need to know what fraction of (implicitly) tested hypotheses is true.
If it’s on the border of the confidence interval, the probability of false positive is 50%. If it’s twice as far as that, it’s 5%. That should give an okay idea of where the range is for that. I’d much prefer something with a 99% confidence interval, but a 95% one is still pretty good. If the effect is just barely statistically significant, the odds ratio is still 10:1
I’m not sure how likely it is for their hypothesis to be true, but it’s likely enough for them to risk spending money checking.
There are so many things wrong with this paragraph, but I’ll limit my comments.
Where did I state or imply that the results of a clinical trial are not evidence?
What epistemic claims do you think you’re justified in making about an individual based on a failure to reject a null hypothesis for a sample statistic for a population for a particular treatment regiment?
I would note, that if you’re a Frequentist, failure to reject an hypothesis is technically, literally, no evidence of anything. But I”m not a frequentist, so I’m free to actually use the data collected in this “failure” for my inferences, but generally, failures to reject at 95% proves little.
And ignoring the evidence that something works for you will not change whether it work for you either.
Really? I can’t sleep. I take a pill, and I fall asleep. I don’t take a pill, and I don’t fall asleep. How many trials of that do you need to be convinced?
You can’t find a lot of solutions to medical problems that way, because there is an ocean of problems that no one has the financial interest to spend millions of dollars for the clinical trial.
You admit that you can’t find life extension solutions that way. Maybe you’ll equally admit that the same applies to a host of other solutions. Whatever you include in the class of “not amenable to solution by clinical trial”, what is your solution in those cases? Do nothing?
You certainly didn’t state it, but I got the impression that you don’t think you should pay much attention to clinical trials. Perhaps I inferred what wasn’t there.
If failure to reject the null hypothesis proves little, then you’re unlikely to reject it even if it’s false, which raises the question of why you did the study in the first place. You clearly should have made a big enough study to actually notice things.
I guess in that case (and many others, such as pain relief) it would work. The example in the original comment was life extension, where it clearly would not. Even so, you could rely on studies to tell you how likely it is to work, and how well it will work if it does. You also can’t just rely on testing them yourself, given that only a tiny fraction of things help you sleep, but the same goes for studies.
I meant that you can’t find them by testing them on yourself, since you can only do it once, unlike something to help you sleep. You can find them by clinical trials. You just saw an example of that.