Given that both are competent and Bob doesn’t have strong priors X should be about the same as Y.
Why? X is P(results >= what we saw | effect = 0), whereas Y is P(effect < costs | results = what we saw). I can see no obvious reason those would be similar, not even if we assume costs = 0; p(results = what we saw | effect = 0) = p(effect = 0 | results = what we saw) iff p_{prior}(result = what we saw) = p_{prior}(effect = 0) (where the small p’s are probability densities, not probability masses), but that’s another story.
You have two samples: one was given the drug, the other was given the placebo. You have some metric for the effect you’re looking for, a value of interest.
The given-drug sample has a certain distribution of the values of your metric which you model as a random variable. The given-placebo sample also has a distribution of these values (different, of course) which you also model as a random variable.
The statistical questions are whether these two random variables are different, in which way, and how confident you are of the answers.
For simple questions like that (and absent strong priors) the frequentists and the Bayesians will come to very similar conclusions and very similar probabilities.
For simple questions like that (and absent strong priors) the frequentists and the Bayesians will come to very similar conclusions and very similar probabilities.
Yes, but the p-value and the posterior probability aren’t even the same question, are they?
Why? X is P(results >= what we saw | effect = 0), whereas Y is P(effect < costs | results = what we saw). I can see no obvious reason those would be similar, not even if we assume costs = 0; p(results = what we saw | effect = 0) = p(effect = 0 | results = what we saw) iff p_{prior}(result = what we saw) = p_{prior}(effect = 0) (where the small p’s are probability densities, not probability masses), but that’s another story.
You have two samples: one was given the drug, the other was given the placebo. You have some metric for the effect you’re looking for, a value of interest.
The given-drug sample has a certain distribution of the values of your metric which you model as a random variable. The given-placebo sample also has a distribution of these values (different, of course) which you also model as a random variable.
The statistical questions are whether these two random variables are different, in which way, and how confident you are of the answers.
For simple questions like that (and absent strong priors) the frequentists and the Bayesians will come to very similar conclusions and very similar probabilities.
Yes, but the p-value and the posterior probability aren’t even the same question, are they?
No, they are not.
However for many simple cases—e.g. where we are considering only two possible hypotheses—they are sufficiently similar.