I’ve come to the conclusion that the strength and weakness of the Bayesian method is that it’s simultaneously your experimental procedure and your personal decision-making method (given a utility function, that is). This makes it more powerful, but can also make getting a good answer more demanding.
The strength is that it makes mathematical things that are done informally in frequentism. This is the way in which Bayes makes assumptions explicit that frequentism hides. For example, suppose a frequentist does an experiment that shows that broccoli causes cancer with 99% significance. Frequentist: “If broccoli didn’t cause cancer, there would be a 1% chance of a result like that happening. That’s an absolute fact, none of your Bayesian assumptions. So let it henceforth be Science that broccoli causes cancer.” Bayesian: “Wait, that last sentence wasn’t justified! In order to take any action based on your results, you have to invoke some kind of prior. What would you do if you got the same result with ‘the transit of venus’ in the place of ‘broccoli’? You could just kind of wing it, but then you’re hiding assumptions.”
The weakness is that a human is liable to put too many zeroes in the ‘absurd’ prior that the transit of Venus causes cancer and end up ignoring overwhelming evidence if it turns out that this effect is somehow real. In an open scientific setting, sharing the likelihood ratio might solve the problem here, but if you give your machine learning algorithm a bad prior, that might not get noticed. And if you forget to give your rock-paper-scissors bot a nonzero prior that the opponent has its source code, it can get beaten 100% of the time by an opponent who can simulate it (since it’ll have no reason to revert to random).
Frequentists might do better by implicitly using a ‘one-size-fits-all’ prior that beats what we would write down (because we’re not perfect Bayesians). This is the only advantage of frequentism I can see apart from computational concerns.
I’ve come to the conclusion that the strength and weakness of the Bayesian method is that it’s simultaneously your experimental procedure and your personal decision-making method (given a utility function, that is). This makes it more powerful, but can also make getting a good answer more demanding.
The strength is that it makes mathematical things that are done informally in frequentism. This is the way in which Bayes makes assumptions explicit that frequentism hides. For example, suppose a frequentist does an experiment that shows that broccoli causes cancer with 99% significance. Frequentist: “If broccoli didn’t cause cancer, there would be a 1% chance of a result like that happening. That’s an absolute fact, none of your Bayesian assumptions. So let it henceforth be Science that broccoli causes cancer.” Bayesian: “Wait, that last sentence wasn’t justified! In order to take any action based on your results, you have to invoke some kind of prior. What would you do if you got the same result with ‘the transit of venus’ in the place of ‘broccoli’? You could just kind of wing it, but then you’re hiding assumptions.”
The weakness is that a human is liable to put too many zeroes in the ‘absurd’ prior that the transit of Venus causes cancer and end up ignoring overwhelming evidence if it turns out that this effect is somehow real. In an open scientific setting, sharing the likelihood ratio might solve the problem here, but if you give your machine learning algorithm a bad prior, that might not get noticed. And if you forget to give your rock-paper-scissors bot a nonzero prior that the opponent has its source code, it can get beaten 100% of the time by an opponent who can simulate it (since it’ll have no reason to revert to random).
Frequentists might do better by implicitly using a ‘one-size-fits-all’ prior that beats what we would write down (because we’re not perfect Bayesians). This is the only advantage of frequentism I can see apart from computational concerns.