But now we have a problem. For every ϵ,∏ni=1U(Bi),P(A|B1,B2,…,Bn)>0, there exists P(A)>0 such that P(A|B1,B2,…,Bn)<ϵ. In other words, for any finite observable evidence, there exists a sufficiently strong prior such that we can ignore the evidence.
I would have worded this differently. For any finite set of pieces of evidence, a sufficiently strong prior can make our posterior probability arbitrarily low.
Practically speaking, this may cause us to disregard the evidence. This would be akin to resisting a Pascal’s mugging. Imagine I do a literature review on the evidence for telepathy, put all the pieces of evidence along with my prior into a spreadsheet, and crunch the numbers. Let’s say the result is that my priors on telepathy being real remain extremely low. You then run another study on telepathy that again finds that it’s real. I might decide that it’s not even worth my time to plug the data from your study into my spreadsheet.
I think that this is a very real issue. In addition to governing where people place their attention, I also see this turning up in the cocktail napkin math that people do. When they make up statistics for napkin math, somehow “low probability” is always 1%, rather than 0.1% or 0.01%. You can prove anything with 1% probabilities. I would love it if we could shift toward insisting on at least one piece of concrete evidence justifying the chosen order of magnitude.
In favor of Bayesianism being “more objective” than you’re making it out to be is that there’s no mathematical reason to have a threshold for our prior below which we disregard evidence. We can also make purely objective statements about the Bayesian chains you outlined above: “if your prior is X1, then your posterior is Y1 given this evidence. If your posterior is X2, then your posterior is Y2 given the same evidence.”
In favor of frequentism being “more subjective” that you’re making it out to be is that it still requires us to make assumptions about the underlying distribution when computing a confidence interval, although there are also techniques that let us evaluate the fit of various standard distributions to our data. However, there is no guarantee that a given dataset fits any standard distribution. Deciding that an underlying distribution is correct, best, or good enough for our purposes is a subjective decision. It seems to me that with frequentist statistics, the term by which the objective calculation becomes a subjective interpretation is left out of the statistic, whereas in Bayesian statistics it is stated explicitly. It seems to me that the latter approach is more clear.
As far as standards for science, one way Bayesian statistics could be used in a more objective or standards-based manner is by choosing a “standard prior” and “standard posterior” for a given field in order for findings to be considered “significant.” This would be analogous to a field-specific standard p value (i.e. of 0.05). I imagine this would run into all the same issues: instead of p-hacking, we’d have posterior-hacking. It’s just not clear to me that Bayesian statistics is any more fundamentally broken than frequentism.
Practically speaking, this may cause us to disregard the evidence. This would be akin to resisting a Pascal’s mugging. Imagine I do a literature review on the evidence for telepathy, put all the pieces of evidence along with my prior into a spreadsheet, and crunch the numbers. Let’s say the result is that my priors on telepathy being real remain extremely low. You then run another study on telepathy that again finds that it’s real. I might decide that it’s not even worth my time to plug the data from your study into my spreadsheet.
Imagine you live earlier in history. Attempts to triangulate the distance away from the earth that stars are fails—in order to calculate that you need two perspectives that are far enough apart. But the method fails. And this tells you stars are unimaginably far away.
After looking at the numbers, you discount this possibility. There’s no way they’re that far away.
I would have worded this differently. For any finite set of pieces of evidence, a sufficiently strong prior can make our posterior probability arbitrarily low.
Practically speaking, this may cause us to disregard the evidence. This would be akin to resisting a Pascal’s mugging. Imagine I do a literature review on the evidence for telepathy, put all the pieces of evidence along with my prior into a spreadsheet, and crunch the numbers. Let’s say the result is that my priors on telepathy being real remain extremely low. You then run another study on telepathy that again finds that it’s real. I might decide that it’s not even worth my time to plug the data from your study into my spreadsheet.
I think that this is a very real issue. In addition to governing where people place their attention, I also see this turning up in the cocktail napkin math that people do. When they make up statistics for napkin math, somehow “low probability” is always 1%, rather than 0.1% or 0.01%. You can prove anything with 1% probabilities. I would love it if we could shift toward insisting on at least one piece of concrete evidence justifying the chosen order of magnitude.
In favor of Bayesianism being “more objective” than you’re making it out to be is that there’s no mathematical reason to have a threshold for our prior below which we disregard evidence. We can also make purely objective statements about the Bayesian chains you outlined above: “if your prior is X1, then your posterior is Y1 given this evidence. If your posterior is X2, then your posterior is Y2 given the same evidence.”
In favor of frequentism being “more subjective” that you’re making it out to be is that it still requires us to make assumptions about the underlying distribution when computing a confidence interval, although there are also techniques that let us evaluate the fit of various standard distributions to our data. However, there is no guarantee that a given dataset fits any standard distribution. Deciding that an underlying distribution is correct, best, or good enough for our purposes is a subjective decision. It seems to me that with frequentist statistics, the term by which the objective calculation becomes a subjective interpretation is left out of the statistic, whereas in Bayesian statistics it is stated explicitly. It seems to me that the latter approach is more clear.
As far as standards for science, one way Bayesian statistics could be used in a more objective or standards-based manner is by choosing a “standard prior” and “standard posterior” for a given field in order for findings to be considered “significant.” This would be analogous to a field-specific standard p value (i.e. of 0.05). I imagine this would run into all the same issues: instead of p-hacking, we’d have posterior-hacking. It’s just not clear to me that Bayesian statistics is any more fundamentally broken than frequentism.
Imagine you live earlier in history. Attempts to triangulate the distance away from the earth that stars are fails—in order to calculate that you need two perspectives that are far enough apart. But the method fails. And this tells you stars are unimaginably far away.
After looking at the numbers, you discount this possibility. There’s no way they’re that far away.