While Bayesian statistics are obviously a useful method, I am dissatisfied with the way “Bayesianism” has become a stand-in for rationality in certain communities. There are well-developed, deep objections to this. Some of my favorite references on this topic:
Probability Theory Does Not Extend Logic by Dave Chapman. Part of what is missing from simulating every logically possible universe is indeed reasoning, in the sense that probabilistic inference nicely extends propositional logic but cannot solve problems in first order logic. This is why practical planning algorithms also use tools like tree search.
A substantial school in the philosophy of science identifies Bayesian inference with inductive inference and even rationality as such, and seems to be strengthened by the rise and practical success of Bayesian statistics. We argue that the most successful forms of Bayesian statistics do not actually support that particular philosophy but rather accord much better with sophisticated forms of hypothetico-deductivism. We examine the actual role played by prior distributions in Bayesian models, and the crucial aspects of model checking and model revision, which fall outside the scope of Bayesian confirmation theory
Even the standard construction of probability is potentially suspect. Why is the Dutch Book argument correct? There are serious potential problems with this, as John Norton has discussed The Material Theory of Induction, which also discusses the shortcomings of Bayesianism as a foundational inference method (chapter 10).
Bayesianism is ultimately a mathematical formalism that is actually useful only to the extent that two quantifications succeed in practice: the quantification of the state of the world into symbolic form, and the quantification of potential interventions and their results (necessary if we wish to connect reasoning to causation). There are many choices which need to be made both at the conceptual and the practical level when quantifying, as I have tried to discuss here in the context of data journalism. Quantification might be the least studied branch of statistics.
Finally, I’d note that explicitly Bayesian calculation is rarely used as the top level inference framework in practical decision-making, even when the stakes are high. I worked for a decade as a data journalist, and you’d think that if Bayesianism is useful anywhere then data journalists would use it to infer the truth of situations. But it is very rarely useful in practice. Nor is Bayesianism the primary method used in e.g. forecasting and policy evaluation. I think it’s quite instructive to ask why not, and I wish there was more serious work on this topic.
In short: Bayesianism is certainly foundational, but it is not a suitable basis for a general theory of rational action. It fails on both theoretical and practical levels.
I am happy that you mention Gelman’s book (I am studying it right now). I think lots of “naive strong bayesianists” would improve from a thoughtful study of the BDA book (there are lots of worked out demos and exercises available for it) and maybe some practical application of Bayesian modelling to some real-world statistical problems. The practice of “Bayesian way of life” of “updating my priors” sounds always a bit too easy in contrast to doing a genuine statistical inference.
For example, a couple of puzzles I am still myself unsure how to answer properly and with full confidence: Why one would be interested in doing stratified random sampling with your epidemiological study instead of naive “collect every data point that you see and then do a Bayesian update?” Or how multiple comparisons corrections for classical frequentist p-values map into Bayesian statistical framework? Does it matter for LWian Bayesianism if you are doing your practical statistical analyses with frequentist or Bayesian analysis tools (especially if many frequentist methods can be seen as clever approximations to full Bayesian model, see e.g. discussion of Kneser-Ney smoothing as ad hoc Pitman-Yor process inference here: https://cs.stanford.edu/~jsteinhardt/stats-essay.pdf ; similar relationship exists between k-means and EM-algorithm of Gaussian mixture model.) And if there is no difference, is the philosophical Bayesianism then actually that important—or important at all—for rationality?
While Bayesian statistics are obviously a useful method, I am dissatisfied with the way “Bayesianism” has become a stand-in for rationality in certain communities. There are well-developed, deep objections to this. Some of my favorite references on this topic:
Probability Theory Does Not Extend Logic by Dave Chapman. Part of what is missing from simulating every logically possible universe is indeed reasoning, in the sense that probabilistic inference nicely extends propositional logic but cannot solve problems in first order logic. This is why practical planning algorithms also use tools like tree search.
Philosophy and the Practice of Bayesian Statistics by Andrew Gelman (who wrote the book on Bayesian methods) and Cosma Shalizi. Abstract:
Even the standard construction of probability is potentially suspect. Why is the Dutch Book argument correct? There are serious potential problems with this, as John Norton has discussed The Material Theory of Induction, which also discusses the shortcomings of Bayesianism as a foundational inference method (chapter 10).
Bayesianism is ultimately a mathematical formalism that is actually useful only to the extent that two quantifications succeed in practice: the quantification of the state of the world into symbolic form, and the quantification of potential interventions and their results (necessary if we wish to connect reasoning to causation). There are many choices which need to be made both at the conceptual and the practical level when quantifying, as I have tried to discuss here in the context of data journalism. Quantification might be the least studied branch of statistics.
Finally, I’d note that explicitly Bayesian calculation is rarely used as the top level inference framework in practical decision-making, even when the stakes are high. I worked for a decade as a data journalist, and you’d think that if Bayesianism is useful anywhere then data journalists would use it to infer the truth of situations. But it is very rarely useful in practice. Nor is Bayesianism the primary method used in e.g. forecasting and policy evaluation. I think it’s quite instructive to ask why not, and I wish there was more serious work on this topic.
In short: Bayesianism is certainly foundational, but it is not a suitable basis for a general theory of rational action. It fails on both theoretical and practical levels.
I am happy that you mention Gelman’s book (I am studying it right now). I think lots of “naive strong bayesianists” would improve from a thoughtful study of the BDA book (there are lots of worked out demos and exercises available for it) and maybe some practical application of Bayesian modelling to some real-world statistical problems. The practice of “Bayesian way of life” of “updating my priors” sounds always a bit too easy in contrast to doing a genuine statistical inference.
For example, a couple of puzzles I am still myself unsure how to answer properly and with full confidence: Why one would be interested in doing stratified random sampling with your epidemiological study instead of naive “collect every data point that you see and then do a Bayesian update?” Or how multiple comparisons corrections for classical frequentist p-values map into Bayesian statistical framework? Does it matter for LWian Bayesianism if you are doing your practical statistical analyses with frequentist or Bayesian analysis tools (especially if many frequentist methods can be seen as clever approximations to full Bayesian model, see e.g. discussion of Kneser-Ney smoothing as ad hoc Pitman-Yor process inference here: https://cs.stanford.edu/~jsteinhardt/stats-essay.pdf ; similar relationship exists between k-means and EM-algorithm of Gaussian mixture model.) And if there is no difference, is the philosophical Bayesianism then actually that important—or important at all—for rationality?