A nice middle-ground between purely Bayesian and purely frequentist methods is to use a Bayesian model coupled with frequentist model-checking techniques; this gives us the freedom in modeling afforded by a prior but also gives us some degree of confidence that our model is correct. This approach is suggested by both Gelman [9] and Jordan [10].
Just to pile on a little bit here: A Bayesian might argue that uncertainty about which model you’re using is just uncertainty, so put a prior on the space of possible models and do the Bayesian update. This can be an effective method, but it doesn’t entirely get rid of the problem—now you’re modeling the structure of your uncertainty about models in a particular way, and that higher level model could be wrong. You’re also probably excluding some plausible possible models, but I’ll sidestep that issue for now. The Bayesian might argue that this case is analogous to the previous—just model your (model of the structure of uncertainty about models) - put a prior on that space too. But eventually this must stop with a finite number of levels of uncertainty, and there’s no guarantee that 1) your model is anywhere near the true model (i.e. the actual structure of your uncertainty) or 2) you’ll be able to get answers out of the mess you’ve created.
On the other hand, frequentist model checking techniques can give you a pretty solid idea of how well the model is capturing the data. If one model doesn’t seem to be working, try another instead! Now a Bayesian might complain that this is “using the data twice” which isn’t justified by probability theory, and they would be right. However, you don’t get points for acting like a Bayesian, you get points for giving the same answer as a Bayesian. What the Bayesian in this example should be worried about is whether the model chosen at the end ultimately gives answers that are close to what a true Bayesian would give. Intuitively, I think this is the case—if a model doesn’t seem to fit the data by some frequentist model checking method, e.g. a goodness of fit test, then it’s likely that if you could actually write down the posterior probability that the particular model you chose is true (i.e. it’s the true structure of your uncertainty), that probability would be small, modulo a high degree of prior certainty that the model was true. But I’m willing to be proven wrong on this.
Just to pile on a little bit here: A Bayesian might argue that uncertainty about which model you’re using is just uncertainty, so put a prior on the space of possible models and do the Bayesian update. This can be an effective method, but it doesn’t entirely get rid of the problem—now you’re modeling the structure of your uncertainty about models in a particular way, and that higher level model could be wrong. You’re also probably excluding some plausible possible models, but I’ll sidestep that issue for now. The Bayesian might argue that this case is analogous to the previous—just model your (model of the structure of uncertainty about models) - put a prior on that space too. But eventually this must stop with a finite number of levels of uncertainty, and there’s no guarantee that 1) your model is anywhere near the true model (i.e. the actual structure of your uncertainty) or 2) you’ll be able to get answers out of the mess you’ve created.
On the other hand, frequentist model checking techniques can give you a pretty solid idea of how well the model is capturing the data. If one model doesn’t seem to be working, try another instead! Now a Bayesian might complain that this is “using the data twice” which isn’t justified by probability theory, and they would be right. However, you don’t get points for acting like a Bayesian, you get points for giving the same answer as a Bayesian. What the Bayesian in this example should be worried about is whether the model chosen at the end ultimately gives answers that are close to what a true Bayesian would give. Intuitively, I think this is the case—if a model doesn’t seem to fit the data by some frequentist model checking method, e.g. a goodness of fit test, then it’s likely that if you could actually write down the posterior probability that the particular model you chose is true (i.e. it’s the true structure of your uncertainty), that probability would be small, modulo a high degree of prior certainty that the model was true. But I’m willing to be proven wrong on this.