While you claim to accept 6 and 10, your answer to 9 seems to suggest that for something labeled Bayesian, you do in fact believe your criterion is universal, that no one should be allowed to use any Bayesian procedure unless it satisfies your particular notion of optimality. Respectfully, I think it’s indisputably true that:
The goal of approximating a single / fixed mental posterior with bounded effort is completely reasonable and coherent.
There are many cases where that goal is best served by choosing your prior in a data-dependent way. If your data happens to show you that the exact amount of prior weight you give to the the [0.90 to 0.91] intervals is crucial, then you would give that interval some extra thought. This will improve performance on the above goal, relative to picking a fixed partition.
In short, I continue to believe my post is correct.
There is no “particular notion of optimality”, nor is it “my criterion”, these are just the properties of Bayesian conditioning. What is provided here is instead a rationalization, which has no truth-tracking properties, Bayesian or otherwise. Approximating a single fixed posterior is coherent (which is to say, it is legitimate to elicit it). Justifying an approximation to a posterior by pretending to perform an update on old evidence after changing the likelihood is not; the resulting numbers are not meaningful (notice Mateusz’s comment elsewhere stating that deconditioning should return the deconditioned probabilities, which is what fails here for the likelihood). “Without seeing the poll results, I never would’ve predicted that I should control for age and class. The answer is much less insane (p<0.05) if I report the likelihood I would have assigned” - π-hacking. When one is being Bayesian, one should strive to be Bayesian.
To put it another way, the post goes: The (Bayes) machine is provided poor inputs (misspecified prior and likelihood) and one receives a poor output. The proposal is to kick the machine (change the marginal likelihood post-hoc); one provides an example where the output is less indecent. All of the screws come loose (coherence, the defining property of doing Bayes, no longer holds). Possibly the internal gears are still fine (it has completely undefined truth-tracking properties). It seems as though it can just as easily take good inputs and return bad outputs, or in fact output just about anything (it can produce any, arbitrary posterior odds for the arbitrary likelihood ratios). No mention is made to nonviolent methods, and the risks are not properly warned against.
We may have reached the limits of this debate, but I’ll state my position one last time: I believe that the abstraction of “approximate the posterior you would find if you had an infinite amount of time to think about it, while using a less-than-infinite amount of time” is completely reasonable, and in fact is a good model of a situation people often face in practice. And I believe that under that abstraction, making your prior after looking at the data (rather than before) is often a good idea, provided you do it carefully. I think you have offered no real counterargument, but instead gesturing at abstract definitions. But those abstract definitions are just ways of smuggling in a different abstraction, under which the problem I am trying to solve doesn’t exist. I understand that a solution built for one abstraction may have poor properties under a different one, but that doesn’t impress me.
I appreciate you taking so much time to discuss this, and I’d like to take something productive from this discussion, but I find it very hard, because you seem determined to show that the post is “fractally wrong”, because you seem unwilling to make any durable concessions, and because you seem to interpret every statement I make in the least-charitable possible way. If you’re truly trying to help me make the post better, that will be impossible unless you adopt a less adversarial posture.
Look, let me explain my perspective, here, then. The discussion took place on 3 entirely different ‘epochs’ (one 7 months ago, then another 6 months ago), and between one of these you directly told me to take a different angle; what you perceive as me being “determined to show that the post is fractally wrong” is, to me, just a result of rereading the full thing and finding different problems.
I will concede that my first approach was misguided; I was more appalled by the combination of title and closing thesis as not properly discussing the cons, and thought the useful thing to post for others to read was, roughly, the costs related to going down this road (incoherence, that is, credence-numbers that are not obtained by Bayes updating and as such do not have the advantageous properties one purportedly uses and defends Bayes numbers through, and of the kind that is hard to repair or argue is ‘approximately’ coherent in some implicitly-Bayesian sense) and the many unmentioned alternatives. I still think a better article would do both of those; I do not think this would make it too technical, especially the former.
Then you told me the thing has ‘specific rules’; you specifically used the word ‘rules’. I went and looked! That’s been the attempt, here, with what you call a ‘different abstraction’. My conclusion so far is that I have found no viable, good formalization of this idea; it either collapses back in the coarse model (insofar as one wants to keep coherence) or it has no properties that I can state as good or bad (if one allows the marginal likelihood to shift away). I think that the non-technicality of the post has led to an idea that is unsalvageable, at least as a procedure with ‘rules’, in the current form. If you wanted me to discuss it as a practical, pragmatic device for thinking about things in a pseudo-Bayesian way, then, I think I get to pout at the fact that you pointed me in the exact opposite direction. (Not only that, but, when I stated my understanding of the thing, you took it upon yourself to agree that it was, at least roughly, correct!)
So, okay. I will put away the snark and the rhetoric and just state my problems with the thing directly. Afterwards I will elaborate on what I believe can be patched, what I believe lacks nuance and warning as presented (of the kind that you have displayed in these comments after), and what I cannot help but conclude is just misguided, or at least requires substantial refinement. For whatever reason I think it is best to express these in pairs. Apologies for length.
Coherence is hard. Doing a full analysis of any problem, even simple, informal checks, while staying fully within the Bayesian formalism is sometimes basically impossible.
However, it is not impossible. Sometimes, with a little difficulty, it can be done. This often requires care; it sometimes requires accepting things that seem unintuitive or even nonsensical. This is because most of the results that point to Bayesianism as the thing to do, the reasons why one performs Bayesian updating and not Dempster-Shafer updating or Jeffrey updating or ad hoc numerical reasoning, usually come in the form of ’if and only if’s; the Dutch book converse, the CCT converse, the accuracy theorems, Cox’s theorem, forecasts-as-martingales, all of these uniquely identify probabilities-as-credences or Bayes updating as, specifically and uniquely, the thing to do; if you end up doing something else, you lose the properties instantly.
This difficulty means that, often, approximating coherence is essentially required.
If and only ifs are very fragile things; you step away from the formalism in one subtle way and all of these things have suddenly broken into pieces somewhere. What you must ensure, then, is that most of the pieces are still there. Sometimes this can be done formally; I mentioned logical induction; there it is shown that one is only boundedly Dutch-Booked in a relatively benign way. Safebayes is a more pragmatic one; it is very clear to see how that is very approximately Bayesian. Empirical Bayes as approximate hierarchical Bayes.
Sometimes the approximation looks right, and becomes unsalvageable very quickly, in some mathematical sense. This is the case for many objective priors; everyone can nod at the univariate Jeffreys’ rule, but not even Jeffreys defended the multivariate one. You may fondly remember the right-Haar prior, but the left-Haar inspires little fondness. Very classically, nobody puts up a fight to defend flat priors anymore. And so on. I tried putting your procedure in some formalizations and this seemed to be the case; but you (now) reject them, so I will not reiterate.
Sometimes, there is no formal procedure to analyse the proximity of.
Other times, the approximate-incoherence is incurred from some (sometimes necessary) pragmatic data-procedure. You change the prior after the sampler runs for too long, you pick the best model out of a bag after looking at a predictive score, you throw out the model because the predictions look kind of bad; nobody (in the 21st century) would dispute that these things happen.
However, what is missing in your post and not missing whenever Bayesian statisticians discuss these things is that when you do this you are treading very dangerous ground. So, generally, the best of them (which is to say, Gelman’s cohort, Stephen Walker, Rubin, etc), if one cannot work out some alternative justification or method to do these things that is still within the formalism (as with Gelman in the model expansion cases, and in Rubin’s case something as trivial as randomization), at the very least you run a bunch of checks to see that you are really not shooting yourself in the foot somewhere.
This is because, when you do abandon coherence in a way that does not give you some kind of ‘bounded’ error, even in some conceptual sense (as in your case, where the marginal likelihood shifts arbitrarily; I will reiterate that the problem is not the prior becoming more detailed, but the marginal likelihood shifting alongside it in a way that does not reduce the model back to it), you really are playing with fire. The numbers obtained by such a method do not have the philosophical clarity of the numbers obtained with Bayesian updating, even if you could imagine that you might have obtained the numbers through some Bayesian update that was not performed.
I understand that you may raise an eyebrow at this, but, the analogy with p-hacking is relevant: you do a little tweak here or there, in ways that might make a lot of common sense, hell, sometimes, it really would be better if you had ran the analysis you wish you had ran after you saw the data; and, really, if the null your tweak led to be rejected really was wrong, then one may feel some solace that no damage was done. But, still, the ‘0.05’ number just no longer means a 0.05 error rate; the number has ceased having a meaning. You may run some garden-of-forking-paths analysis that might elucidate what that number might mean, but it is no longer the p-value it purported itself to be.
Besides, frankly, sometimes you have to put common sense and other statistical properties over following Bayesian edicts directly.
So, lacking a formalization through which to analyse approximation quality and observing that the procedure can output ultimately arbitrary results (thus even a conceptual approach to approximation seems hard), since they are no longer clean Bayes-rule probabilities obtained from a prior and posterior update, what are they for?
It is easy to make this kind of procedure look good if one picks out examples where it happens to reach the correct answer (when the null is false and you performed a terrible analysis, p-hacking can look absurdly, genuinely convincing!). I don’t know about you, but, for me, and I think for most statisticians, we like to look at what happens if the person doing it is maybe a little tendentious someplace; maybe not too disciplined here or there. A statistical (or a Bayesian) justification can often serve as a little “certificate of formal correctness” for people who will then no longer squint when they do some things that they really should squint at.
So, how does this procedure fare in this respect? I honestly think that the infamous “Bayesian arguments for God” are, frankly, not too far away from this; you could probably put them in exactly this language, or derive them in this way. “I first took a prior and likelihood, did an update, it did not seem sensible to me; I then found the little tweak to the model that made it make sense with my intuition, so now I have obtained a Bayesian justification for my belief”; I think you know how bad this can be. I am not saying this to imply that the idea fares as poorly as that in general, I imagine in most situations it acts moderately more benignly, but that it is a natural thing to reach for, and easy to imagine that you are doing nothing wrong through, regardless of how right you are (or aren’t).
Of course, you say that things should only be done in the right way, in the post. You do not, I think, specify how to do this; honestly the bisection you performed in the aliens example would look pretty tendentious to an aliens-believer, if you imagine that a few hundred years from now we discover that aliens really are real I think you know that the toy example looks pretty bad. A demonstration of a good method survives the particular example being wrong; one can at least be wrong gracefully, or at least where the flaw lies can be shown to subside in the specifications of the method.
In any event, this is a good model for something that is encountered in practice.
I think that you are right, and indeed you are right that people do some version of this, often. But, one virtue I can name of the Bayesians of the 21st century at least is that they do not do this, typically; if their model was wrong, they are fine eating the bullet, without performing any rigmarole that puts their objectivity into question, they are fine with putting “money on the table”. Here, you salvage a broken inference by a method that simply would not convince a skeptic. But, the problem that you salvage lies in the likelihood, not in your prior; just allow yourself to observe some more data and you’ll be fine, or split the sample first, or cautiously observe through a prior predictive check that your model allows no information to be gained regardless of what happens, or in fact just obtain a better prior (I believe witness unreliability to be the Hume thesis on miracles in the 1700s, so it can certainly be prior to the data in this example to consider it).
Ultimately, when I looked for rules, all I found was the rough sense that the posterior should match one’s intuition. Of course intuition is often right and good, possibly often better than our Bayesian calculations. But, ultimately, I think that performing this kind of calculation to affirm one’s intuitions is at best just an elucidation of at least part of why you think you think the things you think; at worst, and to be frank, very typically in this framing at least, a way to smuggle vague intuition as formally justifiable insight. The best cases of Bayesian inference are when you perform it and convince yourself out of an intuition that was actually wrong; spiritually, at least, this seems to basically prohibit this, there is always an out, always a way to slice up the likelihood to get numbers that you are more concordant with.
I think I will finish on that note. On how the idea can be patched: As Mateusz mentioned earlier, having the partition be data-independent solves the coherency issue entirely (qua Savage). Or, at least, a formal justification that binds one not to change the likelihood too much. Or, in some sense an argument that one will not easily mislead themselves, that one will not be able to convince themselves of false conclusions as easily as true ones, or a sensitivity analysis of some kind that shows, okay, maybe your new flimsy post-hoc model is not too flimsy. Or, as a last resort, if you want a number for an intuition, you can elicit a full posterior from your intuitions via all the other methods in a way that doesn’t make it look like reasoning was performed when it really wasn’t. Mostly, if the idea has formalizable value, it ought to be formalized, at least partially; if it can’t be, then it should at least be argued that one does not easily mislead themselves through it.
On how the post can be improved, I already mentioned a fair few. If you think that your last paragraph is really only specifically about the toy aliens example and nothing else, I think it should be stated, since it is very unusual for articles to end on a note about the examples rather than what they illustrate. I also think alternatives and risks should be mentioned, if one cannot improve the idea as above to avoid them; a post on a heuristic is useless without showing where and how the heuristic fails, otherwise one falls off cliffs. Reframe the post as being about changing your model post-data, since that’s what actually changes the inference, not the prior (the prior changing is the most benign part of this, ultimately; it just opens the door to an entirely different model, which does not match on the marginal).
Bunglesome to respond to ‘you are tendentiously trying to show this is fractally wrong’ with the longest reply yet, but, you put me between Scylla and Charybdis by asking me for clearly stating a full, coherent position in as neutral a light as possible. Would’ve written a shorter letter, etc.
Addendum: Possibly some of this comes down to a difference in intuition: you imagine your procedure as mimicking a clean, contiguous cut of a 1d curve; there it can at least kind of be imagined what a sensible cut may look like. Insofar as one applies this procedure to questions like your example, I am imagining that the cut is really happening on some infinite-dimensional, abstract space—God only knows what reasonable cuts look like, I have no idea if my particular one is random, tendentious, an improvement or a bias. Certainly I trust no-one else’s.
Thanks. I continue to believe all the propositions in my previous message, and see no direct counterarguments. (The closest you come is “do not have the philosophical clarity of the numbers obtained with Bayesian updating”.) Yet more details on your position are not helpful. Your position is very standard.
While you claim to accept 6 and 10, your answer to 9 seems to suggest that for something labeled Bayesian, you do in fact believe your criterion is universal, that no one should be allowed to use any Bayesian procedure unless it satisfies your particular notion of optimality. Respectfully, I think it’s indisputably true that:
The goal of approximating a single / fixed mental posterior with bounded effort is completely reasonable and coherent.
There are many cases where that goal is best served by choosing your prior in a data-dependent way. If your data happens to show you that the exact amount of prior weight you give to the the [0.90 to 0.91] intervals is crucial, then you would give that interval some extra thought. This will improve performance on the above goal, relative to picking a fixed partition.
In short, I continue to believe my post is correct.
There is no “particular notion of optimality”, nor is it “my criterion”, these are just the properties of Bayesian conditioning. What is provided here is instead a rationalization, which has no truth-tracking properties, Bayesian or otherwise. Approximating a single fixed posterior is coherent (which is to say, it is legitimate to elicit it). Justifying an approximation to a posterior by pretending to perform an update on old evidence after changing the likelihood is not; the resulting numbers are not meaningful (notice Mateusz’s comment elsewhere stating that deconditioning should return the deconditioned probabilities, which is what fails here for the likelihood). “Without seeing the poll results, I never would’ve predicted that I should control for age and class. The answer is much less insane (p<0.05) if I report the likelihood I would have assigned” - π-hacking. When one is being Bayesian, one should strive to be Bayesian.
To put it another way, the post goes: The (Bayes) machine is provided poor inputs (misspecified prior and likelihood) and one receives a poor output. The proposal is to kick the machine (change the marginal likelihood post-hoc); one provides an example where the output is less indecent. All of the screws come loose (coherence, the defining property of doing Bayes, no longer holds). Possibly the internal gears are still fine (it has completely undefined truth-tracking properties). It seems as though it can just as easily take good inputs and return bad outputs, or in fact output just about anything (it can produce any, arbitrary posterior odds for the arbitrary likelihood ratios). No mention is made to nonviolent methods, and the risks are not properly warned against.
We may have reached the limits of this debate, but I’ll state my position one last time: I believe that the abstraction of “approximate the posterior you would find if you had an infinite amount of time to think about it, while using a less-than-infinite amount of time” is completely reasonable, and in fact is a good model of a situation people often face in practice. And I believe that under that abstraction, making your prior after looking at the data (rather than before) is often a good idea, provided you do it carefully. I think you have offered no real counterargument, but instead gesturing at abstract definitions. But those abstract definitions are just ways of smuggling in a different abstraction, under which the problem I am trying to solve doesn’t exist. I understand that a solution built for one abstraction may have poor properties under a different one, but that doesn’t impress me.
I appreciate you taking so much time to discuss this, and I’d like to take something productive from this discussion, but I find it very hard, because you seem determined to show that the post is “fractally wrong”, because you seem unwilling to make any durable concessions, and because you seem to interpret every statement I make in the least-charitable possible way. If you’re truly trying to help me make the post better, that will be impossible unless you adopt a less adversarial posture.
Look, let me explain my perspective, here, then. The discussion took place on 3 entirely different ‘epochs’ (one 7 months ago, then another 6 months ago), and between one of these you directly told me to take a different angle; what you perceive as me being “determined to show that the post is fractally wrong” is, to me, just a result of rereading the full thing and finding different problems.
I will concede that my first approach was misguided; I was more appalled by the combination of title and closing thesis as not properly discussing the cons, and thought the useful thing to post for others to read was, roughly, the costs related to going down this road (incoherence, that is, credence-numbers that are not obtained by Bayes updating and as such do not have the advantageous properties one purportedly uses and defends Bayes numbers through, and of the kind that is hard to repair or argue is ‘approximately’ coherent in some implicitly-Bayesian sense) and the many unmentioned alternatives. I still think a better article would do both of those; I do not think this would make it too technical, especially the former.
Then you told me the thing has ‘specific rules’; you specifically used the word ‘rules’. I went and looked! That’s been the attempt, here, with what you call a ‘different abstraction’. My conclusion so far is that I have found no viable, good formalization of this idea; it either collapses back in the coarse model (insofar as one wants to keep coherence) or it has no properties that I can state as good or bad (if one allows the marginal likelihood to shift away). I think that the non-technicality of the post has led to an idea that is unsalvageable, at least as a procedure with ‘rules’, in the current form. If you wanted me to discuss it as a practical, pragmatic device for thinking about things in a pseudo-Bayesian way, then, I think I get to pout at the fact that you pointed me in the exact opposite direction. (Not only that, but, when I stated my understanding of the thing, you took it upon yourself to agree that it was, at least roughly, correct!)
So, okay. I will put away the snark and the rhetoric and just state my problems with the thing directly. Afterwards I will elaborate on what I believe can be patched, what I believe lacks nuance and warning as presented (of the kind that you have displayed in these comments after), and what I cannot help but conclude is just misguided, or at least requires substantial refinement. For whatever reason I think it is best to express these in pairs. Apologies for length.
However, it is not impossible. Sometimes, with a little difficulty, it can be done. This often requires care; it sometimes requires accepting things that seem unintuitive or even nonsensical. This is because most of the results that point to Bayesianism as the thing to do, the reasons why one performs Bayesian updating and not Dempster-Shafer updating or Jeffrey updating or ad hoc numerical reasoning, usually come in the form of ’if and only if’s; the Dutch book converse, the CCT converse, the accuracy theorems, Cox’s theorem, forecasts-as-martingales, all of these uniquely identify probabilities-as-credences or Bayes updating as, specifically and uniquely, the thing to do; if you end up doing something else, you lose the properties instantly.
If and only ifs are very fragile things; you step away from the formalism in one subtle way and all of these things have suddenly broken into pieces somewhere. What you must ensure, then, is that most of the pieces are still there. Sometimes this can be done formally; I mentioned logical induction; there it is shown that one is only boundedly Dutch-Booked in a relatively benign way. Safebayes is a more pragmatic one; it is very clear to see how that is very approximately Bayesian. Empirical Bayes as approximate hierarchical Bayes.
Sometimes the approximation looks right, and becomes unsalvageable very quickly, in some mathematical sense. This is the case for many objective priors; everyone can nod at the univariate Jeffreys’ rule, but not even Jeffreys defended the multivariate one. You may fondly remember the right-Haar prior, but the left-Haar inspires little fondness. Very classically, nobody puts up a fight to defend flat priors anymore. And so on. I tried putting your procedure in some formalizations and this seemed to be the case; but you (now) reject them, so I will not reiterate.
Other times, the approximate-incoherence is incurred from some (sometimes necessary) pragmatic data-procedure. You change the prior after the sampler runs for too long, you pick the best model out of a bag after looking at a predictive score, you throw out the model because the predictions look kind of bad; nobody (in the 21st century) would dispute that these things happen.
However, what is missing in your post and not missing whenever Bayesian statisticians discuss these things is that when you do this you are treading very dangerous ground. So, generally, the best of them (which is to say, Gelman’s cohort, Stephen Walker, Rubin, etc), if one cannot work out some alternative justification or method to do these things that is still within the formalism (as with Gelman in the model expansion cases, and in Rubin’s case something as trivial as randomization), at the very least you run a bunch of checks to see that you are really not shooting yourself in the foot somewhere.
This is because, when you do abandon coherence in a way that does not give you some kind of ‘bounded’ error, even in some conceptual sense (as in your case, where the marginal likelihood shifts arbitrarily; I will reiterate that the problem is not the prior becoming more detailed, but the marginal likelihood shifting alongside it in a way that does not reduce the model back to it), you really are playing with fire. The numbers obtained by such a method do not have the philosophical clarity of the numbers obtained with Bayesian updating, even if you could imagine that you might have obtained the numbers through some Bayesian update that was not performed.
I understand that you may raise an eyebrow at this, but, the analogy with p-hacking is relevant: you do a little tweak here or there, in ways that might make a lot of common sense, hell, sometimes, it really would be better if you had ran the analysis you wish you had ran after you saw the data; and, really, if the null your tweak led to be rejected really was wrong, then one may feel some solace that no damage was done. But, still, the ‘0.05’ number just no longer means a 0.05 error rate; the number has ceased having a meaning. You may run some garden-of-forking-paths analysis that might elucidate what that number might mean, but it is no longer the p-value it purported itself to be.
So, lacking a formalization through which to analyse approximation quality and observing that the procedure can output ultimately arbitrary results (thus even a conceptual approach to approximation seems hard), since they are no longer clean Bayes-rule probabilities obtained from a prior and posterior update, what are they for?
It is easy to make this kind of procedure look good if one picks out examples where it happens to reach the correct answer (when the null is false and you performed a terrible analysis, p-hacking can look absurdly, genuinely convincing!). I don’t know about you, but, for me, and I think for most statisticians, we like to look at what happens if the person doing it is maybe a little tendentious someplace; maybe not too disciplined here or there. A statistical (or a Bayesian) justification can often serve as a little “certificate of formal correctness” for people who will then no longer squint when they do some things that they really should squint at.
So, how does this procedure fare in this respect? I honestly think that the infamous “Bayesian arguments for God” are, frankly, not too far away from this; you could probably put them in exactly this language, or derive them in this way. “I first took a prior and likelihood, did an update, it did not seem sensible to me; I then found the little tweak to the model that made it make sense with my intuition, so now I have obtained a Bayesian justification for my belief”; I think you know how bad this can be. I am not saying this to imply that the idea fares as poorly as that in general, I imagine in most situations it acts moderately more benignly, but that it is a natural thing to reach for, and easy to imagine that you are doing nothing wrong through, regardless of how right you are (or aren’t).
Of course, you say that things should only be done in the right way, in the post. You do not, I think, specify how to do this; honestly the bisection you performed in the aliens example would look pretty tendentious to an aliens-believer, if you imagine that a few hundred years from now we discover that aliens really are real I think you know that the toy example looks pretty bad. A demonstration of a good method survives the particular example being wrong; one can at least be wrong gracefully, or at least where the flaw lies can be shown to subside in the specifications of the method.
I think that you are right, and indeed you are right that people do some version of this, often. But, one virtue I can name of the Bayesians of the 21st century at least is that they do not do this, typically; if their model was wrong, they are fine eating the bullet, without performing any rigmarole that puts their objectivity into question, they are fine with putting “money on the table”. Here, you salvage a broken inference by a method that simply would not convince a skeptic. But, the problem that you salvage lies in the likelihood, not in your prior; just allow yourself to observe some more data and you’ll be fine, or split the sample first, or cautiously observe through a prior predictive check that your model allows no information to be gained regardless of what happens, or in fact just obtain a better prior (I believe witness unreliability to be the Hume thesis on miracles in the 1700s, so it can certainly be prior to the data in this example to consider it).
Ultimately, when I looked for rules, all I found was the rough sense that the posterior should match one’s intuition. Of course intuition is often right and good, possibly often better than our Bayesian calculations. But, ultimately, I think that performing this kind of calculation to affirm one’s intuitions is at best just an elucidation of at least part of why you think you think the things you think; at worst, and to be frank, very typically in this framing at least, a way to smuggle vague intuition as formally justifiable insight. The best cases of Bayesian inference are when you perform it and convince yourself out of an intuition that was actually wrong; spiritually, at least, this seems to basically prohibit this, there is always an out, always a way to slice up the likelihood to get numbers that you are more concordant with.
I think I will finish on that note. On how the idea can be patched: As Mateusz mentioned earlier, having the partition be data-independent solves the coherency issue entirely (qua Savage). Or, at least, a formal justification that binds one not to change the likelihood too much. Or, in some sense an argument that one will not easily mislead themselves, that one will not be able to convince themselves of false conclusions as easily as true ones, or a sensitivity analysis of some kind that shows, okay, maybe your new flimsy post-hoc model is not too flimsy. Or, as a last resort, if you want a number for an intuition, you can elicit a full posterior from your intuitions via all the other methods in a way that doesn’t make it look like reasoning was performed when it really wasn’t. Mostly, if the idea has formalizable value, it ought to be formalized, at least partially; if it can’t be, then it should at least be argued that one does not easily mislead themselves through it.
On how the post can be improved, I already mentioned a fair few. If you think that your last paragraph is really only specifically about the toy aliens example and nothing else, I think it should be stated, since it is very unusual for articles to end on a note about the examples rather than what they illustrate. I also think alternatives and risks should be mentioned, if one cannot improve the idea as above to avoid them; a post on a heuristic is useless without showing where and how the heuristic fails, otherwise one falls off cliffs. Reframe the post as being about changing your model post-data, since that’s what actually changes the inference, not the prior (the prior changing is the most benign part of this, ultimately; it just opens the door to an entirely different model, which does not match on the marginal).
Bunglesome to respond to ‘you are tendentiously trying to show this is fractally wrong’ with the longest reply yet, but, you put me between Scylla and Charybdis by asking me for clearly stating a full, coherent position in as neutral a light as possible. Would’ve written a shorter letter, etc.
Addendum: Possibly some of this comes down to a difference in intuition: you imagine your procedure as mimicking a clean, contiguous cut of a 1d curve; there it can at least kind of be imagined what a sensible cut may look like. Insofar as one applies this procedure to questions like your example, I am imagining that the cut is really happening on some infinite-dimensional, abstract space—God only knows what reasonable cuts look like, I have no idea if my particular one is random, tendentious, an improvement or a bias. Certainly I trust no-one else’s.
Thanks. I continue to believe all the propositions in my previous message, and see no direct counterarguments. (The closest you come is “do not have the philosophical clarity of the numbers obtained with Bayesian updating”.) Yet more details on your position are not helpful. Your position is very standard.