vae

Karma: 121

vae 6 Aug 2026 21:45 UTC
5 points
0
on: The goalposts are shrouded, not moving
This is a classical Lakatosian conflict. In his terms, Alice is doing something called ‘monster-barring’: Alice has an (implicit) ‘conjecture’, Bob produces (counter)examples which Alice bars by adding conditions on what counts as an example. Your defense is also in Lakatos: the counterexamples which at first are barred are eventually synthesized into the ‘proof-generated concept’ which is the definition Alice did not have when she started (or the conjecture is bled into vacuity, which is informative either way). A more productive form of discourse is lemma-incorporation (identify what the counterexample actually refutes and write it as a condition to be satisfied).
There is also an answer there for how one detects whether or not the goalposts have actually shifted or if there really is a refinement towards a correct domain of validity: one observes whether or not the successive barrings decrease content without adding anything back. Since in her case the conjecture is a negative one (there does not exist an X), the content must come from the evidential standard (have you excluded every achievable way to prove X, such that you can no longer prove anything); is the standard applicable globally or is there a reason why it is applied locally; does the exclusion criteria kill only the specific study that was brought up or does it produce a rule for looking for what studies might count somewhere, etc.
Since the question and part of the answer match very closely, I think it would be sensible for you to read on this perspective (not a hard philosopher to read, nor uncommon; the thing to look at is Proofs & Refutations, which is a dialogue on V—E + F = 2, easy to digest for anyone with the typical backgrounds here)

vae 5 Aug 2026 19:48 UTC
1 point
−2
on: vae’s Shortform
Gelman, Bayesian Workflow:
A sometimes annoying habit of Bayesians is to take non-Bayesian methods and give them Bayesian interpretations. For example, “maximum likelihood is just Bayesian inference with a flat prior,” “fixed effects are just random effects with the group-level variance set to infinity,” “lasso is just regression with an exponential prior,” or “confidence intervals are just posterior intervals assuming flat priors.”
This practice can be useful in giving insight into the scenarios in which a statistical procedure will apply, but it can also be misleading. An example of the latter is the interpretation of lasso (Tibshirani 1996) mentioned just above: lasso is just regression with an exponential prior. Lasso is an approach for obtaining more stable regression estimates that pulls coefficient estimates toward or all the way to zero. The lasso estimate can be viewed as an approximate posterior mode assuming independent exponential priors on the coefficients, and we see some value in making this connection—but this does not make lasso a Bayesian method. The problem is that lasso is intended for use in problems of moderate and high dimensions. In such problems, the mode is not necessarily or even usually a good summary of the posterior distribution, because of the phenomenon of concentration of measure. As such, in high dimensional problems, the formulation, “a penalty is just a log prior density,” stops being useful or accurate. This does not mean that lasso is a bad idea, or that a Bayesian version is necessarily better; some of lasso’s desirable computational and applied properties directly arise from it being a mode rather than a full posterior.
More generally, any statistical method can be evaluated on its own, in non-Bayesian terms, by studying its statistical properties under various assumptions. Yes, least squares regression can be derived as the maximum likelihood estimate assuming independent normally-distributed errors with equal variance. But you can use least squares without making those assumptions, and you can evaluate the robustness of least squares under various alternative assumptions. That said, the theoretical derivation of least squares can provide some insight and practical guidance. For example, if you have reason to believe that the model errors have some particular covariance structure, you could consider incorporating that into an improved estimate. Conversely, if least squares estimation does not seem to be performing well, you could go back and check its implicit assumptions.
Still, careful Bayesian interpretations offer potential benefits. In applied settings where estimates are problematically noisy, leading to replication failures (Button et al. 2013), we can get some perspective by recognizing that simple means, differences, and least squares regression correspond to Bayesian estimates assuming flat priors. When the posterior distribution includes implausible parameter values, it can make sense to add further information. This can take the shape of an informative prior. Another way to say this: When the estimate is too large to believe, this implies that inference can be improved by incorporating prior information.
Yudkowsky, Searching for Bayes-Structure:
In fact, any part of a cognitive process that contributes usefully to truth-finding must have at least a little Bayesian structure—must harmonize with Bayes, at some point or another—must partially conform with the Bayesian flow, however noisily—despite however many disguising bells and whistles—even if this Bayesian structure is only apparent in the context of surrounding processes. Or it couldn’t even help.
(...)
But perhaps it is not quite as exciting to see something that doesn’t look Bayesian on the surface, revealed as Bayes wearing a clever disguise, if: (a) you don’t unravel the mystery yourself, but read about someone else doing it (Newton had more fun than most students taking calculus), and (b) you don’t realize that searching for the hidden Bayes-structure is this huge, difficult, omnipresent quest, like searching for the Holy Grail.
It’s a different quest for each facet of cognition, but the Grail always turns out to be the same. It has to be the right Grail, though—and the entire Grail, without any parts missing—and so each time you have to go on the quest looking for a full answer whatever form it may take, rather than trying to artificially construct vaguely hand-waving Grailish arguments. Then you always find the same Holy Grail at the end.
(...)
This left me in a bit of a pickle when it came to trying to explain in advance where I was going. I know from experience that if I say, “Bayes is the secret of the universe,” some people may say “Yes! Bayes is the secret of the universe!”; and others will snort and say, “How narrow-minded you are; look at all these other ad-hoc but amazingly useful methods, like regularized linear regression, that I have in my toolbox.”
(...)
To see through the surface adhockery of a cognitive process, to the Bayesian structure underneath—to perceive the probability flows, and know how, not just know that, this cognition too is Bayesian—as it always is—as it always must be—to be able to sense the Force underlying all cognition—this, is the Bayes-Sight.
So, to Gelman, the “Bayes-structure” is only sometimes helpful, but sometimes misleading as to why the thing works and in what ways it does, but to Yudkowsky, it is the Holy Grail and the only reason why these methods work is because of the Bayesian explanation underneath (Neatly, they both mention the same example as demonstrating the opposite things).
In my experience as a statistician, I think Gelman has the right of it (it is certainly the case with LASSO; I think Castillo is essentially correct that the LASSO posterior is a “useless object” that does away with why people use it), but I wonder about some more orthodox Bayesian perspectives on this contrast?

vae 2 Aug 2026 0:25 UTC
1 point
0
in reply to: dynomight’s comment on: Good if make prior after data instead of before
Look, let me explain my perspective, here, then. The discussion took place on 3 entirely different ‘epochs’ (one 7 months ago, then another 6 months ago), and between one of these you directly told me to take a different angle; what you perceive as me being “determined to show that the post is fractally wrong” is, to me, just a result of rereading the full thing and finding different problems.
I will concede that my first approach was misguided; I was more appalled by the combination of title and closing thesis as not properly discussing the cons, and thought the useful thing to post for others to read was, roughly, the costs related to going down this road (incoherence, that is, credence-numbers that are not obtained by Bayes updating and as such do not have the advantageous properties one purportedly uses and defends Bayes numbers through, and of the kind that is hard to repair or argue is ‘approximately’ coherent in some implicitly-Bayesian sense) and the many unmentioned alternatives. I still think a better article would do both of those; I do not think this would make it too technical, especially the former.
Then you told me the thing has ‘specific rules’; you specifically used the word ‘rules’. I went and looked! That’s been the attempt, here, with what you call a ‘different abstraction’. My conclusion so far is that I have found no viable, good formalization of this idea; it either collapses back in the coarse model (insofar as one wants to keep coherence) or it has no properties that I can state as good or bad (if one allows the marginal likelihood to shift away). I think that the non-technicality of the post has led to an idea that is unsalvageable, at least as a procedure with ‘rules’, in the current form. If you wanted me to discuss it as a practical, pragmatic device for thinking about things in a pseudo-Bayesian way, then, I think I get to pout at the fact that you pointed me in the exact opposite direction. (Not only that, but, when I stated my understanding of the thing, you took it upon yourself to agree that it was, at least roughly, correct!)
So, okay. I will put away the snark and the rhetoric and just state my problems with the thing directly. Afterwards I will elaborate on what I believe can be patched, what I believe lacks nuance and warning as presented (of the kind that you have displayed in these comments after), and what I cannot help but conclude is just misguided, or at least requires substantial refinement. For whatever reason I think it is best to express these in pairs. Apologies for length.
Coherence is hard. Doing a full analysis of any problem, even simple, informal checks, while staying fully within the Bayesian formalism is sometimes basically impossible.
However, it is not impossible. Sometimes, with a little difficulty, it can be done. This often requires care; it sometimes requires accepting things that seem unintuitive or even nonsensical. This is because most of the results that point to Bayesianism as the thing to do, the reasons why one performs Bayesian updating and not Dempster-Shafer updating or Jeffrey updating or ad hoc numerical reasoning, usually come in the form of ’if and only if’s; the Dutch book converse, the CCT converse, the accuracy theorems, Cox’s theorem, forecasts-as-martingales, all of these uniquely identify probabilities-as-credences or Bayes updating as, specifically and uniquely, the thing to do; if you end up doing something else, you lose the properties instantly.
This difficulty means that, often, approximating coherence is essentially required.
If and only ifs are very fragile things; you step away from the formalism in one subtle way and all of these things have suddenly broken into pieces somewhere. What you must ensure, then, is that most of the pieces are still there. Sometimes this can be done formally; I mentioned logical induction; there it is shown that one is only boundedly Dutch-Booked in a relatively benign way. Safebayes is a more pragmatic one; it is very clear to see how that is very approximately Bayesian. Empirical Bayes as approximate hierarchical Bayes.
Sometimes the approximation looks right, and becomes unsalvageable very quickly, in some mathematical sense. This is the case for many objective priors; everyone can nod at the univariate Jeffreys’ rule, but not even Jeffreys defended the multivariate one. You may fondly remember the right-Haar prior, but the left-Haar inspires little fondness. Very classically, nobody puts up a fight to defend flat priors anymore. And so on. I tried putting your procedure in some formalizations and this seemed to be the case; but you (now) reject them, so I will not reiterate.
Sometimes, there is no formal procedure to analyse the proximity of.
Other times, the approximate-incoherence is incurred from some (sometimes necessary) pragmatic data-procedure. You change the prior after the sampler runs for too long, you pick the best model out of a bag after looking at a predictive score, you throw out the model because the predictions look kind of bad; nobody (in the 21st century) would dispute that these things happen.
However, what is missing in your post and not missing whenever Bayesian statisticians discuss these things is that when you do this you are treading very dangerous ground. So, generally, the best of them (which is to say, Gelman’s cohort, Stephen Walker, Rubin, etc), if one cannot work out some alternative justification or method to do these things that is still within the formalism (as with Gelman in the model expansion cases, and in Rubin’s case something as trivial as randomization), at the very least you run a bunch of checks to see that you are really not shooting yourself in the foot somewhere.
This is because, when you do abandon coherence in a way that does not give you some kind of ‘bounded’ error, even in some conceptual sense (as in your case, where the marginal likelihood shifts arbitrarily; I will reiterate that the problem is not the prior becoming more detailed, but the marginal likelihood shifting alongside it in a way that does not reduce the model back to it), you really are playing with fire. The numbers obtained by such a method do not have the philosophical clarity of the numbers obtained with Bayesian updating, even if you could imagine that you might have obtained the numbers through some Bayesian update that was not performed.
I understand that you may raise an eyebrow at this, but, the analogy with p-hacking is relevant: you do a little tweak here or there, in ways that might make a lot of common sense, hell, sometimes, it really would be better if you had ran the analysis you wish you had ran after you saw the data; and, really, if the null your tweak led to be rejected really was wrong, then one may feel some solace that no damage was done. But, still, the ‘0.05’ number just no longer means a 0.05 error rate; the number has ceased having a meaning. You may run some garden-of-forking-paths analysis that might elucidate what that number might mean, but it is no longer the p-value it purported itself to be.
Besides, frankly, sometimes you have to put common sense and other statistical properties over following Bayesian edicts directly.
So, lacking a formalization through which to analyse approximation quality and observing that the procedure can output ultimately arbitrary results (thus even a conceptual approach to approximation seems hard), since they are no longer clean Bayes-rule probabilities obtained from a prior and posterior update, what are they for?
It is easy to make this kind of procedure look good if one picks out examples where it happens to reach the correct answer (when the null is false and you performed a terrible analysis, p-hacking can look absurdly, genuinely convincing!). I don’t know about you, but, for me, and I think for most statisticians, we like to look at what happens if the person doing it is maybe a little tendentious someplace; maybe not too disciplined here or there. A statistical (or a Bayesian) justification can often serve as a little “certificate of formal correctness” for people who will then no longer squint when they do some things that they really should squint at.
So, how does this procedure fare in this respect? I honestly think that the infamous “Bayesian arguments for God” are, frankly, not too far away from this; you could probably put them in exactly this language, or derive them in this way. “I first took a prior and likelihood, did an update, it did not seem sensible to me; I then found the little tweak to the model that made it make sense with my intuition, so now I have obtained a Bayesian justification for my belief”; I think you know how bad this can be. I am not saying this to imply that the idea fares as poorly as that in general, I imagine in most situations it acts moderately more benignly, but that it is a natural thing to reach for, and easy to imagine that you are doing nothing wrong through, regardless of how right you are (or aren’t).
Of course, you say that things should only be done in the right way, in the post. You do not, I think, specify how to do this; honestly the bisection you performed in the aliens example would look pretty tendentious to an aliens-believer, if you imagine that a few hundred years from now we discover that aliens really are real I think you know that the toy example looks pretty bad. A demonstration of a good method survives the particular example being wrong; one can at least be wrong gracefully, or at least where the flaw lies can be shown to subside in the specifications of the method.
In any event, this is a good model for something that is encountered in practice.
I think that you are right, and indeed you are right that people do some version of this, often. But, one virtue I can name of the Bayesians of the 21st century at least is that they do not do this, typically; if their model was wrong, they are fine eating the bullet, without performing any rigmarole that puts their objectivity into question, they are fine with putting “money on the table”. Here, you salvage a broken inference by a method that simply would not convince a skeptic. But, the problem that you salvage lies in the likelihood, not in your prior; just allow yourself to observe some more data and you’ll be fine, or split the sample first, or cautiously observe through a prior predictive check that your model allows no information to be gained regardless of what happens, or in fact just obtain a better prior (I believe witness unreliability to be the Hume thesis on miracles in the 1700s, so it can certainly be prior to the data in this example to consider it).
Ultimately, when I looked for rules, all I found was the rough sense that the posterior should match one’s intuition. Of course intuition is often right and good, possibly often better than our Bayesian calculations. But, ultimately, I think that performing this kind of calculation to affirm one’s intuitions is at best just an elucidation of at least part of why you think you think the things you think; at worst, and to be frank, very typically in this framing at least, a way to smuggle vague intuition as formally justifiable insight. The best cases of Bayesian inference are when you perform it and convince yourself out of an intuition that was actually wrong; spiritually, at least, this seems to basically prohibit this, there is always an out, always a way to slice up the likelihood to get numbers that you are more concordant with.
I think I will finish on that note. On how the idea can be patched: As Mateusz mentioned earlier, having the partition be data-independent solves the coherency issue entirely (qua Savage). Or, at least, a formal justification that binds one not to change the likelihood too much. Or, in some sense an argument that one will not easily mislead themselves, that one will not be able to convince themselves of false conclusions as easily as true ones, or a sensitivity analysis of some kind that shows, okay, maybe your new flimsy post-hoc model is not too flimsy. Or, as a last resort, if you want a number for an intuition, you can elicit a full posterior from your intuitions via all the other methods in a way that doesn’t make it look like reasoning was performed when it really wasn’t. Mostly, if the idea has formalizable value, it ought to be formalized, at least partially; if it can’t be, then it should at least be argued that one does not easily mislead themselves through it.
On how the post can be improved, I already mentioned a fair few. If you think that your last paragraph is really only specifically about the toy aliens example and nothing else, I think it should be stated, since it is very unusual for articles to end on a note about the examples rather than what they illustrate. I also think alternatives and risks should be mentioned, if one cannot improve the idea as above to avoid them; a post on a heuristic is useless without showing where and how the heuristic fails, otherwise one falls off cliffs. Reframe the post as being about changing your model post-data, since that’s what actually changes the inference, not the prior (the prior changing is the most benign part of this, ultimately; it just opens the door to an entirely different model, which does not match on the marginal).
Bunglesome to respond to ‘you are tendentiously trying to show this is fractally wrong’ with the longest reply yet, but, you put me between Scylla and Charybdis by asking me for clearly stating a full, coherent position in as neutral a light as possible. Would’ve written a shorter letter, etc.
Addendum: Possibly some of this comes down to a difference in intuition: you imagine your procedure as mimicking a clean, contiguous cut of a 1d curve; there it can at least kind of be imagined what a sensible cut may look like. Insofar as one applies this procedure to questions like your example, I am imagining that the cut is really happening on some infinite-dimensional, abstract space—God only knows what reasonable cuts look like, I have no idea if my particular one is random, tendentious, an improvement or a bias. Certainly I trust no-one else’s.

vae 1 Aug 2026 13:29 UTC
1 point
0
in reply to: dynomight’s comment on: Good if make prior after data instead of before
There is no “particular notion of optimality”, nor is it “my criterion”, these are just the properties of Bayesian conditioning. What is provided here is instead a rationalization, which has no truth-tracking properties, Bayesian or otherwise. Approximating a single fixed posterior is coherent (which is to say, it is legitimate to elicit it). Justifying an approximation to a posterior by pretending to perform an update on old evidence after changing the likelihood is not; the resulting numbers are not meaningful (notice Mateusz’s comment elsewhere stating that deconditioning should return the deconditioned probabilities, which is what fails here for the likelihood). “Without seeing the poll results, I never would’ve predicted that I should control for age and class. The answer is much less insane (p<0.05) if I report the likelihood I would have assigned” - π-hacking. When one is being Bayesian, one should strive to be Bayesian.
To put it another way, the post goes: The (Bayes) machine is provided poor inputs (misspecified prior and likelihood) and one receives a poor output. The proposal is to kick the machine (change the marginal likelihood post-hoc); one provides an example where the output is less indecent. All of the screws come loose (coherence, the defining property of doing Bayes, no longer holds). Possibly the internal gears are still fine (it has completely undefined truth-tracking properties). It seems as though it can just as easily take good inputs and return bad outputs, or in fact output just about anything (it can produce any, arbitrary posterior odds for the arbitrary likelihood ratios). No mention is made to nonviolent methods, and the risks are not properly warned against.

vae 1 Aug 2026 3:47 UTC
1 point
0
in reply to: dynomight’s comment on: Good if make prior after data instead of before
1. Agreed, minor comment. Iterating the partitioning or choosing a finer partition are identical. It is hard to interpret “such that the probability of the data is constant-ish for each term” as anything but a rule for how to partition. If you disavow it, fine. Frankly, that was more of a steelman, since at least with that procedure there’s some definition on what a ‘reasonable’ partition looks like—without it, it’s all arbitrary (a kind of reversed reverse bayes).
2. True. Also irrelevant. You have added a new parameter such that the models are no longer nested (observe the likelihood ratios). Thus, it is not model expansion, it is just a different model. In fact, if you were to do this coherently, the parameter would have a prior, instead of having been fixed a priori by introspection. ‘a+x’ is a model expansion of the model ‘x’; ‘1+x’ is just a different model.
3. True.
4. True; if the data-dependent partition is obtained identically regardless of what the data happens to be, the procedure is coherent and the resulting numbers are probabilities (and in fact one imagines that they are probably better, insofar as the prior information from the resulting introspection was accurate). Otherwise, unclear; would have to study the properties of the other numbers.
5. Not particularly. A 1:1 likelihood ratio is converted to 1:9 by pure post-hoc introspection. If Model 2 refines over Model 1, then Model 2 returns the exact same thing on the Model 1 level of analysis; marginalizing should return the Model 1 numbers; they are not. As stated prior, the numbers obtained are not Bayesian probabilities per se, and have unknown epistemic properties. If I am to judge “does it look more reasonable at the end”, yes. If I am to judge “is it a reasonable thing to do”, no. The thought-exercise has produced numbers and the numbers do not mean what they say they mean. It is unclear to me what they do, or are for.
6. Yes.
7. Indeed.
8. Different theorem. That’s admissibility/the complete class theorem. I am stating non-domination in the Dutch Book sense, which is a different thing. Your prior and likelihood are fixed and I get a sure gain and you a sure loss at no risk without extra information. No other priors or likelihoods appear.
9. I don’t know what to do about this one. The point of my post was, essentially, “If you are using Bayesian probabilities, you are using them for these reasons; these reasons do not apply” (dutch-bookability, etc.) “If you are just using the probability numbers as helpful pistimetric guidelines, then they are pragmatically very dubious in pretty much every formal sense I can imagine”. If I understand your meaning, you seem to be going in direction 2? At which point one wonders what properties these numbers/this approximation has which makes it worth abandoning whatever credit one gives to the properties of direction 1. The coarse prior’s coarse posterior retains the virtues of being Bayesian. What virtues do these numbers have?
10. Of course.
11. It ‘may well do better’, it is possible that it can. The question is how one evaluates if it might, or if there is any guarantee that it could, or if there is any sense that it may. There are many procedures that amount to data-driven priors (empirical Bayes, some design-based inference, some ‘default’ priors in software); what is done when one proposes such a procedure is, to please the Bayesians, verify that the thing in some decent sense approximates something actually Bayesian, and to please the Frequentists, show that it has good empirical properties and can verifiably improve predictions. It is unclear to me that this is the case even in the toy example; the inference obtained is not convincing. When presented with a probability obtained from this method I see no reason to trust it more than an elicited posterior.

vae 1 Aug 2026 0:18 UTC
4 points
0
in reply to: dynomight’s comment on: Good if make prior after data instead of before
A prompt reply! Wonderful. Thank you. Apologies for the bold, it would indeed have been more proper in any academic context to append “(emphasis mine)”. A crass error.
I wouldn’t endorse the last sentence, but I don’t think that’s a crux for your criticism.
An agreeable amendment to the post, since that line was copied from it (including the ‘-ish’):
The better solution is what I did in the second model: To create a finer categorization of the space of things that might be true, such that the probability of the data is constant-ish for each term. (emphasis mine)
Either you apply the procedure with this stopping rule for when to stop discretizing, or it boils down to “discretize until the subjective prior and likelihood lead to agreement with what you imagine the posterior intuitively ought to look like”, which seems empty; one may as well state the subjective posterior without rationalizing it, all the prior is doing is adding discretization error (you indeed do state that just eliciting the posterior directly is not cogent, but if the procedure only runs on intuition, then I don’t know that this is any different).
I believe your Dutch Book argument is equivalent to the statement that the above procedure is not decision-theoretic optimal.
No. Suboptimal is “you could have done better”, of course you could. Dominated is “you lose something for free every state.” It may be true that the theoretical full prior is preferable to the coarse prior; the data-dependent in-between is incoherent, and an easy rectification is available which does not lead to such problems. This is not a claim about optimality.
The only decision-theoretic optimal procedure is to state your full prior (without any approximation) and make decisions based on that.
No. Proceeding with the coarse prior is coherent and undominated, as are averaging, stacking, nonparametrics. Also coherent: data splitting. Etc. It is sometimes true that it is too expensive or inconvenient to be fully, coherently Bayesian; typically when this is argued some binding argument is provided that things will still ‘work out okay’ somehow (say, logical induction, or most proper approximations generally), showing that one is being appropriately approximately Bayesian. I do not believe there is such a bound here, it seems to me as though this produces arbitrary answers in general (through careful picking of level-sets, say). It is sometimes hard to notice that by loosening one of the Bayesian screws one has been led to something that produces arbitrary, uncheckable answers, but this seems like one of those cases.
if you follow my procedure but only make the discretization more fine that is literally an example of model expansion
No. Model 1 gives a different likelihood ratio for the data when compared to Model 2 (9:1 vs. 1:1). Model expansion qua Gelman means the model agrees with the simpler model as a special case; that’s what makes it different from just changing the model. If you marginalize the finer model here, you recover a different model. Indeed, it is possible to do model expansion coherently, while this is not coherent, thus it is not that.
The comparison is fruitful in what the post does not recommend: when Gelman recommends expansion as a sensible way to make inferences from data, there are a trillion little sense checks he suggests so that one doesn’t fool themselves into modelling towards a foregone conclusion (prior and posterior predictive checks, prior sensitivity analysis, there’s a whole diagram), and indeed he is constantly careful in making sure not to give yourself too many ‘researcher degrees of freedom’, to have a coherent workflow where each step is checkable and has sense. Of course, you don’t need to explain the whole thing, but sufficient warning that this is dangerous territory seems at least appropriate if not just correct.
This is closer to ‘model criticism’, which he makes no qualms about separating from the properly, fully Bayesian part of an analysis. Though it still isn’t that, because there’s a methodical aspect to it which (something like) “the posterior looks implausible” does not capture.
There is a place in the world for posts targeting a general audience, so I can’t accept this criticism.
Fair enough. Still, “this is one particular way to address this well-known problem; others are A, B or C” seems non-technical and cogent. When presenting one method among many, it seems best to present weaknesses as well as strengths. If you observe the other comments, it seems as though some other commenters seem to believe that this is perfectly Bayesian and in little to no tension with the formalism. I believe this to be a reasonable reading of what was written, and that something else should have been written instead.
Not only am I not claiming that this is the only way to deal with the general problem, I’m not even claiming that it’s the only way to deal with this specific example. I am stating that the the only way to choose a better discretization for this example is to look at the data.
I don’t know. Is that true? If you imagine that the prior is fixed, then that the likelihood is fixed, then the posterior it produces is the posterior it produces.
If the probabilities are for betting, these are plainly worse than the coarse ones. If they are meant to be more accurate, in the Bayesian sense, they are not—see the accuracy theorems. If they are for convincing, I don’t know that they convince anyone—it seems to me like they might be able to help you convince yourself that your answer is Bayesian in some sense that it is not, but I don’t know that you can in any sense argue persuasively against someone who simply dismisses the post-data model out of hand as post-hoc and ad-hoc. Maybe you can state the ‘rules’ more plainly? You seemed to agree with my description and that one allows any conclusion (really, you disagreed with the only part that could have disallowed it).
If we are allowed to complain about the man’s prior, a simple predictive check would have sufficed (his opinion does not change after seeing the data, so even before seeing anything the model may be coherently rejected). To be frank I am not convinced that it is impossible to consider “what if some witnesses are unreliable” without necessarily seeing the data. Then, whichever refinement one produces is perfectly coherent and possibly much better, insofar as it incorporates whichever other relevant prior information may work out.
All I’ve claimed is that there are often perfectly good reason to change your prior after looking at the data. And as far as I can tell, you accept that this is true. So what am I missing? How do I square this with the very broad and very sharp criticisms you made in your first comment?
There are “often good reasons for changing the prior after seeing the data”, yes. The problem is that your suggestion is that it should be fine to do this in an ad hoc way until the posterior either reaches that arbitrary stopping point (with the near-constant likelihood) or until the Bayes rule is made to agree with intuition (which seems to be the stopping rule in the example?). Just because I concede “it is sometimes good to surgically open a man up” does not mean I agree to clefting the poor Bayesian in twain.
Other procedures preserve coherence (as above) perfectly well (or require a psychic diachronic bookie, unlike this one), a third type have either efficiency or robustness guarantees that make the model modification at least somewhat convincing. With your method, I expect an aliens-believer to roll his eyes and produce his own bisection that accords with his principles.
My initial comment was fairly vague, yes. As I said, it is a general fact that such a data-dependent prior produces incoherency, so I thought that the existence statement would be enough to convince one that maybe this has a pretty hefty (and avoidable) cost. I am glad to see that producing the counterexample instead of saying it exists seems to have helped communicate the sense in which this is bad. And, fair enough, I was incautious in saying that you abide by any generic double-counting.
On the quote being out of context: a more general claim appears before any aliens are in mention (for real problems, I’ve come to believe that refusing to change your prior after you see the data often leads to tragedy). But at the end, somehow, the thesis stops referring to “real problems” and only to “aliens”. The sentence is the post’s last line, following “So”. The idea is that you meant to say that this refers not only to specifically the toy “aliens” example, but also specifically in picking a discretization? If that is truly the intention, I humbly ask that you consider appending your post with “at least for picking a discretization in this aliens example”. I don’t think I am alone in supposing you meant to say more than this.

vae 31 Jul 2026 7:01 UTC
1 point
0
in reply to: dynomight’s comment on: Good if make prior after data instead of before
Fair enough. When I stated “a prior that depends on the observed data is not coherent”, I relied on it being true, when I should’ve been more specific. To my understanding, the idea you are proposing goes, roughly, like:
You have some idealized prior in your head, which you cannot access directly but can feel out and compare against a working prior. You observe the data. The data tells you how finely to partition the working prior. You then fill in the cells with what you judge you would have said before observing anything. Do this until is constant-ish inside each cell.
Call the pre-observation model Model 1, the data-dependent refinement Model 2. Specifically, say the refinement occurs only after one has observed the realization , if one sleeps peacefully (What I take “Without seeing the actual data in our world, I would never have predicted that we would have so many pilots that report seeing tic-tacs” to mean).
The Dutch Book is this. I ask you your , before actually having observed the data. Insofar as that last quoted sentence means something, I assume it means that, before observation, I get your Model 1 odds (say, ); if you say you’d refine in advance, then you can refine in advance, and in fact the refinement may as well not exist at all (and it’s just a calculation error; technically those still fall prey to the diachronic Dutch Book, but nobody has ever been diachronically Dutch Booked); besides, it seems a little surprising to say one would find this refinement from just being asked what the sky looks like.
Say . You do not refine, so, as before, your posterior credence in is . If , as you say, it gets refined down to . So, I sell you quoted from your prior at ; if , you lose when I buy it back, otherwise nothing, whatever is (unless , to be fair). Sure loss, irrationality, etc.
I imagine one responds with the notion that the idea is only acceptably non-Bayesian, because we are all mortal finite beings who can only ever do with approximating Bayesian rationality, for which this is a simple, sensible and suitable procedure. My earlier posts were arguing that, no, while it is known that this can be done, it is disrecommended (Technically, it would be fine if you would refine in the same way regardless of what you observe, but this seems to run opposite of the post). Obviously you are not charged with ‘solving those problems’ in full generality. But, when you say
So the only practical way to get good results is to first look at the data to figure out what categories are important, and then to ask yourself how likely you would have said those categories were, if you hadn’t yet seen any of the evidence.
Then I think you are wrong; you can do any of those other things I mentioned, which apply to any such “I observed the data and think my model was wrong”-type problems while being more sensible either in Bayesian terms (e. g., not incurring in incoherence) or in pragmatic terms (not double-counting and easily tricking yourself into overconfidence; I don’t know what makes this any procedure different from asking for an intuitive posterior, one would have to prove that it is not possible to bisect tendentiously until some arbitrary ‘constant-ish’ probability. In fact, it seems like it starts at a very flat probability, so I am confused about when one stops).

vae 25 Jul 2026 15:54 UTC
9 points
0
on: vae’s Shortform
I’ve been writing a comprehensive Lakatos-style dialogue on Yudkowsky’s Bayesianism. There have been rather a lot of criticisms of Yudkowsky’s Bayesianism over the years, both directly related to him and to the general subjective Bayes as optimal reasoner programme; I get the sense that, for the people on this website, the kind of overarching, technical takedown on all the specifics is not quite useful, but instead something focused on the parts of his Bayesianism that have actually been passed down.
I would like to ask you to tell me what those are. I would summarize Yud’s take as, roughly,
0. Think laws, not tools. Bayes is not one instrument among others, to be picked up when convenient and put down when awkward; it is the law that governs whatever you pick up. Everything below follows from taking that seriously.
1. The ideal reasoner is Bayesian.
1.1 What is meant by ‘ideal’ is that no other procedure can ‘do better’; a la Carnot’s engine. The comparison is his: “Bayes’ rule is to reasoning as the Carnot cycle is to engines: Nobody can be a perfect Bayesian, but Bayesian reasoning is still the theoretical ideal.”
1.2 What is meant by ‘do better’ is that “you can’t get a higher expected score by making any different update”, with the expectation being taken over the prior, and the score being log-loss.
1.21 Equivalently: “You can’t extract any more evidence from an observation than is given by its likelihood ratio.”
1.22 This is not a “normative fact”, per se; the idea “begins life as a descriptive assertion, not as a normative assertion”. But: “If you want to assign higher probability to the correct hypothesis, it’s a short step from that preference to regarding Bayesian updates as a normative ideal”. (The reading this invites, and which is what actually gets passed down: if you reason by something other than a Bayes update, you are assigning lower probability to the correct hypothesis. Note that this does not follow from 1.2 - an expected-score claim under your own prior is not an accuracy claim about the truth—but it is what the passage is for.)
1.23 He is aware of the distinction between proper and strictly proper scoring rules, and treats the logarithmic score as the working case.
1.3 We may also mean that breaking with the ideal makes you vulnerable to a sure loss by Dutch booking: “anything that is not Bayesian must fail one of the coherency tests.”
1.4 A third meaning is by the complete class theorem’s converse; an admissible decision-rule is Bayes or generalized-Bayes, so ideal reasoners are Bayesian.
1.5 A fourth is uniqueness: “Given what you knew, and what you saw, the maximally accurate state of belief for you to be in is completely pinned down.” Hard to find in practice, known in principle. There is only ever one answer.
1.6 And so, definitionally: “Bayesian reasoner” is “the technically precise codeword that we use to mean rational mind.”
1.7 This ideal is fully general in the sense that it, at the very least, must bound why Einstein discovered relativity (so, at the very least, general in the sense of function-spaces, logical uncertainty, old-evidence). It explains all reasoning.
1.71 Any theoretical gap in the current Bayesian formalism is to be explained by further Bayesianism, not by a retreat from Bayesianism (law-likeness).
2. It is more precisely a subjectivist Bayesianism qua Jaynes.
2.1 Uncertainty is ‘in the map, not in the territory’.
2.11 Priors ought to be coherent probability distributions, not weighing functions. A prior that is data-dependent or which does not normalize to a probability distribution is non-Cox and is thus Dutch-bookable.
2.12 The prior is a summary of ‘all known prior information’, not properties of the posterior.
2.2 A single case has no probability at all except somebody’s credence in it. Coverage of posterior intervals is irrelevant; frequency concerns with repeated sampling are nonsense.
2.21 The frequentist’s objective/subjective distinction is confused rather than merely austere: the ‘objective’ long-run statement is itself obtained as a limit over the probabilities of finite sequences, every intermediate one of which he calls meaningless.
2.22 The positive replacement: a single-case probability is cashed out by a logloss correspondence against what is subsequently observed.
3. Distance from the Bayesian is distance from the ideal.
3.1 ‘Whatever approximation you use, both its failures and its successes are explainable in Bayesian terms’.
3.11 Frequentist/statistical tools work to the extent that they track the ideal Bayesian calculation, and fail to the extent that they depart.
3.111 If a Frequentist tool looks like it outperforms the Bayesian law, extra, subjective information has been smuggled in. (Licensed by 1.5: since the maximally accurate state is unique, an apparent improvement on it must be an illusion or a theft.)
3.12 This does not imply Bayesian ‘tools’ are more useful directly; but, the Bayesian way ‘suggests the path’ to the ideal—knowledge of the law “helps you get as close to the ideal efficiency as you can.”
3.2 As such, it has sense to ‘look for Bayes-structure’.
3.21 Bayes-structure is nontrivial, generically; the Bayesian version of the thing that works is ‘why’ it works.
3.22 Thus, if something has trivial Bayes-structure, in that the prior corresponds to no prior or the approximation taken is one that makes no Bayesian sense to take, it will malfunction in proportion to the departure.
4. We may justify ‘rational’ behaviour for human behaviour and scientific inference on the basis that it is Bayesian.
4.1 Stopping-rules must be made irrelevant—two researchers with the same data must give the same conclusions, regardless of the process to get the data.
4.2 Randomness has no power—“there is no beauty in entropy, nor strength from noise”. An algorithm is improved by randomization only where some step was already doing worse than chance.
4.21 Wherever the environment cares only about your actions and not your algorithm, anything improvable by randomization is further improvable by derandomization.
4.22 One exception is granted: superintelligent or cryptographic adversaries, where entropy acts as an antidote to intelligence. Otherwise, without adversariality, no.
4.3 All the knowledge contained in the data is contained in the likelihood of the data; procedures that depend on anything else are to be disrecommended.
4.31 Frequentist error rates are irrelevant; confidence intervals are meaningless, since repeated sampling is incoherent. The reference class is fixed by the experimenter’s intentions and is therefore not a fact about the world.
4.32 Inference must not change on the basis of likelihood-preserving experiment design; all conclusions depend on the probabilistic element alone, not on extraneous elements.
4.4 An improvement would be to report full likelihood functions, rather than p-values, tests, estimates or intervals.
4.5 The sinister misdeeds p-values are meant to prevent “are just flatly mathematically impossible in the first place under this system.”
4.51 The ground given is conservation of expected evidence: you would have to know in advance which direction you would update, which is impossible. “If you start out thinking it’s 70% probable that some coin is fair, nothing you can possibly plan to do by gathering more data… can result in you expecting for that analysis to make you believe on average that the coin is not 70% probably fair.”
4.6 Frequentist/statistical theory is thus only relevant because of computational-approximate concerns; pragmatic, not epistemic or quite scientific ones. “For so long as we do not have infinite computing power, there may yet be a place in science for non-Bayesian statistics.”
5. Apparent counterexamples are faults in the counterexample, rather than faults in the methodology. (Pre-emptive monster-barring/adjustment.)
5.0 The general policy: shown a purported paradox, “look for the division by zero; or the infinity that is assumed rather than being constructed as the limit of a finite operation”—something illegal. “Trust Bayes. Bayes has earned it.”
5.1 Infinite sets lead to trouble, so only priors on finite sets approximating the infinite set are coherent.
5.11 The pathologies of infinite-set Bayesianism are illusions, as the finite-set Bayesian is fine.
5.2 Simultaneously, the Bayesian must place no zero- or one- probabilities anywhere.
5.21 In particular, insofar as Einstein was bounded by the ideal Bayesian, the ideal Bayesian must place a nonzero probability on the equations of GR.
5.22 Technically speaking, zero and one are not probabilities, per se. The fact that the probability formalism has them is a kind of gap; it is sensible to imagine a theory that is rid of them that is better for it.
5.23 Consequently [this is an entailment I am drawing, not a claim he states]: the Bayesian contains the correct model of the world, and concerns of ‘model misspecification’ are pragmatic, of no concern to the ideal epistemologist.
5.3 In particular, Solomonoff solves all such troubles.
5.31 Indeed, the Solomonoff prior gives a unified proof of Ockham’s principle in some general way.
5.32 Despite UTM-relativity, we know roughly what ‘good’ UTMs look like, and anyhow the constant washes out with enough data, which is fine for our ideal. Abusing the freedom requires constructing “a downright embarrassing Universal Turing Machine”—though he concedes that fully objective priors are not to be had by deduction, not “without principles that are unknown to me and beyond the scope of Solomonoff induction.”
5.33 It is “something that bootstraps to good epistemology rather than being all of good epistemology by itself.”
Which of these do you actually hold? Any places where your reading of him differs from mine? There are ~20 years of stuff in here, after all.

vae 5 Feb 2026 4:22 UTC
1 point
0
in reply to: dynomight’s comment on: Good if make prior after data instead of before
Thanks for replying. Given that it’s been a month, sadly, I don’t fully remember all the details of why I wrote what I wrote in my initial comment, but I’ll try to roughly rewrite my objections in a more specific way so that you get where it makes contact with your post (“if I had more time, I would’ve written a shorter letter”). Forgive me if it was somehow hard to understand, English is my second language.
My first issue: the post is titled “Good if make prior after data instead of before”. Yet, the post’s driving example is a situation where the (marginal) prior probability of what you’re interested in doesn’t actually change, but instead is coupled to a larger model with a larger probability space where the likelihood is different at these different points. So, what you’re talking about isn’t really post-hoc changes to the prior, but something like model expansion, as you write in the comment.
In the context of methodologies for Bayesian model expansion, there is a lot of controversy and much ink has been spilled, because being ad-hoc and implicitly accepting a data-driven prior/selected model leads to incoherence; the decision-procedure you now derive from this is not actually Bayesian in the sense that it satisfies all the nice properties people expect of Bayesian decision rules and Bayesian reasoning, it just vaguely follows Bayes’s rule for conditioning. When you write
So the only practical way to get good results is to first look at the data to figure out what categories are important, and then to ask yourself how likely you would have said those categories were, if you hadn’t yet seen any of the evidence.
you are sidestepping all of these issues (what I called “solving by not solving”) and accepting incoherence as OK. And, well, this can be a fine approach—being approximately incoherent can be approximately no problem. But, I think that the post not only fails to address the negatives of this particular approach, positioning it as kind of the only thing you can reasonably do (which is in itself a sufficiently large problem), but fails to consider any other ones (A classic objection to this type of methodology in a canonical introductory textbook, providing one of the alternatives I mentioned, is here, for example, in which the idea is to have a model flexible and general enough that it can learn in essentially any situation; I mentioned other methods in the comment). Do you not see the incoherence of a data-driven prior as bad somehow?
To be clear, the other approach you consider of “never change your model/prior after seeing the data, even if your model makes no sense, your posterior is stuck as it is” is also bad for all the obvious model misspecification reasons. But, at the very least it is coherent (and, of course, by data-splitting you get to enjoy this coherency without being rigid at the cost of a little data, so there’s another approach, much less technical to explain than the nonparametric approach mentioned prior). This is my main problem with the article, really: it proposes just this one idea among several without discussing its positives or negatives in relation to any of the other ones.
My point with this article endorsing “double-counting” is that one way in which this approach (roughly summarized as “construct the model after seeing the data, pretending like you haven’t seen the data”) is that, in comparison to either a nonparametric approach or some M-open idea like model mixing or stacking, it will privilege the particular model which you happened to construct on the basis of the data more so than a fully coherent theoretical approach.
An easy way to see this is to imagine if you were to try this approach while being knowledgeable about all possible models you could have picked (i. e. in model averaging, they wave at a similar critique to this idea in this other intro); in this representation, instead of observing the data and updating yourself towards one particular model representation which fits best with the data, your method is to set one model’s probability to 1 and all others to zero, which is a rather extreme version of double-counting^[1].
So, in my perspective, a good version of this article would not talk about anything being “the only practical way to get good results”, and would situate this idea alongside all the other ones in this vein which have been discussed for decades, or at least sort of gestures at the more common approaches you consider sensible and that you think you can explain nontechnically (hopefully referenced by their names), and at the bare minimum it should explain the pros and cons of what it advocates with more balance. Admittedly, this is a much harder article to write, because the issue has become nuanced, and I would not know how to write it non-technically, at least immediately. However, the issue seems to be nuanced, at least to me, and this level of simplification misleads more than it helps.
1. ^
  In the original comment I decided to talk more about how easy it is to make double-counting methods seem arbitrarily good by way of constructing examples where you know the truth in advance, since of course it looks better if you get to the truth twice as fast, but the double-counting when the data happens to be misleading gets you doubly wrong too, but this objection seems kind of petty and irrelevant compared to the other ones, in hindsight.

vae 30 Dec 2025 4:14 UTC
24 points
10
in reply to: Eliezer Yudkowsky’s comment on: Why does Eliezer make abrasive public comments?
It is a damn shame to hear of this tiredness, and I hope that your mood improves, somehow, somewhen, hopefully sooner than you expect.
This reply, though, I am forced to say, does not quite persuade, and to be totally frank even disappoints me a little. It was my understanding that both MIRI and to some extent one of your goals as of now was one of public outreach and communication (as a sub-goal for policy change) - this was at least how I understood this recent tweet describing what MIRI is doing and why people should donate to it, as well as other things you’ve been doing somewhat recently going on a bunch of podcasts and interviews and things of that nature (as well as smaller things such as separating out a ‘low-volume’ public persona account for reach as well as a shitpost-y side one).
Therefore, to put it maybe somewhat bluntly, I thought that thinking deeply and being maximally deliberate about what you communicate and how, and in particular how well it might move policy or the public, was, if not quite the whole idea, maybe a main goal or job of your organization and indeed your public persona. So, though of course a great many allowances are to be made when it comes to the tiredness, I don’t understand how to square the idea that
you don’t really have an option of continuing to hear from a polite Eliezer; I’d just stop talking instead.
with what you and what MIRI are currently, well, for. You said, and I totally understand why, that the plan is to get other people to ‘take over’ that role, but this doesn’t really make it any less of a bad thing, just more of a hopefully temporary one. Is it truly such an all-or-nothing thing, that you’d just abandon all hope at such an important part of MIRI’s goals instead of trying to learn to be (as other commenters have put it) less abrasive?
I hope it is not too rude to hope that you come upon a different attitude when it comes to this at some point, because as it stands it seems kind of contradictory and self-defeating.

vae 28 Dec 2025 21:21 UTC
26 points
−4
on: Good if make prior after data instead of before
Hmm. I’m sorry to have to say this, but I think this post is not very good.
In summary, I would say that this article is discussing a set of very common, well-trodden issues (Bayesian model checking/expansion/selection, generally speaking problems of insufficient ‘model space’) in a very nontechnical way (which is of course a fine approach on its own, but it’s strange that the thing that this is talking about is not referenced or referred to as the thing it’s called anywhere, instead everything here is presented as if it was a novel first-principles contribution), in such a way that proposes the most wishy-washy solution to the problem (just ignoring it) without reference to any alternatives and without really discussing the disadvantages of this approach.
To go line by line: first, though the title and the framing of this article surround priors, as you kind of point out, the prior isn’t really what’s changing here, but the likelihood (and the part of the prior that interacts with it). You can consider yourself to have, in theory, a larger prior that you just weren’t aware you should be considering. So, the problem that this is dealing with is the classic “I don’t actually believe my likelihood is correct” model, for which there are a billion proposed solutions, which you don’t really touch on for some reason (to name a few, “M-open” analyses, everything in the Gelman BDA chapter on model expansion, nonparametric methods which to be fair you hint at with ‘infinite-dimensional’, etc).
To briefly explain some of these: one classic approach is to consider your prior attempt some model $M_{1}$ , with some prior probability, and your next one an $M_{2}$ , and your final posterior will be an averaging out of each, with some prior model credences. In this specific case, you could probably nest both of the likelihoods into a single model, somehow. The most common applied approach, as many have pointed out, is to just split the data and do whatever exploratory analyses you want on the head of the table or some such approach.
The problem with this classic solving-by-not-solving approach is that, well, you are not being an actual Bayesian anymore; if your (technical) prior depends on observing the data, your procedure no longer actually follows the laws of probability, no longer benefits from Cox’s theorem, has no guarantee relating to Dutch books and does not result in decision-optimal rules (roughly the same reasons orthodox Bayesians reject empirical-Bayes methods, which are a non-Bayesian half-solution to this issue, kind of). So, well, I don’t see the point.
There are good ways and bad ways to be approximately Bayesian, and this particular one seems no good to me, at least without further argument (especially when avoiding it is so easy and common. Just data-split!). Double-counting methods always look good when you are in a thought experiment and are doubling down on what you know to be the right answer, the problem is that they are doubling down on the wrong answers for no reason, too; it seems perfectly reasonable to me to admit that ⁵⁰⁄₅₀ posterior after seeing just a little bit of data, then some model refinement happens, and your posterior upon seeing the full data looks more ‘reasonable’.
So, I don’t know. These are good (classic) questions, but I am forced to disagree with the notion that this is a good answer.

vae 31 Aug 2025 22:52 UTC
1 point
0
in reply to: Richard_Ngo’s comment on: Contra Yudkowsky’s Ideal Bayesian
Neat post, thank you! Also, you seem to have posted this twice.

vae 31 Aug 2025 22:50 UTC
1 point
0
in reply to: Ape in the coat’s comment on: Contra Yudkowsky’s Ideal Bayesian
Maybe! But I would expect him to change his view to something like this in case you managed to persuade him that there is some crucial flaw in Bayesianism. While your goal seems to be to propagate the toolbox-view as the only valid approach. So you might as well engage with a stronger version of law-view right now.
Well, maybe, I don’t know. As it stands it just seems best to argue against what he has said at his word then to assume otherwise, though, insofar as other people take this view at face value. Though, if such a thing does come about, I would of course have to write a different post. This could be some part of LessWrong culture that I am just ignorant of, though, so apologies.
And then I assign equiprobable prior between these models and start collecting experimental data—see how well all of them perform. Do I understand correctly, that Walker considers such approach incoherent?
It depends on what you mean by see how well all of them perform. In this situation, where you can easily get a reasonably small set of models that might represent your total uncertainty, and then (crucially) obtain whatever estimates or uncertainties you desire by updating the posterior of the complete model (including these sub-models—i. e. $P (p_{1}, p_{2}, \dots, p_{6} | X)$ must be marginalized over M).
To a Bayesian, this is simply the uniquely-identified distribution function which represents your uncertainty about these parameters—no other function represents this and any other probability represents some other belief. This of course includes the procedure of maximizing some data score (i. e. finding an empirical ‘best model’), which would be something like $P (p_{1}, p_{2}, \dots, p_{6} | X, M = m_{i}^{*})$ where $m_{i}^{*} = {arg}_{m \in M} max T (X, m)$ in which T is some model evaluation score (possibly just the posterior probability of the model).
This seems like a very artificial thing to report as your uncertainty about these parameters and essentially guarantees that your uncertainties will be underestimated—among other things, there is no guarantee that such a procedure follows the likelihood principle (for most measures of model correctness other than something in proportion to the posterior probability of each model, but if you have those at hand you might as well just marginalize over them), and by the uniqueness part of Cox’s proof it will break one of the presuppositions there (and therefore likely fall pray to some Dutch Book). Therefore, if you consider these the reasons to be Bayesian, to be a Bayesian and do this seems ill-fated.
Of course, he and I and most reasonable people accept that this could be a useful practical device in many contexts, but it is indeed incoherent in the formal sense of the word.
But why would the prior, capturing all your information about a setting, be sufficiently far off from the true value, in the first place?
Well, you told me to grab you the simplest possible example, not the most cogent or common or general one!
But, well, regardless if you’re looking for this happening in practice, this complaint is sometimes levied at Bayesian versions of machine learning models where it is especially hard to convincingly supply priors for humongous quantities of parameters. Here’s some random example I found of this happening in a much less complex situation. This is all beside the point, though, which is again that there is no guarantee that I know of for Bayesians to be finite-sample calibrated AT ALL. There are asymptotic arguments, and there are approximate arguments, but there is simply no property which says that even an ideal Bayesian (under the subjectivist viewpoint) will be approximately calibrated.
Note that I also have no reason to assume the prior must be sufficiently close to the true value—since a prior for a subjectivist is just an old posterior, this is tantamount to just assuming a Bayesian is not only consistent but well on his way to asymptotic-land, so the cart is put before the horse.
Interesting! (...) “so the question is still open”.
Few Frequentists believe that there is an absolute principle to which an estimate is the unique best one, no, but this doesn’t make the question ‘still open’, just only as open as any other statistical question (in that, until proven otherwise, you could always pose a method which is just as good as the original in one way or another while also being better in some other dimension). ¹⁄₂ seems hard to beat, though, unless you specify some asymmetrical loss function instead (which seems like an appropriate amount of openness to me, in that in those situations you obviously do want to use a different estimate).
though everyone is in agreement that Bayes theorem naturally follows from the axioms of probability theory.
Yes, of course, but from this does not follow that any particular bit of thought can be represented as a posterior obtained from some prior. Clearly, whatever thought might be in my head there could easily just not follow the axioms of probability, and frankly I think this is almost certain. Maybe there does exist some decent representation of these sorts of practical mental problems such as these, but I would have to see it to believe it. Not only that, but I am doubtful of the value of supposing that whatever ideal this thought ought to aspire to achieve is a Bayesian one (thus the content of the post—consider that a representation of this practical problem is another formally nonparametric one, in that the ideal list of mathematical properties must be comically large—if I am to assume some smaller space, I am implicitly claiming a probability of zero on rather a lot of mathematical laws which could be good which I cannot conceive of immediately, which seems horrible as an ideal.)

vae 30 Aug 2025 15:36 UTC
2 points
0
in reply to: jelly’s comment on: Contra Yudkowsky’s Ideal Bayesian
Thank you for spotting that.

vae 30 Aug 2025 13:18 UTC
1 point
0
in reply to: Cole Wyeth’s comment on: Contra Yudkowsky’s Ideal Bayesian
Is it? Certainly Yudkowksy takes a lot of inspiration from Jaynes but I don’t remember him beating on this particular drum. Of course there were arguments as to why being Bayesian is correct from a philosophical perspective, just not so much that everything must be approximating it. Though, it’s been years since I read him, so I could be wrong.

vae 30 Aug 2025 10:56 UTC
1 point
0
in reply to: Ape in the coat’s comment on: Contra Yudkowsky’s Ideal Bayesian
You can have a law-view interpretation of such synthesis where we conceptualize Bayesianism as an imperfect approximation, a special case of the True Law, which should also capture all the good insights of Frequentism.
Yes, but such an interpretation falls outside of Yudkowsky’s view, as I understand it (for example on that X thread in another comment in this post, and his comments on other statistics topics I’ve seen around—could fish for the quotes, but I’m a bit held up at this precise moment), which is what that I’m focusing on here.
On Walker, on that paragraph he is criticizing the specific (common) practice of comparing separate Bayesian models and picking the best (via ratios or errors or some such) when there is uncertainty about the truth instead of appropriately representing this uncertainty about your sampling model in the prior.
Rolling a die is a bit of a nifty example here since it’s the case where you assign a separate probability to each label in the sample space, so that your likelihood is in fact fully general, which is where the idea for a Dirichlet prior comes from in an attempt to generalize this notion of covering all possible models for less trivial problems. In the rest of the intro, Walker points to Bayesians fitting models with different likelihoods (i. e., Weibull v. s. Lognormal, I think he is a survival guy), each with their own “inner” priors, comparing them against each other and then picking the one which is “best” as incoherent, since picking a prior just to compare posteriors on some frequentist property like error or coverage is not an accurate representation of your prior uncertainty (instead, he wants you to pick some nonparametric model).
On Bayesian finite-sample miscalibration, simply pick a prior which is sufficiently far off from the true value and your predictive intervals will be very bad for a long time (you may check by simulation on some conjugate model). This is a contrived example, of course, but this happens on some level all the time, since Bayesian methods make no promise of finite-sample calibration—your prediction intervals just reflect belief, not what future data might be (in practice, I’ve heard people complain about this in Bayesian machine learning type situations). Of course, asymptotically and under some regularity conditions you will be calibrated, but one would rather be right before then. If you want finite-sample calibration, you have to look for a method which promises it. In terms of the coverage of credible intervals more generally, though, unless you want to be in the throes of asymptotics, you’d have to pick what is called a matching prior, which again seems in conflict with subjectivist information input
On minimax: I don’t know how to format math in this site on a phone, so I will be a bit terse here, but the proof is very simple (I thought). in statistics, when we call an estimator “minimax” it means that it minimizes the maximum risk, which is the expectation of the loss over the distribution of the estimator. Since we have no data, all estimators are some constant c. The expectation of the loss with is just the loss with respect to the parameter (i. e. (c-p)^2). Clearly the maxima are taken when p is 0 or 1, so minimize the maximum of c^2, (1-c)^2, which has a minimum at c=0.5. Which is to say, ¹⁄₂ has this nice Frequentist property, which is how one could justify it.
On your last comment, it seems like a bit of an open question to attribute the existence of practical intuition and reasoning about mathematical constructs like this to a Bayesian prior updating process. Certainly I reason, and I change my mind, but to me personally I see no reason to imagine this was Bayesian in some way (or that those thoughts were expressed in credence-probabilities which I shifted by conditioning on a type of sense-data), nor that I would be ideally doing this instead. But, I suppose such a thing could be possible.

vae 29 Aug 2025 16:08 UTC
2 points
0
in reply to: Ape in the coat’s comment on: Contra Yudkowsky’s Ideal Bayesian
I think you’ve misunderstood the point somewhat. On the question of ‘taking the best from both’, that is what Yudkowsky calls a “tool” view, whereas I’m trying to argue against his view of its status as a necessary law of correct reasoning (see my other comment for related points). Insofar as you acknowledge that Frequentists can produce good answers independently of there existing some Bayesian prior-likelihood combination they must be approximating, we agree.
Still, the problem of ‘taking the best from both’ from a philosophically Bayesian view is that it is incoherent—you can’t procedurally pick a method which isn’t derived through a Bayes update of your prior beliefs without incurring in all the Dutch book/axiom-breaking behaviours coherence is supposed to insure against.
There is more responsibility on the Bayesian: she gets more out in the form of a posterior distribution on the object of interest. Hence more care needs to be taken in what gets put into the model in the first place. For the posterior to mean anything it must be representing genuine posterior beliefs, solely derived by a combination of the data and prior beliefs via the use of the Bayes theorem. Hence, the prior used must genuinely represent prior beliefs (beliefs without data). If it does not, how can the posterior represent posterior beliefs? So a “prior” that has been selected post data via some check and test from a set of possible “prior” distributions cannot represent genuine prior beliefs. This is obvious, since no one of these “priors” can genuinely represent prior beliefs. The posterior distributions based on such a practice are meaningless.
- Stephen J. Walker (first chapter of Bayesian Nonparametrics)
Not only that, but insofar as you ‘want both’ (finite-sample) calibration and coherence, you are called to abandon one or the other—Insofar as there are Bayesian methods that can get you the former, they are not derived by prior distributions that represent your knowledge of the world (if they even exist in general, anyway—not something I know of).
On your query about coins, ¹⁄₂ is minimax for the squared error, I believe. But, on a more fundamental level, at least to me most of the point of being Frequentist is to believe in no unique and nontrivial optimal framework for reasoning (i. e. not a hodgepodge of principle-less methods), that there are only good properties which a method can or can’t obtain.

vae 28 Aug 2025 14:11 UTC
2 points
0
in reply to: rotatingpaguro’s comment on: Contra Yudkowsky’s Ideal Bayesian
Hello! Thank you for the comment, these are good points.
- I do not consider myself a Rationalist nor know much of anything about Yudkowsky’s more current positions on this subject, but I probably should have mentioned somewhere in the post that this article was partly motivated by this discussion on X, and his comment. I must admit I do not really grasp what he is gesturing towards with the point he makes there, but it seems like he still believes some version of the original point as stated.
- This post is not about Bayesian inference as practiced by mortal statistical workers; I have other reasons to justify my Frequentism there, but I wrote this so as to eschew the “Tool vs. Law” distinction that seems to be sometimes drawn here. Of course, Bayesian methods in statistics are sometimes useful (it’s hard to justify a hierarchical model without reference to conditioning, “H-likelihood” feels like the sort of post-hoc methodological loop-the-loop that I criticize Bayesians for), and I have used them myself here and there. I am very interested to hear what methods you derived through Bayesian thinking which are not equivalent to a Frequentist estimate, though!
- I agree with you here, almost completely—it just doesn’t seem like what Yudkowsky is saying. To wit:
I have been justly chastised by the discussants for spreading alarm about the health of the body Bayesian. Certainly it has held together more successfully than any other theory of statistical inference, and I am not predicting its imminent demise. But no human creation is completely perfect, and we should not avert our eyes from its deficiencies. In its solipsistic satisfaction with the psychological self-consistency of the schizophrenic statistician, it runs the risk of failing to say anything useful about the world outside.
(Though I would personally add that, even though it’s probably the best unifying principle in statistics, there is no need to adhere to any such general principle when there are better alternatives.)
- This is one of those practical questions which I tried to avoid here (maybe I should just write a separate Frequentism post eventually), but yes, I agree, and would characterize this as probably the biggest advantage of Bayesian methods in practice—that they are “plug-and-play”, that if you specify a minimally sensible model you have strong guarantees (in nice, parametric, problems) that your answers will be sensible too.
  
  I imagine this is why they are most often seen in fields like astrophysics, where you don’t want to seek out the best methods for really complicated physical models, you just want something that works well without having to worry. Still, the comparative strength of Frequentism is being able to specify and more directly obtain exactly what you want, sometimes optimally. An easy example is exact finite-sample calibration: if I want my predictions to be calibrated (and there are many situations in which I do), the methods which will guarantee that I get this will involve conformal inference method or the like. I don’t have to wrangle a prior which matches this or hope everything works out. Other examples are, say, robustness, or in experiment design.
  
  You comment on assumptions, here, but in my opinion you have it backwards—if your Bayesian model handles non i.i.d-ness well, this is because the dependency shows up in the likelihood, which (say) the MLE still handles quite well (vaguely asymptotically efficient and so on). What if you want to be distribution-free, or want to check if your answers are robust to your model being wrong in some directions? Maybe there will be better Bayesian answers here someday, statistics generally is a young field, but (in practice) I think the Frequentists just take the cake on this one.
- This is again correct, of course, but I am specifically criticizing the essence of Yudkowsky’s point of “if it’s any good, it must be approximating a Bayesian answer”: who’s approximating who? Here it seems much more sensible to say that we have a good answer (the OLS estimate), one that we have reasons to prefer in some scenarios (e. g. Gauss-Markov, general distribution-free niceness) that a Bayesian method, strictly speaking, can only approximate, and which seems at odds with a pure subjectivist point of view (because the prior is incoherent, but this is much more salient in the Cox model example). Indeed in practice this is irrelevant.

vae 27 Aug 2025 17:10 UTC
3 points
0
in reply to: kave’s comment on: Contra Yudkowsky’s Ideal Bayesian
Yep—this is the standard term for the property (e. g. in that Seidenfeld paper).

vae 3 Dec 2020 0:56 UTC
3 points
0
on: The LessWrong 2018 Book is Available for Pre-order