You Heard of “Bayesians Don’t Predictably Update” Because of Robin Hanson
Robin Hanson didn’t invent the idea of a martingale process. The lilim say that Ville in 1939 gave the term its mathematical meaning and Doob turned it into the modern theory. But to state this is not to realize that your own probability assignments over time ought to form a martingale in your own expectations.
Robin Hanson didn’t invent the Efficient Markets Hypothesis, which (correctly restated in its weak and relativized form) says that you cannot predict almost all of the short-term relative price movements in asset markets. Bachelier in 1900, say the lilim, was first to think of market prices as a fair gambling game with random-walk dynamics; Samuelson in 1965 gave the martingale argument; Fama’s 1965 thesis named the idea the Efficient Markets Hypothesis and organized it into weaker and stronger forms.
The EMH is very near, but not quite identical to, the principle that you should not be able to predict a net direction in your own price movements, if you think of your own probability assignments as if they were prices. It is a very similar set of mental motions and ideas. But if Fama ever spelled that part out in words, the lilim do not know it.
The lilim think “Bayesians Don’t Predictably Update” would probably have been folk knowledge among Savage, Good, Raiffa, &co in the 1960s and 1970s.
The lilim say that von Fraassen’s reflection principle in 1984, and Goldstein in 1983 on “the prevision of prevision”, are the first explicit philosophy-literature statements they are confident about.
You have heard of it because of Robin Hanson.
Robin Hanson is the person I know who went around saying to people that they ought not to update in a predictable direction, as a consequence of the martingale principle underlying EMH, applied to their personal epistemic lives.
So far as I know, all popularization of this exact idea is downstream of Robin Hanson.
“But Eliezer!” you say. “The lilim said that some of this popularization is due to your having concretized it into the principle of Conservation of Expected Evidence! This seems like sufficiently basic probability theory that you probably would’ve seen it at some point without Robin Hanson prompting you into it?”
Perhaps, but where was Conservation of Expected Evidence first written up? Overcoming Bias, Robin Hanson’s group blog that he’d invited myself and Nick Bostrom to contribute to. Who’s to say, absent Robin Hanson, that the Less Wrong Sequences would ever have happened? At least some parts of history are fragile; I don’t look back and see it as an inevitability. Who’s to say you would’ve heard of any of the pieces of probability theory that I popularized, without Robin Hanson?
“Surely an idea like that would’ve become popular as the result of people playing prediction markets. If you don’t think that way while playing prediction markets, if you don’t generalize market thinking to predictions, you’ll lose all your money!”
There’s a big gap between tacit knowledge and explicit knowledge, which is why you don’t find market traders explicitly stating EMH or von Fraassen’s reflection principle much much earlier in history.
But also, who the hell do you think is responsible for your engagement with prediction markets, if not Robin Hanson? Yes yes, papal conclave betting dates back to the 16th century, and the Iowa Political Stock Market in 1988 seems to have Hanson-independently implemented that idea about US elections instead of papal conclaves, with bookies no doubt running books on elections earlier. But Hanson wrote up the academic case for prediction markets generally, invented policy prediction markets, pointed out prediction markets’ potential uses in science and governance and corporate governance, and pushed for early prediction markets to get started. And was upstream of the community that, having heard about Hanson’s ideas, then founded and funded Manifold; which used Hanson’s notion of a market scoring rule to not require market makers in every idea’s market.
Maybe you’d have somehow ever come across the idea “Bayesians shouldn’t predictably update”, without Hanson’s role as a public intellectual turning that into a forceful, in-your-face application of previously invented principles to internal mental life.
And without Robin Hanson’s blog, upstream of my own blogging mentioning the topic, and indeed upstream of the subsequent formation of a rationalist community.
And without Robin Hanson’s creation of prediction markets, upstream of Manifold.
But it seems to me, looking back over the pieces of history that I know with any intimacy, that history looks causally fragile where its flows are not massively locally incentivized. I make no claim that the Less Wrong Sequences or the formation of a rationalist community happen inevitably with or without Robin Hanson. I maybe figure out the same principle without copying off Hanson, because that’s pure math and similar to many other Bayesian applications I figured out. Sociological history is more fragile.
Robin Hanson was not the first to think that Bayesians should not predictably update. Robin Hanson was not first to publish the idea. Robin Hanson is upstream of all the causality by which you actually heard about the idea, and I do not particularly buy an inevitabilist case for how it surely counterfactually happens either way; there was nobody who made a billion dollars immediately off each local step on the way to you hearing about it. The prior potential energy field for society does not have a massive incentive slope leading surely to it.
Robin Hanson credits as especially influential on prediction markets Eric Drexler, Mark Miller of the same crowd, and Phil Salin of Xanadu which influenced Drexler. Plausibly none of this happens without Drexler either. Robin Hanson and Eliezer Yudkowsky as you now know them may not happen if Eric Drexler does not exist.
Nor do I know of any person in history who looks to have been “95% of Eric Drexler”, ready to step into Drexler’s place and create 95% of the same ideas and institutions. Person-space is sufficiently multidimensional that the Curse of Dimensionality applies in massive force; the edges of weird-person-space are sparsely populated in many of the ways that matter to historical counterfactuals. People who are unusual on historical scales often have no visible neighbors in that massively multidimensional space of people. Apple could not go to the Steve Jobs club and hire another Steve Jobs after the first one died, despite having infinite money to pay one.
It is in this sense, and exactly this sense, that I sometimes remark that AI companies do not look like they would have had any clue about some concept that existed earlier in the literature and much earlier in science fiction, if I had not precisified it and popularized it as an answer sheet for them to copy from. I am not claiming to have created whichever idea full-formed from scratch without a vast trailing lattice of intellectual history. I am looking at the historical counterfactual in a world where locally unincentivized history generally strikes me as fragile, and knowing as a matter of sociology which ideas were (not) previously common conversation at Artificial Intelligence conferences.
This post is further evidence (if any were needed) that Yudkowsky should get off of Twitter. What’s the point of wasting his brainpower on this sort of thing?
(Just had the thought what if a main effect of twitter discourse overall is to suck people in and away from more productive actions like eg. meeting with representatives? Like it seems fairly addictive.)
This post itself is Yudkowsky getting off of Twitter… for a moment.
For the confused, this post is a subtweet of several replies to this tweet of his.
Is this in response to someone claiming that Robin and you haven’t been pivotal in both popular and expert thinking about these topics? Or more generally, is it a statement that history (recently, at least, but probably in most cases) is highly contingent on individuals?
I missed the other side of the debate, so most of this post seems obvious and/or unnecessary.
Sometimes it is, sometimes it isn’t, you need to look at the details. Two people have invented telephone during the same week. Did two people start blogging about artificial superintelligence in the same decade?
I’m pretty sure I learned about this formally from Taleb if not earlier fwiw. I also think it’s an extremely obvious idea once someone groks Bayesianism, which I partially learned from you and partially from Silver and partially from reality.
(I’ve only met Hanson once and not at a party).
Martingales or personal probability assignments?
Both martingales and the idea that martingales should apply to personal subjective probability assessments, actually I’m not sure how they are used elsewhere. There’s a decent chance I studied the pure math behind it at some point (I studied both pure math and economics in undergrad), but it’s definitely the type of thing I think about subjectively/intuitively much more than as a proof technique.
This makes me wonder about the flip side: what ideas (both good/true and bad/false) caught on due to inevitable historical forces, rather than the luck of a weird advocate happening to exist and partially succeed?
Probably most of the basic science we enjoy owes to it being true and inevitable. It’s useful to have engineers who know Newtonian physics. Any particular diet fad is fragile, but the concepts of “calories” or “carbohydrates” to be measured are presumably convergent because the chemistry admits those natural abstractions.
The prospect of convergent bad ideas is unsettling. (And necessarily more controversial: if it was easy to convince everyone that an idea is bad, it wouldn’t be convergent.) Rent control might be a locally-less-controversial example. (It’s just the obvious policy for renters to advocate for; no one needed to invent it.)
Some ideas might be convergent given a particular context which was itself fragile. Once you have a multi-ethnic democracy, the blank slate becomes convergent (or at least has a large basin volume) as the Schelling point for preventing group conflicts (Arthur Jensen seems more counterfactual than any of his environmentalist foes), but the position of your multi-ethnic democracy might be the fragile result of previous wars.
The tendency towards overregulation and crippling innovation generally seems convergent. Some ideas could’ve gone differently had they got cemented earlier before that trend, though.
For example, I suspect that if history played out a little differently, we’d have not crippled nuclear power and thus have cheap green energy. Maybe it would’ve even worked if some of the early environmentalist advocates were championing it. Alternatively, maybe if nuclear power (or some crucial scaling innovations) had just been invented a couple decades earlier, it would’ve taken over the energy market and maybe even if it gets kinda crippled later it still manages to be powering the vast majority of a larger energy supply.
I think you’d enjoy this video, which includes an extended discussion about the history, sociology, and economics of energy consumption as an explanation for why we don’t have flying cars.
Disclaimer: the presenter isn’t always the most epistemically responsible. For example, the picture that he paints of Ignaz Semmelweis is misleading, but he seems to stick to the facts on matters of engineering.
I don’t tend to watch videos much, sorry (though I still am glad I get recommended them just in case). If it’s mostly content covered in Where’s My Flying Car then this rec will bump it up my intuitive ordered to-read list.
That’s respectable. The video’s presenter is the same as the book’s author, so I assume that the book covers the subject in even greater detail.
I think there are two possible equilibria: radiophobia plus non-proliferation or widespread use of nuclear energy and every country having nuclear weapons. Restricting nukes but not reactors was not a real option.
One clear alternative would be for basically the current nuclear powers to use a bunch of nuclear energy. I don’t think France’s nuclear policy has led to proliferation. Then they sell to the rest of the globe, or at least the parts that like the cheap energy more than the (political) power it gives up and have enough infrastructure to transfer the power and aren’t having e.g. the US controlling all the powerlines going in. But idk exactly how feasible international energy transfer is (would nuclear power world have a bunch of hydrogen fuel?). It’d still be pretty different if ‘only’ US + EU (via France and UK) and Russia and China were mostly nuclear powered. Or alternatively the powers manage their allies and enemy’s allies to allow them to have nuclear power but not have nukes, which is probably how e.g. Japan has some nuclear power without having any nukes. I don’t think Japan would suddenly become a nuclear weapons state if they went all in on nuclear power.
Long-range energy transmission was not practical back then, and hydrogen is a terrible storage medium. As for power stations abroad, quite a few were built. But slow-neutron reactor usage cannot be scaled very much, probably less than order of magnitude. Practical uranium reserves just are not that huge. And fast-neutron reactors produce plutonium in large quantities.
But the main reason is vibes. The non-proliferation regime is maintained mostly not by a threat of force (Israel made nukes and nobody stopped them), but by a widely held opinion that they are only good for a MAD deterrence. Without radiophobia, nukes would be used as tactical weapons, and a lot more countries would want them.
Rent control is an example of a more general appealing bad idea of top-down price-setting.
But the obvious idea that is intuitively appealing and also catastrophically bad, is communism.
The headline claim seems true for people invested in the rationalist community but I don’t understand the case you’re making. It seems like the argument here is that the lineage of Robin Hanson->yourself->LW in general is upstream of people in general treating this phrase as a reason to cultivate the corresponding mental habit? As in:
This is probably true for people alive today, because communities using probability as a formal framework for refining their epistemics are rare, but I’m not convinced that this is relevant to the headline claim. As a child I remember realizing this within an hour of being taught Bayes’ Theorem, and while it’s possible that my math camp instructors were themselves influenced by yourself or Hanson and guided me in this direction subtly, I doubt it. I was also not particularly precocious among precocious children, such that I doubt my experience here is rare. That is to say, I think you may be overestimating how basic this fact is, and I think the reason it’s hard to find it formally written up in the relevant contexts is that it’s so obvious as to be embarrassing to try to explain to anybody who already has relevant skill in probability. Many people have engaged with probability as a formal discipline in history, very few of them have taken its simplest exercises as seriously as you have in their writing, and the discipline is old enough that those who did were mostly not writing in such a well-preserved medium. (This is a judgment but I’m unsure of its valence. Taking simple things seriously is often a good idea and often naval-gazing and I struggle to differentiate the two.)
By comparison, not many people have thought about AI safety as a formal discipline, and so I think analogous claims in that field are much more likely to be true—the relevant ideas are legitimately understudied, very few capable people have tried to build useful frameworks for that project, and you’re correct to claim a bottleneck position in that story. Indeed, very few people took AI safety seriously until a couple of years ago, and so ~all of the expertise is held in a highly-noncomformist social group with a ~unique and controversial culture around sex, emotional well-being, communication, values concerning future people and the futures of current people, etc. This is in my view the main problem for AI safety communicators right now! If this bottleneck didn’t exist, neither would this problem. No comparable thing seems true for probability.
[Of course, if the target audience for this post is “people who were first introduced to Bayesian epistemics from the writings of Eliezer Yudkowsky” rather than “people who might read a front-page post on LW” then I’m entirely incorrect and this comment can safely be deleted.]
In contrast, it took me about 10 hours of reading and rereading, carefully parsing, and doing exercises, when I was about 28, to get to the point of properly understanding Bayes’ Theorem. (This is after having read the sequences years earlier, and having been embedded in the Bay area rationalist community, and working at CFAR for half a decade.)
I do not think I would have hit on this insight independently, without it being pointed out to me, or without my doing a bunch of independent thinking about technical epistemology that I would have been very unlikely to do, without someone like Eliezer pointing me in the right direction.
I think you might be underestimating how non-intuitive even the basic facts are to almost everyone (possibly even to people you would generally regard as a “smart person”, though that is indeed less obvious).
Since the Sequences expanded probabilistic thinking to a broader audience, who were mostly not reading the works of the probabilistic thinkers of Eld or even thinking about the math by themselves much, the Sequences would be the source of probabilistic principles among almost all of that community, even if they were capable of coming up with conservation of expected evidence within a couple hours of thinking about basic probability.
I also find it intriguing that speck did not say “It took me a few minutes to figure out it out after learning about conditional probability as a kid?”, which presumably they learned before Bayes law, and is the plausible pathway through which a significant chunk of the new probabilistic thinkers could’ve rediscovered it.
For myself, I think I was familiar with how conditional probability worked before I knew algebra, at a level sufficient to solve a simple problem embodying P(X) = P(E)P(X|E) + P(-E)P(X|-E), and people who did math competitions as kids are liable to passively have that knowledge—yet at no point did I realize that the expected evidence is conserved, even after I learned enough algebra to write that equation down. I would give myself a 10% chance of having come up with it had I had the concept of updating beliefs in response to evidence. As a benchmark, Bayes Law and Conservation of Expected Evidence were easy for me to learn from the Sequences at ~13 - that is, someone with the math skill to find them easy as a kid is still probably not going to come up with CoEE!
I recall learning these simultaneously, but it’s possible I had seen and not internalized conditional probability before. The pathway as I remember it was: broad description of what it means for an event to have probability p, description of what it means for an event to have probability p conditional on E, probabilities of Boolean combinations of events, derivation of Bayes, and then in exercises derivation of and application of the Law of Total Probability. After you’ve used P(X) = P(E)P(X|E) + P(-E)P(X|-E) like twice, I think there’s a reasonable chance of noticing that you keep computing 0.6P(X|E) + 0.4P(X|~E) or 0.2P(X|E) + 0.8P(X|~E), thinking a little bit about units, recognizing this as a weighted average, and then there’s nothing really to observe.
If people are learning about probability in ways other than writing down lots of toy numbers and doing arithmetic, or aren’t making the leap of treating your own credences as probabilities and pieces of evidences as events to condition on, then it’s a bit more plausible to me that figuring this out is difficult. For example, @Eli Tyre, you write:
and depending on what you mean by ‘exercises’ here, one possible explanation for the difference in our experiences here is that reading, parsing, participating in the rationalist community, reading the sequences, and working at CFAR are all something other than mechanically computing dozens of toy probabilities until the steps become muscle memory. Another possible explanation is that I’m just having a theory of mind failure, which is the main thing I’m trying to settle here.
It’s also plausible that by ‘Bayesians don’t predictably update’ you mean ‘conservation of expected evidence’ you mean ‘the law of total probability’? My general model here is that the law of total probability is a fairly obvious fact, the sort you couldn’t hope to miss by doing enough rote computation on the relevant problems. Meanwhile ‘conservation of expected evidence’ is a way of interpreting that law by treating your credence in an event as a random variable distributed against the evidence you might see, and ‘Bayesians don’t predictably update’ is the mental habit of ensuring that your beliefs represent truth values by ensuring that they follow this law. Is this roughly in line with the way y’all are using these terms?
I meant to use “conservation of expected evidence” as that interpretation of the law of total probability, and was using no particular word or phrase for the mental habit (which I would’ve just called “the mental habit of trying to do this”).
Personally I’m unsure whether I’d have noticed it if you erased it from my mind but kept my math ability and had me toy around with probabilities, simply because most of the time I’m using the law of total probability I am not in fact interpreting the conditioned variable as evidence for a hypothesis. I think the best shot I have at thinking to look in that direction is if I got into an argument with someone and then ended up reinventing the idea that you shouldn’t expect to update on average via economics reasoning on stock prices.
...I wonder how many more people would know Bayes law if someone just convinced the common national math competition orgs to put questions using it on their tests… after all, that’s how I ended up passively having an intuitive understanding of expected values and conditional probability.
I disagree. Up until the proof-based courses of undergraduate upperclassmen, math education involves an enormous amount of hand-holding. This is not a boast—I’m not saying that the classes are easy. What I mean is that students aren’t expected to reach any conclusions on their own, and every conclusion that they’re presented with is reinforced through practice exercises and exam questions, often including word problems.
This depends on the education tract, but broadly this is true and doesn’t seem relevant to what I’m saying here unless I’m missing something. Most people don’t ever develop any relevant skill in probability! Most people don’t treat math as an epistemic foundation, and so wouldn’t think the headline claim is obvious or meaningful. That is to say: I agree that math education involves a lot of handholding, even at an upperclassman level, but that’s because math education is targeted toward people who don’t really think about math outside of the classroom. People who do think about math outside of the classroom, for example who think of their beliefs as probabilities, are much more likely to eventually find their way to these conclusions.
For what it’s worth when I was doing the research for 100 Years Of Existential Risk I noticed the same thing. Of all the people in that story Drexler seems like the most unique contribution with the least antecedent or precedent. As far as I can tell it’s Feynman giving his talk on plenty of room at the bottom and then Eric Drexler, which is a wild leap in sophistication going from just the roughest sketch of the basic idea to Nanosystems.
https://www.lesswrong.com/posts/kFRn77GkKdFvccAFf/100-years-of-existential-risk
EDIT: Realizing that at the time I researched that I didn’t have access to LLMs, I just asked Astra about this and it pointed out something funny (along with a handful of extremely marginal less developed precursors to Drexler), Drexler actually came up with the idea before finding the Feynman talk:
Yeah, Drexler was obviously-to-a-polymath on a separate track from Feynman, and Feynman was the distant prestigious person who said a thing helpful to Drexler but wasn’t remotely being Drexler about it.
I am not sure that this implies that the formation of a rationalist community[1] was severely contingent.
I cited the example of a Soviet book popularizing science and mocking a mysterious answer to the origins of life and to the properties of opium causing the users to fall asleep.
The USSR had a famous children’s book where the chapters “How Elektronik was born” and “How Elektronik was taught” describe the experiment which led to Elektronik’s creation in much detail, without a detailed AND famous Western counterpart.[2]
Yudkowsky’s endgoal was to ensure that someone in AI labs seriously thinks about alignment and utopia and arrives at conclusions close to the truth. How strongly would one update against Yudkowsky’s hypothesis of wildly multidimensional personalities if ideas like alignment being extremely hard to test were discovered in cranks’ books? As for utopia, I am not sure that the necessary measures are AI labs thinking about utopia instead of AI-2040′s authors writing down their plan (see, e.g. the Public’s Perspective, speculative Epilogue, the Space Governance Plan and other governance supplements) and politicians implementing them.
According to Claude Opus 5.5, the closest counterpart at the gears level is James P. Hogan’s The Two Faces of Tomorrow (1979), followed by Destination: Void (Frank Herbert, 1966), Galatea 2.2 (Richard Powers, 1995), The Lifecycle of Software Objects (Ted Chiang, 2010).
It is perhaps ironic that the object level claim this post uses as its exemplar is mistaken: ideal Bayesians often update in a predictable direction.
In essence: unpredictable direction only follows when the underlying martingale is symmetrical. If it is skewed, you can have ‘I expect it is more likely my credence in X will be lower tomorrow, but there’s a smaller chance it is much higher, such that E(credence tomorrow) = credence today’.
You get skew—thus bias in future updates, and ‘predictable net direction’ - for many financial instruments (e.g. options and time decay), ~all “will [event] happen by [date]?” forecast questions, and (I think) any current credence not at equipoise. So, if anything, good Bayesians will typically be updating in a predictable direction.
It also seems to me this misunderstanding is more common within than without the rationality community (although Abram Demski identified and tried to correct it back in 2019). I’ve only seen rationalists and not fellow forecasters express confusions like “Why is this Metaculus question steadily trending up/down? Why hasn’t everyone front-run the obvious directional trend by now? EMH, right?”. I don’t know whether Hanson was the primary wellspring, but he might not want to take credit for this one.
From Conservation of Expected Evidence, which is not at all mistaken and not at all ambiguous on this.
It is not unheard of for an author to get something right in one piece, yet wrong in another.[1] Given my reply is to this post, not that one, the words here, not there, are the relevant ones.
Although natural language can be ambiguous, “[Y]ou ought not to update in a predictable direction” is generally false on plain reading: you can reliably predict the direction of travel for many beliefs, even if held by an ideal Bayesian agent. “[Y]ou should not be able to predict a net direction in your own price movements” has a bit more wiggle room thanks to ‘net’, but I think most would take (e.g.) ‘I expect my credence in X to continue to rise’ as predicting a net direction.
You ought not (per martingale) have expectations for your credences in future which differ from their current values. But this offers little constraint in terms of (commonsensical) ‘predictable update direction’: if I think P(X) = 0.6, it follows the likeliest ‘net direction’ of my credence is to continue climbing up to P(X) = 1 - it also usually follows the modal (i.e. likeliest) future trajectory is for my P(X) to monotonically climb to P(X) = 1.
This means ‘predictable update direction’ is a very poor diagnostic of irrationality: “I currently think P(X) = 0.6, but next week I expect to think P(X) = 0.7” only violates martingale on specific meanings of ‘expect’, and “So why haven’t you updated already?” an invitation to mistake on common ones. ~Monotonic trends, not more symmetrical random walks, should be the typical observation in prediction markets/forecasts—where nonetheless you can’t beat the market in expectation by front-running the obvious trend.
So a maxim of ‘you ought not to update in a predictable direction’ is apt to mislead, ditto analogies to the EMH (where the assets people have in mind tend to move ~sideways rather than steadily up or down). It is a recipe for people to front-run themselves into overconfidence, or to be confused by (or sneer at) correct epistemic practice because its history shows clear directional trends which ‘shouldn’t happen’ if things were rational/efficient.
E.g. The author of conservation of expected evidence also wrote:
And:
I think these are clearly mistaken per the third paragraph in my original comment, perhaps modulo heroic exegesis (in the other direction, a pedant could complain you should—contra the last sentence in the second quote—be updating at least a little off evidence you predicted you would see so long as you weren’t certain you’d see it).
… wasn’t this clarification included in the sequence post? With a high probability of a small update balancing a low probability of a large update? Sure, explaining the thing is going to lead to a different sort of error from people who misunderstood, but it at least creates the opportunity for people to understand what it means to not update on average. I don’t think the harm of causing the misunderstanding outweighs people not having any of the idea at all as well as there being a good chunk that understood!
I’m going to be that annoying “well, actually” guy, and nitpick one exception to “Bayesians don’t predictably update” (BDPU): the Sleeping Beauty Problem. I am specifically referring to the analysis in “Sleeping Beauty Resolved?”, https://www.lesswrong.com/posts/u7kSTyiWFHxDXrmQT/sleeping-beauty-resolved , in which it is argued that Beauty’s epistemic P(heads) must be 1⁄2 at the instant of awakening, and asymptotically decay to 1⁄3 as she receives sensory data. In this case a Bayesian Beauty can confidently predict in advance that her P(heads) will change, and yet this does not affect her current P(heads).
This doesn’t in practice invalidate BDPU, because that assumes the future evidence is of the form
(SomeVariable = SomeValue)
or
(SomeVariable in SomeSetOfValues),
which is the usual case, whereas the future evidence in Sleeping Beauty is of the very unusual form
(Variable1 = SomeValue OR Variable2 = SomeValue).
Again, see the link and the followup article for details.
Just as many stories contain transhumanist concepts, yet stories rarely turn people into transhumanists.
People adjust their level of investment in a theory based on how invested others are in it.