I’m still mystified by the Born rule

(This post was originally intended as a comment on Adele’s question, but ballooned to the point where it seems worthy of a top-level post. Note that I’m not trying to answer Adele’s (specific fairly-technical) question here. I consider it to be an interesting one, and I have some guesses, but here I’m comentating on how some arguments mentioned within the question relate to the mysteries swirling around the Born rule.)

(Disclaimer: I wrote this post as a kind of intellectual recreation. I may not have the time and enthusiasm to engage with the comments. If you point to a gaping error in my post, I may not reply or fix it. If I think there’s a gaping error in your comment, I may not point it out. You have been warned.)

My current take is that the “problem with the Born rule” is actually a handful of different questions. I’ve listed some below, including some info about my current status wrt each.

Q1. What hypothesis is QM?

In, eg, the theory of Solomonoff induction, a “hypothesis” is some method for generating a stream of sensory data, interpreted as a prediction of what we’ll see. Suppose you know for a fact that reality is some particular state vector in some Hilbert space. How do you get out a stream of sensory data? It’s easy enough to get a single sensory datum — sample a classical state according to the Born probabilities, sample some coordinates, pretend that there’s an eyeball at those coordinates, record what it sees. But once we’ve done that, how do we get our next sense datum?

Or in other words, how do we “condition” a quantum state on our past observations, so that we can sample repeatedly to generate a sequence of observations suitable for linking our theories of induction with our theories of physics?

To state the obvious, a sensory stream generated by just re-sampling predicts that you’re constantly teleporting through the multiverse, and a sensory stream generated by putting a delta spike on the last state you sampled and then evolving that forward for a tick will… not yield good predictions (roughly, it will randomize all momenta).

Current status: I assert that additional machinery is required to turn QM into a hypothesis in the induction-compatible sense — ie, I’d say “the Born rule is not complete (as a rule for generating a hypothesis from a quantum state)”. My guess is that the missing machinery involves something roughly like sampling classical states according to the Born rule and filtering them by how easy it is to read the (remembered) sense history off of them. I suspect that a full resolution of this question requires some mastery of naturalized induction. (I have some more specific models than this that I won’t get into at the moment. Also there are things to say about how this problem looks from the updateless perspective, but I also won’t go into that now.)

ETA: I am not claiming to be the first person to notice this problem. As best I can tell, this problem or something close to it is what physicists refer to as the “measurement problem”. I have not seen anyone clearly frame it as a challenge of segueing a quantum state into an inductor-compatible sensory stream; I’d guess that’s b/​c most physicists don’t (think most other physicsts) natively speak inductor-tongue. I’m aware of the fact that various people have worked on the problem of identifying qualitative branches in a quantum state, and that one explicit motivation for that research is resolving this issue. @interstice linked some below, thanks interstice. That’s not my preferred approach. I still think that that research is cool.

Q2. Why should we believe the Born rule?

For instance, suppose my friend is about to roll a biased quantum die, why should I predict according to the Born-given probabilities?

The obvious answer is “because we checked, and that’s how it is (ie, it’s the simplest explanation of the observed data so far)”.

I suspect this answer is correct, but I am not personally quite willing to consider the case closed on this question, for a handful of reasons:

  • I’m not completely solid on how to twist QM into a full-on sensory stream (see Q1), and I suspect some devils may be lurking in the details, so I’m not yet comfortable flatly declaring “Occam’s razor pins the Born rule down”.

  • There’s an intuitive difference (that may or may not survive philosophical progress) between indexical uncertainty, empirical uncertainty, and logical uncertainty, and it’s not completely obvious that I’m supposed to use induction to manage my indexical uncertainty. For example, if I have seen a million coin tosses in my past, and 23 of them came up heads (with no other detectable pattern), and I have a bona fide guarantee that I’m an emulation running on one of 2^2000000 computers, each of which is halfway through a simulation of me living my life while two million coins get flipped (in literally all combinations), then there’s some intuition that I’m supposed to predict the future coins to be unbiased, in defiance of the observed past frequency. Furthermore, there’s an intuition that QM is putting us in an analogous scenario. (My current bet is that it’s not, and that the aforementioned intuition is deceptive. I have models about precisely where the disanalogy is that I won’t go into at the moment. The point I’m trying to make is that it’s reasonable to think that the Born rule requires justification beyond ‘Occam says’. See also Q4 below.)

  • It’s not clear to me that the traditional induction framework is going to withstand the test of time. For example, the traditional framework has trouble dealing with inductors who live inside the world and have to instantiate their hypotheses physically. And, humans sure are keen to factor their hypotheses into “a world” + “a way of generating my observations from some path through that world’s history”. And, the fact that QM does not naturally beget an observation stream feels like something of a hint (see Q1), and I suspect that a better theory of induction would accommodate QM in a way that the traditional theory doesn’t. Will a better theory of reasoning-while-inside-the-world separate the “world” from the “location therein”, rather than lumping them all into a single sensory stream? If so, might the Born rule end up on the opposite side of some relevant chasm? I suspect not, but I have enough confusion left in this vicinity that I’m not yet comfortable closing the case.

My current status is “best guess: we believe the Born for the usual reason (ie “we checked”), with the caveat that it’s not yet completely clear that the usual reason works in this situation”.

Q3. But… why the Born rule in particular?

Why is the Born rule natural? In other words, from what mathematical viewpoint is this a rule so simple and elegant as to be essentially forced?

Expanding a bit, I observe that there’s a sense in which discrete mathematics feels easier to many humans (see, eg, how human formalizations of continuous math often arise from taking limits or other εδmanship built atop our formalizations for discrete math). Yet, physics makes heavy use of smooth functions and differential equations. And, it seems to me like we’re supposed to stare at this and learn something about which things are “simple” or “elegant” or “cheap” with respect to reality. (See also gauge theory and the sense that it is trying to teach us some lessons about symmetry, etc.)

I think that hunger-for-a-lesson is part of the “but whyyyy” that many people feel when they encounter the Born rule. Like, why are we squaring amplitude? What ever happened to “zero, one, or infinity”? When physics raises something to a power that’s not zero, one, or infinity, there’s probably some vantage point from which this is particularly forced, or simple, or elegant, and if you can find it then it can likely help you predict what sorts of other stuff you’ll see.

Or to put it another way, consider the ‘explanation’ of the Born rule which goes “Eh, you have a complex number and you need a real number, there aren’t that many ways you can do it. Your first guess might be ‘take the magnitude’, your second guess might be ‘take the real component’, your third guess might be ‘multiply it by its own complex conjugate’, and you’ll turn out to be right on the third try. Third try isn’t bad! We know it is so because we checked. What more is there to be explained?”. Observe that there’s a sense in which this explanation feels uncompelling — like, there are a bunch of things wrong with the objection “reality wasn’t made by making a list of possible ways to get a real from a complex number and rolling a die”, but there’s also something to it.

My current status on this question is that it’s significantly reduced — though not completely solved — by the argument in the OP (and the argument that @evhub mentions, and the ignorance+symmetry argument @Charlie Steiner mentions, which I claim all ground out in the same place). In particular, I claim that the aforementioned argument-cluster grounds out the Born rule into the inner product operator, thereby linking the apparently-out-of-the-blue 2 in the Born rule with the same 2 from “L2 norm” and from the Pythagorean theorem. And, like, from my vantage point there still seem to be deep questions here, like “what is the nature of the connection between orthonormality and squaring”, and “is the L2 norm preferred b/​c it’s the only norm that’s invariant under orthonormal change of basis, or is the whole idea of orthonormality somehow baking in the fact that we’re going to square and sqrt everything in sight (and if so how)” etc. etc. I might be willing to consider this one solved in my own book once I can confidently trace that particular 2 all the way back to its maker; I have not yet done so.

For the record, on the axis from “Gentlemen, that is surely true, it is absolutely paradoxical; we cannot understand it, and we don’t know what it means. But we have proved it, and therefore we know it must be the truth” to… whatever the opposite of that is, I tend to find myself pretty far on the “opposite of that” end, ie, I often anticipate finding explanations for logical surprises. In this regard, I find arguments of the form “the Born rule is the only one that satisfies properties X, Y, and Z” fairly uncompelling — those feel to me like proofs that I must believe the Born rule is good, not reasons why it is good. I’m generally much more compelled by arguments of the form “if you meditate on A, B, and C you’ll find that the Correct Way (tm) to visualize the x-ness of (3x, 4y) is with the number (3^2/​5)” or suchlike. Fortunately for me, an argument of the latter variety can often be reversed out of a proof of the former variety. I claim to have done some of that reversing in the case of the Born rule, and while I haven’t fully absorbed the results yet, it seems quite plausible to me that the argument cluster named by Adele/​Evan/​Charlie essentially answers this third question (at least up to, say, some simpler Qs about the naturality of inner products).

Q4. wtf magical reality fluid

What the heck is up with the thing where, not only can we be happening in multiple places, but we can be happening quantitatively more in some of them?

I see this as mostly a question of anthropics, but the Born rule is definitely connected. For instance, you might wish to resolve questions of how-much-you’re-happening by just counting physical copies, but this is tricky to square with the continuous distribution of QM, etc.

Some intuition that’s intended to highlight the remaining confusion: suppose you watch your friend walk into a person-duplicating device. The left copy walks into the left room and grabs a candy bar. The right copy walks into the right room and is just absolutely annihilated by a tangle of whirring blades — screams echo from the chamber, blood spatters against the windows, the whole works. You blink in horror at the left clone as they exit the door eating a candy bar. “What?” they say. “Oh, that. Don’t worry. There’s a dial in the duplicating device that controls how happening each clone is, and the right clone was happening only negligibly — they basically weren’t happening at all”.

Can such a dial exist? Intuition says no. But quantum mechanics says yes! Kind of! With the glaring disanalogy that in QM, you can’t watch the negligibly-happening people get ripped apart — light bouncing off of them cannot hit your retinas, or else their magical-happening-ness would be comparable to yours. Is that essential? How precisely do we go about believing that magical happening-ness dials exist but only when things are “sufficiently non-interacting”? (Where, QM reminds us, this interacting-ness is a continuous quantity that rarely if ever hits zero.) (These questions are intended to gesture at confusion, not necessarily to be answered.)

And it feels like QM is giving us a bunch of hints — ie, if physics turned out to look like a discrete state plus a discrete time evolution rule, we would have been able to say “aha, that’s what happening” and feel content about it, never quite noticing our deeper confusion about this whole “happening-ness” thing. But reality’s not like that. Reality is like a unit vector in an extraordinarily high-dimensional room, casting complex-valued shadows on each wall in the room, and each wall corresponds to a way that everything can be arranged. And if we cast our gaze to the walls in accordance with the degree to which that wall is supporting the overall magnitude of the reality-vector (ie, in accordance with the shadow that the shadow-on-the-wall casts back onto reality, ie in proportion to the shadow times its conjugate, ie in proportion to the squared amplitude of the shadow) then our gaze occasionally falls on arrangements of everything that look kinda like how everything seems to be arranged. And if we cast our gaze using any other rule, we find only noise. And, like, one thing you can do is be like “haha weird” and then figure out how to generate an observation stream from it and chalk it up to “we followed Occam’s razor and this is what we found”. But it seems to me that this is ignoring this great big surprise that reality handed us. This is an unexpected breed of object for reality to be. This shadow-of-a-shadow thing feels like a surprising way for happening-ness to meta-happen. It all feels like a hint, a hint about how our beliefs about what the heck is going on with this whole “existence” thing are built atop false assumptions. And it’s a hint that I can’t yet read.

And… this is somewhat related to the beef I have with measure non-realism. Like, one thing a person can say is “everything is happening; I’m built to optimize what happens in places in accordance with how simple they are; it seems that the simplest way you find me in the logical multiverse is by flitting your gaze along those walls in accordance with the shadow-of-a-shadow and in accordance with some as-yet-unnamed rule about following coherent histories starting from the birth of a particular child; the shadow-of-a-shadow rule is elegant, ridiculously overdetermined by the data, and has no special status relative to any other part of the description of how to find me; what remains to be explained?” And… well, I’m still pretty confused about this whole “stuff is happening” thing. And I’m suspicious of a metaphysics that places physics on the same status as every other mathematical object, b/​c I am not yet sure which of physics and math “comes first”. And yes, that’s a confused question, but that doesn’t make me any less confused about the answer. And, yeah, there are deflationary measure-non-realist replies to these inarticulate gesticulations, but they leave me no less confused. And all the while, reality is sitting there having this counter-intuitive shadow-casting form, and I cannot help but wonder what false assumptions it would reveal, what mysteries it would lay bare, what lessons it would teach about which sorts of things can meta-exist at all, if only I could find my errant intuitions and put them in contact with this surprise.

And, like, there’s a way in which the hypothesis “everything is; we are built to attend to the simple stuff” is a curiosity-stopper — a mental stance that, when adopted, makes it hard to mine a surprise like “reality has the quantum nature” for information about what sort of things can be.

I have a bunch more model than this, and various pet hypotheses, but ultimately my status on this one is “confused”. I expect to remain confused at least until the point where I can understand all these blaring hints.

In sum, there are some ways in which I find the Born rule non-mysterious, and there are also Born-rule-related questions that I remain quite confused about.

With regards to the things I consider non-mysterious, I mostly endorse the following, with some caveats (mostly given in the Q2 section above):

The Born rule is on the same status as the Fourier transform in quantum mechanics — it’s just another equation in the simple description of where to find us. It gets an undeservedly bad rep on account of being just barely on the reality-side of the weird boundary humans draw between “reality” and “my location therein” in their hypotheses, and it has become a poster-child for the counter-intuitive manner in which we are embedded in our reality. Even so, fixing the nature of the rest of reality, once one has fully comprehended the job that the Born rule does, the Born rule is the only intuitively natural tool for its job.

(And, to be clear, I’ve updated in favor of that last sentence in recent times, thanks in part to meditating on the cluster of arguments mentioned by Adele/​Evan/​Charlie.)

With regards to the remaining mystery, there is a sense in which the Born rule is the star in a question that I consider wide-open and interesting, namely “why is ‘trace your eyes across these walls in accordance with the Born rule’ a reasonable way for reality to be?”. I suspect this question is confused, and so I don’t particularly seek its answer, but I do seek mastery of it, and I continue to expect such mastery to pay dividends.