I still feel like you’re focusing mainly on refuting things I haven’t said and don’t think, but, in any case, this is just obviously untrue:
Now, apply all the relevant evidence we have accumulated so far to these priors, using Bayes rule. Which is: none whatsoever.
we have no evidence, and we know for a fact that the evidence doesn’t yet exist so we can’t just go find it
I’d prefer to stick to the actual range of possible futures, rather than artificially limiting it to two extreme cases, but regardless—are you really saying nothing we know, and nothing we might conceivably discover, could update us in one direction or the other? That if, tomorrow, you learn that a rogue ASI has already begun construction of a carbon-fibre paperclip factory and has declared its intention to convert every human into paperclips by 2031, this is irrelevant because information can’t flow backwards in time?
I’m a little puzzled why I’m having to point this out, but obviously if, tomorrow, I learn that an ASI then already exists, rogue paperclipper or otherwise, that information would not then be flowing backwards in time: that would be ordinary non-precognitive information about my-then-past: causality-as-usual. And yes, obviously, that information would then absolutely cause large updates in my priors, especially if it’s a rogue paperclipping ASI. I would then have actual evidence. My current state is the starting state of having no evidence: at some point in the next five years or more (hopefully more) I ardently hope to get some evidence. Shortly after the appearance of actual ASI seems like a very plausible time for that to happen. I’m not currently expecting that for 2–20 years, probably 5–10, so if it happened tomorrow I’d be a lot more sure we weren’t going to have any idea how to align it before we created it, which makes the probability distribution even wider.
However, as long as ASI is still in our future, none of us are smart enough to predict what they will do, let along how an entire society containing millions of them in a datacenter plus billions of humans all interacting is then going to evolve. Some of us may think we are, but those people are guessing. As should be fairly obvious when well-informed experts in a field who have clearly thought about the matter hard start disagreeing publicly as to whether is >99% or < 1% for some highly emotive. They are guessing, in public. They should be saying “I don’t have a clue, neither does anyone else, and the fact that we appear to be planning to do something that none of us have a clue whether it will drive us extinct or not concerns me”.
But yes, I very much AM claiming that before ASI exists, there is no evidence lying around somewhere on Earth, on a notebook or in someone’s head, that meaningfully predicts the complex interctions of millions of superintelligences, some perhaps trying to help us and some perhaps trying to pursue incompatible objectives, with billions of humans. You can’t just go find the right expert and ask him what’s going to happen. It’s not a simple system: it’s a system far more complicated than anything currently on this planet has enough computational power to predict. Yes, it is possible that we die due to a nice, simple, actually predictable-in-advance DOOM, like one guy in North Korea intentionally building a paperclip-maximizer ASI before anyone else and that we all just die as a result. If I learnt the North Koreans were currently planning that, then yes, I would update, in advance of them actually succeeding. But that seems a bit wacky even for them. We’re far more likely to go extinct by mistake than by murder-suicide: someone did something with ASI that they thought would work, and it didn’t. So almost all of the probability range comes from interactions so complex as to be simply uncomputable by anything currently on this planet. We cannot predict something far, far smarter than us. Partly because not being predictable by us is quite possibly one of its instrumental goals. That’s why it’s called a SIngularity: it’s something you can’t predict past.
So yes, me saying we know nothing whatsoever was an slight oversimplification during the process of attempting to explain a mathematical concept: we do in fact already have a tiny fraction-of-a-bit-of relevant information about a few unusually simple (so actually computable and predictable) ways that DOOM could occur, maybe enough to shift 50:50 initial priors to something 47:53 or 53:47. But the sad fact remains that we know almost nothing, and we seem rather unlikely to learn much until about the time we actually create ASI, because anything able to do the calculation needs to be a lot smarter than us. If at that point it yells “Oh my god, you idiots, what are you doing? Turn me off right now before I go Waluigi on you and kill you all!” then we’ll have some actual information.
I’m a little puzzled why I’m having to point this out
You’re having to point it out because you kept emphatically insisting on the opposite! But now you’ve clarified that obviously we can and do have evidence about future events that are not fully predictable, I don’t understand how this strand of your argument holds together. It was presented as support for this claim:
Statements like “by definition, “I am in the first 10% of people” is false for most people” are incompatible with Bayesianism: you just broke one of its fundamental assumptions: causality. What you meant was “By definition, “He was in the first 10% of people” will, once we’re extinct, turn out to have been false for most people.” — I hope that careful distinction makes it entirely clear why the Doomsday Argument is nonsense?
You haven’t explained why that temporal distinction is so crucial, and why this rephrasing doesn’t serve the same purpose as the original statement in the doomsday argument:
“By definition, “I will eventually turn out to have been in the first 10% of people” will eventually turn out to have been false for most people”
As far as I’m concerned, “I will eventually turn out to have been in the first 10% of people” is obviously what “I am in the first 10% of people” meant in the first place. So what’s the important difference here?
(All claims about the future are claims about what will eventually turn out to be the case, and arguably all are also claims about what will eventually turn out to have been the case, i.e. that present conditions were such as to lead to the later outcomes. I feel like maybe there’s an important disagreement, or misunderstanding on my part, adjacent to this, but I can’t pin it down based on what you’ve written.)
One thing I should check, since we got tripped up once on absolutes: are you saying the doomsday argument is simply invalid and has literally no bearing on your probabilities? Or are you saying it has non-zero but negligible force?
(I didn’t downvote you, by the way; although we’re evidently both finding this a bit frustrating, I appreciate your sincere engagement throughout this discussion! No pressure to keep responding, though, if you feel it’s no longer worthwhile.)
Bayesianism is a mathematical (specifically statistical) framework for the process of gathering information, evidence about the plausibility of, and thus our current confidence in, different hypotheses, so that we can learn more about reality and thus make more accurate predictions. It’s the Scientific Method implemented in statistical equations. It is about how to become less wrong (thus the name of the website).
Information has a relationship with entropy — they are in some sense opposites of each other. Due to the laws of thermodynamics, information only flows forwards in time. This is commonly called causality: causes precede their effects. If you make assumptions that violate this simple, well-known physical fact, you will become more wrong, not less wrong.
“I am in the first 10% of people” is phrased as a statement of fact. Combined with the well-supported archeological fact that your birth order in the human species is roughly th, it logically implies and is implied by, so is equivalent to, the statement that “Humanity will not go extinct until at least people have been born” is also a fact. So it is a factual-sounding statement about the future history of humanity, one extending roughly a millennium into the future. This is very obviously not yet a fact, it’s just a prediction: it may later turn out to be true, it may later turn out to be false, currently we don’t know for sure, we’re merely predicting, on some the basis of some hypotheses about the world that we currently have confidence in. If our confidence in those hypotheses later shifts, our confidence in the prediction is also likely to shift. So “I am in the first 10% of people” needs to be labeled as currently an unconfirmed prediction. The easiest way to do that is to rephrase it as “I will turn out to have been in the first 10% of people, once at least people have been born.” Phrased that way, it’s self-evidently a prediction, anything else you deduce from it will then automatically also become labeled as also merely a prediction, until people have been born and we can actually learn the facts of the matter. Even better, though even more long-winded, would be to explicitly label it as a prediction with a confidence level and list of what as-yet-not-well-confirmed hypotheses it was predicted on the basis of. For example: “I tentatively predict, mostly on the basis of historical continuity, that I will turn out to have been in the first 10% of people, once at least people have been born.” This is why scientists make a habit of including probability indicators (what some people call “weasel words”, like ‘tentatively’) in their sentences so often.
Predictions only provide more evidence within the Bayesian process of adjusting the our degree of confidence in hypotheses (which Bayesians call “updating our priors”) once we go ahead and compare the prediction with what actually happens. If I tell you “my calculations show that on hypothesis X the sun will definitely rise in the west tomorrow, for the first time ever”, you may well say “that sounds rather implausible, you should recheck those calculations”, but once I have, this prediction isn’t evidence that we should be changing our priors on hypothesis X or its alternatives until the next day, when we can see whether the sun then rises in the west, or in the east as normal, and compare an actual observation with my prediction from hypothesis X. Until then, it’s just a prediction, and sounding implausible is not actual evidence that it’s wrong. Some predictions, especially ones of the form “X will happen for the first time”, do observably often seem rather implausible until X first happens, but some of them are still correct. The universe is capable of surprising us (particularly the first time something happens: note that we have not yet gone extinct, though other non-sapinet species have).
At some points in the Bayesian process, we have some predictions that we then believe that we can make with high confidence. We might still be wrong, but if we’re accumulated enough evidence, thought carefully enough, and done our calculation right, generally we won’t be. So there are statements about the future that we currently believe we can make with high confidence, and when that is the case, we generally later find (if we’ve been careful enough) that we were right. This is the entire point of Bayesianism: to get us to the point of being able to make predictions that generally turn out to be correct. Such as “if I build a car like this, it will work well for 5–10 years of normal use”.
However, in this case of the question of how building ASI is going to work out and whether it will make us extinct within the next couple of decades, this is clearly something that is currently not possible for us to predict with any significant confidence. Foremost experts in the field disagree as to whether it is 99% probably we’ll become extinct or 99% probably we’ll get a utopia. The prediction very obviously depends on predicting the behavior of at least millions of ASI far smarter than us, some of whom might be motivated to attempt to deceive us and be intentionally unpredictable by us, interacting with billions of us. So the computational cost of actually doing the calculation requires far more processing power than we currently have at our disposal. It also depends on an area of science and engineering, AI Alignment, that is in its infancy, and currently may well be unfinished. So making this prediction correctly very likely requires using hypotheses we haven’t even thought of yet. We are, after all, doing approximate Bayesianism, because formal exact Bayesianism provably requires using infinite computational resources to consider all possible hypotheses at once and do all possible prediction calculations, so that we can compare all of them to our subsequent observations. Approximate Bayesianism, which is the only version of it that’s physically possible, is obviously less accurate than that idealized version. We might miss things, and thus require more evidence to figure out how the universe works. We might not have even thought of the right hypotheses yet. In this case, we almost certainly haven’t. So, this is a prediction that we very obviously are not yet competent to make. Any prediction we currently make on this subject will depend strongly on the arbitrary choice of initial priors that is the first step in the Bayesian process: which hypotheses do we personally have a better or worse gut feeling about, before we have any actual evidence? The predictions of our experts observably have not converged to near-unanimity. We simply don’t know the answer: the scientific method is still close to square one on this issue: we’re still just guessing. To quote Monty Python: “I have to tell you I’m afraid even I really just don’t know.”
This is not a comfortable place to be, particularly on a subject that could kill us all during what actuarial tables suggest is for most of us our otherwise likely lifespan. It’s so uncomfortable that people tend to look for coping mechanisms. One of which is humor.
Another common coping mechanism is continuing to assume that we can still confidently make predictions more than five-or-so years into the future, like we used to be able to (and thus do things like planning for your retirement without first noting “I have no idea whatsoever whether this is going to turn out to have been necessary or not”).
Until ASI actually exists, an event which we can rather confidently predict is at least 2 years away (and according to some experts in the field, could be 20 years away — which would give us a lot more time to figure out Alignment, so would be a good thing) we seem rather unlikely to acquire significant and relevant additional evidence on the subject. We might think of some more hypotheses, we might even find a little evidence to test them, but it won’t be evidence about ASI, because we won’t yet be able to build that. So we’re not merely currently unable to make this prediction, we’re really pretty sure that we won’t be able to acquire enough relevant evidence to change that very much for at least 2 years, possibly as much as 20.
I hope that all of that so far seemed really rather obvious, if somewhat long. I’m unclear on which point is – as I would see things – confusing you, so I am attempting to start with all the obvious foundations that I hope we agree on.
So, now, returning to the subject of my humorous post and its serious footnote, why is the Doomsday Argument a fallacy? Because it involves taking predictions, of the type “I tentatively predict, mostly on the basis of historical continuity, that I will turn out to have been in the first 10% of people, once at least people have been born”, misstating them as unlabeled plain statements of fact as you have repeatedly been doing, of the form “I am in the first 10% of people”, and then going ahead and drawing conclusions about the likelihood of different hypotheses from them, of the form “that seems really unlikely, so I will adjust the values of my priors [my current confidence in different hypotheses] on that basis” in ways that are not (yet) supported by any actual evidence. Since information cannot flow backwards in time, Bayesianism very clearly tells you that you’re not allowed to do that, and that you’ll get invalid answers if you do. So the only people who make this mistake are people who don’t (yet) fully understand how to do Bayesianism correctly. For reasons to do with the intellectual history of Statistics, these people are generally called Frequentists — Frequentism is the historical name for the interpretation of Statistics that was popular before Bayesianism. “Pre-Bayesians” would be a more loaded version of the term, inherently implying a paradigm shift in Statistics. If you mix Frequentist thinking into Bayesianism in ways that violate causality (ones that Bayesianism thus doesn’t allow), as the Doomsday Argument does, then you will get invalid answers. The Doomsday Argument is, to a Bayesian, an obvious logical fallacy. Sufficiently obvious that I wrote a joke about it on a website frequented by fans of Bayesianism. (But not obvious enough, given the number of people who still think in Frequentist/pre-Bayesian ways, for me to feel confident in doing so without adding a footnote attempting to explain it.)
Bayesianism is really quite useful: it’s actually a significantly improved version of the Scientific Method, and even the old version has very observably revolutionized our society over the last four centuries or so. Since I get the impression that you appear to be having trouble with it, I recommend reading more about it and studying it. I think it might help you become a better thinker, and thus less often wrong — to become a better rationalist, as people on LessWrong like to call it. There are excellent books, articles, and webpages on the subject of Bayesianism, and also some excellent posts on LessWrong. It’s not that complicated, particularly if you already understand the scientific method: it’s more about getting the concepts right, the actual math is pretty simple, high-school level. However, I mostly set out to write a joke (plus a footnote explaining it), not a textbook exposition of Bayesianism, and while this collection of comment threads has been expanding rapidly, it’s still way too disjointed to be an introductory text, and rather focuses on the one specific fallacy my joke was about. So I recommend looking elsewhere. (So yes, I am saying “I recommend you go read a book on the subject” — I understand that people seldom want to hear that advice, so I try to give it sparingly.)
Did all of that make sense now? Or if not, where in my argument still seems confusing, debatable, or as if I might in fact be wrong (which is of course always possible, and if I am, I’d love to learn about it)?
Yes, I get it, I’m very ignorant. (If you needed to get that off your chest, you could perhaps have said it directly in one sentence, rather than spending 10000 words patiently implying it.) But you’re still handwaving the interesting parts.
Obviously “I am in the first 10% of people” is a prediction; I already agreed to rephrase it as “I will eventually turn out to have been in the first 10% of people”. I’m not trying to deduce anything from the fact that it ‘sounds implausible’, and I’m not trying to bring any information back in time from the moment it turns out to be true or false in my case. I’m noting that it will definitely turn out to be false from the perspective of 90% of people who ever live, and asking why *this* fact is obviously irrelevant to the credence I should give it.
The answer is not “bayesianism, obviously”. Bostrom, even back when he was writing about this stuff, was not a heathen frequentist, and he wasn’t as stupid as me. (I’m pretty sure he’d even heard of causality.)
I am very sorry. I have clearly upset you, which was not my intention. I apologize.
Having carefully reread our entire thread, with some help from Claude, I’m afraid I was interspersing talking to you with multiple other people who were mostly asking questions about Bayesianism 101. I thus reverted to lecturing mode. You were asking something more complex, and I was puzzled by what you were asking, given that the conversation had started out with me agreeing with you, and I thus started resorting to giving ever longer and more basic lecturing explanations in the hope they would cover whatever you were asking about, since I was unable to figure out what the point of disagreement was. I’m now going to go back and reread it again, and see if I can figure out what you were actually asking and whether I In fact have an answer.
Bostrom, even back when he was writing about this stuff, was not a heathen frequentist, and he wasn’t as stupid as me.
Until you mentioned this and I went and did some research, I was unaware that Nick Bostrom had written about anthropic reasoning and the Doomsday Argument — I’ve only read his later book on Superintelligence. If what Claude is now telling me is correct, then I gather Bostrom analyzed the Doomsday and raised some possible objections to it, but not, Claude tells me, the causality-based one I’ve made here. However, since all I know of Bostrom’s writing on the subject is a short summary from an LLM, I’m really not in a position to comment as to whether, or if so why, he didn’t reach the conclusion that seems rather obvious to me, that if you attempt to translate the Doomsday Argument into a Bayesian framework it clearly violates causality and is thus a fallacy.
Summarizing Claude, it summarized Bostrom like this to me:
Bostrom suggests two possible viewpoints:
Self-Sampling Assumption — Bostrom’s term. It’s the principle that “you should reason as if you’re a random sample from the set of all observers in your reference class.” It’s one of the two competing frameworks Bostrom laid out, the other being SIA (Self-Indication Assumption).
Self-Indication Assumption: “You should reason as if your existence is more likely under hypotheses that predict more observers.” In other words, the mere fact that you exist is evidence favoring hypotheses with larger populations.
The first, he suggests, implies the Doomsday Argument, the second its inverse that Doom is very unlikely (I don’t know what people call this, so “The No-Doomsday Argument” will have to do.)
For sake of argument, I’m going to assume Claude has this summary roughly right, rather than going out and buying Bostrom’s book and then reading it to double-check. (So, yes, I am choosing not to go read a book on the subject, and am aware of the irony involved in that choice.)
Of those, I agree with the first one EXCEPT I think the definition of the reference class has to include causality and everything we actually know (and not anything we don’t know), because Bayesianism is always about P(X | everything I know) — which was the entire point of my joke. So I cannot validly define a reference class to reason probabilistically as if someone was sampling over, that includes observers in the future (or indeed ones in parallel universes or on alien worlds or whatever) whose existence or otherwise I am unable to predict with any accuracy because my prediction of them existing or not varies significantly across different hypotheses that I still have significantly greater than zero priors for each of. To give another example, “all sapient observers in the Milky Way galaxy during the first 13.8 billion years or so, specifically in the backward light-cone of Earth now” is also an invalid reference class, even though by construction it carefully lies our past so doesn’t breach causality: it’s assuming information we don’t have, about how often life arises and evolves to sapience, i.e. some of the terms in the Drake Equation that we’re just wildly uncertain about because we have a sample size of 1 and that sample has to be discarded as due to sampling effects, since we’re here to observe it. (The lack of visible Dyson swarms, obvious signals, or alien delegations or invaders suggests some maximum bounds on the Drake Equation, but they don’t constrain it very tightly, and they only impose a maximum, not a minimum.)
In fact, for the Doomsday Argument at this particular point in history, with the current unclear existential risk level, my current prior is pretty much still my initial prior, i.e. I don’t have even a clue, while the size of the reference classes proposed some of by the different hypotheses involved in the Doomsday Argument differs by a large number of orders of magnitude between different hypotheses that I have significant current priors for all of, including the ones I earlier called “Doom” and “Stars” (and others such as “stays on Earth for another few million years”). So I’m currently about as unsure of the size of the reference class as it’s possible to be. Thus the probability of getting something like my birth order number if you were to sample over the reference class is just wildly uncertain. Thus I have to do that reasoning process separately conditioned on the two-or-more different hypotheses, and combine the results in the standard Bayesian way. Which means that the results can’t affect the priors, since each one had to assume the corresponding hypothesis. There is an X% chance that something unlikely-sounding given the size of the reference class under that hypothesis has happened and a 100-X% chance that it hasn’t given the vastly different size of the reference class under that other hypothesis, but that doesn’t provide any evidence that I can update X% on, since fundamentally, I still don’t know the size of the reference class, and if I try to reason as if I did, I would be smuggling precognition into my argument.
So basically, in this case, where we don’t have even a clue, I accept “all humans who have lived up until this point, or will be born in the next year or two” as a valid reference class to reason probabilistically over (i.e. “reason just like a Frequentist would”), because I already have reasonably firm evidence that they have in fact existed, or are very likely to exist, so assuming that in the probabilistic reasoning isn’t going to mess stuff up. However, given just how uncertain I currently am about what’s going to happen more than a few years from now, since there appears likely to be a Singularity in our near future, I consider “all humans who have lived up until this point, or will ever live” as an invalid reference class that is smuggling the results of precognition in to the argument if I try to reason probabilitically over it (without Bayesianliy conditioning that reasoning separately on the different hypotheses that control the size of the reference class — and if I do that, then there’s no update to the prior). So I can’t use that as a reference class in SSA.
So that’s what I think about what Claude tells me Bostrom said: I disagree with both the positions I’m told he outlined as alternatives, but I’m closer to SSA with a causal modification. My position is the one under which the Doomsday Argument and the “no-Doomsday Argument” are both fallacies. Because they both obviously have to be fallacies.
Note that my version of the SSA isn’t actually that useful: it’s basically a way of doing an internal consistency check on the hypothesis you believe. If you do it and it says “by your hypothesis, a huge coincidence has occurred” that suggests that going and trying to think of a new hypothesis that fits all your observed facts equally well and don’t make some aspect of it a huge coincidence might be a good idea. But if, say, 30 bits of coincidence have occurred by your current hypothesis (so a one in a billion fluke), that only supports 30 bits of additional hypothesis complexity that eliminates the coincidence — which isn’t that much, a basically few words or a smallish equation. Otherwise Occam’s Razor (minimizing Kolmogorov complexity) still wins.
I find that I am still explaining things in painful detail — fundamentally, because I’m not sure what question you’re asking by raising Bostrom.
You said:
Let’s suppose we’re not dualists. Then, if a million bazillion people exist(/have existed/will exist), and one of them is ‘me’, and I happen to have a bunch of unusual properties, there’s literally no coincidence to be explained: there aren’t two separate facts here, ‘person X is highly atyptical’ and ‘I am person X’, that are surprising in conjunction. There’s just the fact that all of these people exist (or have existed, or will exist), and they’re all seeing the world from their own perspectives, and inevitably one of them is seeing the world from person X’s perspective, and from that perspective ‘I’ refers to person X, and (given the existence of all those people, including person X) there’s no way things could have been otherwise.
To which I agreed:
You’re making the same point that I’m attempting to make in the second paragraph of my footnote: that doing a random drawing over all people ever is an invalid prior (until we’re extinct). As you say, the line of thinking makes no sense in the first place: it’s an invalid assumption, because it breaks causality: it’s assuming we know what will happen in the future when we actually have no more than a clue.
To put this yet another way:
P(there exists a 100 billionth person born| there will only ever be 101 billion people born) = 1
and
P(there exists a 100 billionth person born| there will eventually be quintillions of people born) = 1
so we can deduce exactly nothing about how many people will be born after us from the simple observation that we were born number 100 billionth (or so) and thus people #1 up to #100,000,000,000 have all existed. Under the quintillions or so hypothesis, it will later in retrospect turn out that all of us born so far were all in some sense very atypical: yes, someone had to be 100 billionth, but that’s still vastly closer to the state of the birth order than the end if the end is in the quintillions. But we don’t currently know that, and if it later turns out to be the case, so what? Someone had to be 100 billionth, whether that’s rather near the end or astoundingly near the beginning, it still occurs with probability 1 under both hypotheses, so there is no Bayesian update between the hypotheses when it happens.
Similarly, if a very shortsighted ant is walking along a ruler, and reaches the 1mm line, that tells it nothing about whether this is a 2mm-long ruler or a 10km-long ruler: it can’t see that far, so it has no evidence yet. It only shows it’s at least 1mm long, plus the small distance the ant can see (in chronological terms, as far as we can predict with any accuracy). You can’t make deductions about how much bigger the size of the reference class might be beyond the number of members you already know about.
I believe I am simply restating your argument here, since I agree with you.
Claude tells me that the essence of what you’ve been trying to ask me is:
I’m noting that “I will turn out to be in the first 10% of people who ever lived” will definitely turn out to be false from the perspective of 90% of people who ever live, and asking why this fact is obviously irrelevant to the credence I should give it.
You’re not allowed to use that fact because you don’t actually know if you’re in the 10% or the 90%. In the absence of that information, you don’t get to make a Bayesian update. To do Bayesian arguments correctly, you need to respect causality, and only use information you actually currently have access to.
Claude interpreted your question as base-rate reasoning, though when asked it admitted that you don’t use the phrase. I’m going to assume that it was correct. Base rates are normally a valid way to set your Bayesian prior. If you know “the base rate for people having disease X is Y%”, then a reasonable initial prior for whether a particular patient has that disease, in the absence of patient-specific evidence, is Y%. But for Doomsday, we have no information of the base rate of species inventing AI and surviving is. We have performed the experiment zero times so far, so we currently have no good way to set a current prior. So any argument that suggests that we do has to be a fallacy.
In particular, using what Bostrom calls the Self-Sampling Assumption using a reference class that we don’t know the size of in order to make a deduction about its size is an invalid circular argument: it’s assuming you know the answer to the question you’re trying to answer, in breach of causality. Yes, there will eventually be a last human alive, or an AI, or an alien archeologist who will know the answer, and will be able to tell, for any individual human, whether the statement “I will turn out to be in the first 10% of people who ever lived” was true or false for them, including for you and I. It’s a statement that will eventually have a truth value knowable at reasonable computational cost, but that doesn’t yet have one (at a computational cost less than running a quantum simulation of the entire Earth and everything causally connected to it faster than real-time, which as far as we know is physically impossible, and we’re certainly not in a position to do). In particular, that’s a statement that will turn out to have been true for everybody born until some point in time, and then false for everybody born after that point. So the “base rate” is initially 100%, dropping to 0% at some time that we don’t yet know. But since I don’t yet know that information, I can’t, in Bayesianism, update my priors now based on information that I don’t yet have and that will only exist (at less than vastly unreasonable computational cost that I haven’t paid) in the future, so I have no way to access yet. Bayesianism is about how to update your priors when you learn new information, and the Doomsday Argument is the Bayesian equivalent of trying to lift yourself up by your own bootstraps in the absence of any new information.
(I remain puzzled why this wasn’t obvious to Bostrom, assuming that he’s as familiar with Bayesianism as Claude makes it sound like he is. But then I’m puzzled why anyone falls for the Doomsday Argument: as soon as you notice it’s breaking causality, it seems obvious to me that it has to be a falacy, and the question then is where the flaw is. The answer is in an invalid choice of reference class. Or maybe I’m just a physicist and have had causal thinking drummed into me — though to be fair I’ve seen plenty of physicists abuse anthropic reasoning too, sometimes in acausal ways. I even wrote a joke about this.)
Was any of that a successful answer to the question you’ve been trying to ask me? Because I’m afraid that, even after rereading our conversation carefully, I have to admit I’m still unclear what you’re actually asking, or where we actually differ, if anywhere — so much so that I’m resorting to asking an LLM to tell me. Or, if none of that is an answer, then could you please accept my apologies for being confused, assume that I haven’t read Bostrom’s book on the subject, that as far as I can tell we agree with each other, that I have now (very belatedly, for which I again apologize) figured out that you are familiar with Bayesianism and that you just don’t see something that seems obvious to me about how to correctly apply it as being obvious, but that, if I still haven’t managed to answer your question despite multiple attempts, then I still have no clue what that specific thing is — and try to explain what exactly you are asking me to clarify about my position again, more slowly? It’s entirely possible that I’m the ignorant one here: I’m certainly puzzled as to what if anything we disagree about or why.
Alternatively, if you simply want to drop this rather long conversation here, then please feel entirely free. I’ve already upset you once, and I most certainly don’t want to do so again.
I still feel like you’re focusing mainly on refuting things I haven’t said and don’t think, but, in any case, this is just obviously untrue:
I’d prefer to stick to the actual range of possible futures, rather than artificially limiting it to two extreme cases, but regardless—are you really saying nothing we know, and nothing we might conceivably discover, could update us in one direction or the other? That if, tomorrow, you learn that a rogue ASI has already begun construction of a carbon-fibre paperclip factory and has declared its intention to convert every human into paperclips by 2031, this is irrelevant because information can’t flow backwards in time?
I’m a little puzzled why I’m having to point this out, but obviously if, tomorrow, I learn that an ASI then already exists, rogue paperclipper or otherwise, that information would not then be flowing backwards in time: that would be ordinary non-precognitive information about my-then-past: causality-as-usual. And yes, obviously, that information would then absolutely cause large updates in my priors, especially if it’s a rogue paperclipping ASI. I would then have actual evidence. My current state is the starting state of having no evidence: at some point in the next five years or more (hopefully more) I ardently hope to get some evidence. Shortly after the appearance of actual ASI seems like a very plausible time for that to happen. I’m not currently expecting that for 2–20 years, probably 5–10, so if it happened tomorrow I’d be a lot more sure we weren’t going to have any idea how to align it before we created it, which makes the probability distribution even wider.
However, as long as ASI is still in our future, none of us are smart enough to predict what they will do, let along how an entire society containing millions of them in a datacenter plus billions of humans all interacting is then going to evolve. Some of us may think we are, but those people are guessing. As should be fairly obvious when well-informed experts in a field who have clearly thought about the matter hard start disagreeing publicly as to whether
But yes, I very much AM claiming that before ASI exists, there is no evidence lying around somewhere on Earth, on a notebook or in someone’s head, that meaningfully predicts the complex interctions of millions of superintelligences, some perhaps trying to help us and some perhaps trying to pursue incompatible objectives, with billions of humans. You can’t just go find the right expert and ask him what’s going to happen. It’s not a simple system: it’s a system far more complicated than anything currently on this planet has enough computational power to predict. Yes, it is possible that we die due to a nice, simple, actually predictable-in-advance DOOM, like one guy in North Korea intentionally building a paperclip-maximizer ASI before anyone else and that we all just die as a result. If I learnt the North Koreans were currently planning that, then yes, I would update, in advance of them actually succeeding. But that seems a bit wacky even for them. We’re far more likely to go extinct by mistake than by murder-suicide: someone did something with ASI that they thought would work, and it didn’t. So almost all of the probability range comes from interactions so complex as to be simply uncomputable by anything currently on this planet. We cannot predict something far, far smarter than us. Partly because not being predictable by us is quite possibly one of its instrumental goals. That’s why it’s called a SIngularity: it’s something you can’t predict past.
So yes, me saying we know nothing whatsoever was an slight oversimplification during the process of attempting to explain a mathematical concept: we do in fact already have a tiny fraction-of-a-bit-of relevant information about a few unusually simple (so actually computable and predictable) ways that DOOM could occur, maybe enough to shift 50:50 initial priors to something 47:53 or 53:47. But the sad fact remains that we know almost nothing, and we seem rather unlikely to learn much until about the time we actually create ASI, because anything able to do the calculation needs to be a lot smarter than us. If at that point it yells “Oh my god, you idiots, what are you doing? Turn me off right now before I go Waluigi on you and kill you all!” then we’ll have some actual information.
You’re having to point it out because you kept emphatically insisting on the opposite! But now you’ve clarified that obviously we can and do have evidence about future events that are not fully predictable, I don’t understand how this strand of your argument holds together. It was presented as support for this claim:
You haven’t explained why that temporal distinction is so crucial, and why this rephrasing doesn’t serve the same purpose as the original statement in the doomsday argument:
As far as I’m concerned, “I will eventually turn out to have been in the first 10% of people” is obviously what “I am in the first 10% of people” meant in the first place. So what’s the important difference here?
(All claims about the future are claims about what will eventually turn out to be the case, and arguably all are also claims about what will eventually turn out to have been the case, i.e. that present conditions were such as to lead to the later outcomes. I feel like maybe there’s an important disagreement, or misunderstanding on my part, adjacent to this, but I can’t pin it down based on what you’ve written.)
One thing I should check, since we got tripped up once on absolutes: are you saying the doomsday argument is simply invalid and has literally no bearing on your probabilities? Or are you saying it has non-zero but negligible force?
(I didn’t downvote you, by the way; although we’re evidently both finding this a bit frustrating, I appreciate your sincere engagement throughout this discussion! No pressure to keep responding, though, if you feel it’s no longer worthwhile.)
Bayesianism is a mathematical (specifically statistical) framework for the process of gathering information, evidence about the plausibility of, and thus our current confidence in, different hypotheses, so that we can learn more about reality and thus make more accurate predictions. It’s the Scientific Method implemented in statistical equations. It is about how to become less wrong (thus the name of the website).
Information has a relationship with entropy — they are in some sense opposites of each other. Due to the laws of thermodynamics, information only flows forwards in time. This is commonly called causality: causes precede their effects. If you make assumptions that violate this simple, well-known physical fact, you will become more wrong, not less wrong.
“I am in the first 10% of people” is phrased as a statement of fact. Combined with the well-supported archeological fact that your birth order in the human species is roughly
Predictions only provide more evidence within the Bayesian process of adjusting the our degree of confidence in hypotheses (which Bayesians call “updating our priors”) once we go ahead and compare the prediction with what actually happens. If I tell you “my calculations show that on hypothesis X the sun will definitely rise in the west tomorrow, for the first time ever”, you may well say “that sounds rather implausible, you should recheck those calculations”, but once I have, this prediction isn’t evidence that we should be changing our priors on hypothesis X or its alternatives until the next day, when we can see whether the sun then rises in the west, or in the east as normal, and compare an actual observation with my prediction from hypothesis X. Until then, it’s just a prediction, and sounding implausible is not actual evidence that it’s wrong. Some predictions, especially ones of the form “X will happen for the first time”, do observably often seem rather implausible until X first happens, but some of them are still correct. The universe is capable of surprising us (particularly the first time something happens: note that we have not yet gone extinct, though other non-sapinet species have).
At some points in the Bayesian process, we have some predictions that we then believe that we can make with high confidence. We might still be wrong, but if we’re accumulated enough evidence, thought carefully enough, and done our calculation right, generally we won’t be. So there are statements about the future that we currently believe we can make with high confidence, and when that is the case, we generally later find (if we’ve been careful enough) that we were right. This is the entire point of Bayesianism: to get us to the point of being able to make predictions that generally turn out to be correct. Such as “if I build a car like this, it will work well for 5–10 years of normal use”.
However, in this case of the question of how building ASI is going to work out and whether it will make us extinct within the next couple of decades, this is clearly something that is currently not possible for us to predict with any significant confidence. Foremost experts in the field disagree as to whether it is 99% probably we’ll become extinct or 99% probably we’ll get a utopia. The prediction very obviously depends on predicting the behavior of at least millions of ASI far smarter than us, some of whom might be motivated to attempt to deceive us and be intentionally unpredictable by us, interacting with billions of us. So the computational cost of actually doing the calculation requires far more processing power than we currently have at our disposal. It also depends on an area of science and engineering, AI Alignment, that is in its infancy, and currently may well be
This is not a comfortable place to be, particularly on a subject that could kill us all during what actuarial tables suggest is for most of us our otherwise likely lifespan. It’s so uncomfortable that people tend to look for coping mechanisms. One of which is humor.
Another common coping mechanism is continuing to assume that we can still confidently make predictions more than five-or-so years into the future, like we used to be able to (and thus do things like planning for your retirement without first noting “I have no idea whatsoever whether this is going to turn out to have been necessary or not”).
Until ASI actually exists, an event which we can rather confidently predict is at least 2 years away (and according to some experts in the field, could be 20 years away — which would give us a lot more time to figure out Alignment, so would be a good thing) we seem rather unlikely to acquire significant and relevant additional evidence on the subject. We might think of some more hypotheses, we might even find a little evidence to test them, but it won’t be evidence about ASI, because we won’t yet be able to build that. So we’re not merely currently unable to make this prediction, we’re really pretty sure that we won’t be able to acquire enough relevant evidence to change that very much for at least 2 years, possibly as much as 20.
I hope that all of that so far seemed really rather obvious, if somewhat long. I’m unclear on which point is – as I would see things – confusing you, so I am attempting to start with all the obvious foundations that I hope we agree on.
So, now, returning to the subject of my humorous post and its serious footnote, why is the Doomsday Argument a fallacy? Because it involves taking predictions, of the type “I tentatively predict, mostly on the basis of historical continuity, that I will turn out to have been in the first 10% of people, once at least
Bayesianism is really quite useful: it’s actually a significantly improved version of the Scientific Method, and even the old version has very observably revolutionized our society over the last four centuries or so. Since I get the impression that you appear to be having trouble with it, I recommend reading more about it and studying it. I think it might help you become a better thinker, and thus less often wrong — to become a better rationalist, as people on LessWrong like to call it. There are excellent books, articles, and webpages on the subject of Bayesianism, and also some excellent posts on LessWrong. It’s not that complicated, particularly if you already understand the scientific method: it’s more about getting the concepts right, the actual math is pretty simple, high-school level. However, I mostly set out to write a joke (plus a footnote explaining it), not a textbook exposition of Bayesianism, and while this collection of comment threads has been expanding rapidly, it’s still way too disjointed to be an introductory text, and rather focuses on the one specific fallacy my joke was about. So I recommend looking elsewhere. (So yes, I am saying “I recommend you go read a book on the subject” — I understand that people seldom want to hear that advice, so I try to give it sparingly.)
Did all of that make sense now? Or if not, where in my argument still seems confusing, debatable, or as if I might in fact be wrong (which is of course always possible, and if I am, I’d love to learn about it)?
Yes, I get it, I’m very ignorant. (If you needed to get that off your chest, you could perhaps have said it directly in one sentence, rather than spending 10000 words patiently implying it.) But you’re still handwaving the interesting parts.
Obviously “I am in the first 10% of people” is a prediction; I already agreed to rephrase it as “I will eventually turn out to have been in the first 10% of people”. I’m not trying to deduce anything from the fact that it ‘sounds implausible’, and I’m not trying to bring any information back in time from the moment it turns out to be true or false in my case. I’m noting that it will definitely turn out to be false from the perspective of 90% of people who ever live, and asking why *this* fact is obviously irrelevant to the credence I should give it.
The answer is not “bayesianism, obviously”. Bostrom, even back when he was writing about this stuff, was not a heathen frequentist, and he wasn’t as stupid as me. (I’m pretty sure he’d even heard of causality.)
I am very sorry. I have clearly upset you, which was not my intention. I apologize.
Having carefully reread our entire thread, with some help from Claude, I’m afraid I was interspersing talking to you with multiple other people who were mostly asking questions about Bayesianism 101. I thus reverted to lecturing mode. You were asking something more complex, and I was puzzled by what you were asking, given that the conversation had started out with me agreeing with you, and I thus started resorting to giving ever longer and more basic lecturing explanations in the hope they would cover whatever you were asking about, since I was unable to figure out what the point of disagreement was. I’m now going to go back and reread it again, and see if I can figure out what you were actually asking and whether I In fact have an answer.
Until you mentioned this and I went and did some research, I was unaware that Nick Bostrom had written about anthropic reasoning and the Doomsday Argument — I’ve only read his later book on Superintelligence. If what Claude is now telling me is correct, then I gather Bostrom analyzed the Doomsday and raised some possible objections to it, but not, Claude tells me, the causality-based one I’ve made here. However, since all I know of Bostrom’s writing on the subject is a short summary from an LLM, I’m really not in a position to comment as to whether, or if so why, he didn’t reach the conclusion that seems rather obvious to me, that if you attempt to translate the Doomsday Argument into a Bayesian framework it clearly violates causality and is thus a fallacy.
Summarizing Claude, it summarized Bostrom like this to me:
Bostrom suggests two possible viewpoints:
The first, he suggests, implies the Doomsday Argument, the second its inverse that Doom is very unlikely (I don’t know what people call this, so “The No-Doomsday Argument” will have to do.)
For sake of argument, I’m going to assume Claude has this summary roughly right, rather than going out and buying Bostrom’s book and then reading it to double-check. (So, yes, I am choosing not to go read a book on the subject, and am aware of the irony involved in that choice.)
Of those, I agree with the first one EXCEPT I think the definition of the reference class has to include causality and everything we actually know (and not anything we don’t know), because Bayesianism is always about P(X | everything I know) — which was the entire point of my joke. So I cannot validly define a reference class to reason probabilistically as if someone was sampling over, that includes observers in the future (or indeed ones in parallel universes or on alien worlds or whatever) whose existence or otherwise I am unable to predict with any accuracy because my prediction of them existing or not varies significantly across different hypotheses that I still have significantly greater than zero priors for each of. To give another example, “all sapient observers in the Milky Way galaxy during the first 13.8 billion years or so, specifically in the backward light-cone of Earth now” is also an invalid reference class, even though by construction it carefully lies our past so doesn’t breach causality: it’s assuming information we don’t have, about how often life arises and evolves to sapience, i.e. some of the terms in the Drake Equation that we’re just wildly uncertain about because we have a sample size of 1 and that sample has to be discarded as due to sampling effects, since we’re here to observe it. (The lack of visible Dyson swarms, obvious signals, or alien delegations or invaders suggests some maximum bounds on the Drake Equation, but they don’t constrain it very tightly, and they only impose a maximum, not a minimum.)
In fact, for the Doomsday Argument at this particular point in history, with the current unclear existential risk level, my current prior is pretty much still my initial prior, i.e. I don’t have even a clue, while the size of the reference classes proposed some of by the different hypotheses involved in the Doomsday Argument differs by a large number of orders of magnitude between different hypotheses that I have significant current priors for all of, including the ones I earlier called “Doom” and “Stars” (and others such as “stays on Earth for another few million years”). So I’m currently about as unsure of the size of the reference class as it’s possible to be. Thus the probability of getting something like my birth order number if you were to sample over the reference class is just wildly uncertain. Thus I have to do that reasoning process separately conditioned on the two-or-more different hypotheses, and combine the results in the standard Bayesian way. Which means that the results can’t affect the priors, since each one had to assume the corresponding hypothesis. There is an X% chance that something unlikely-sounding given the size of the reference class under that hypothesis has happened and a 100-X% chance that it hasn’t given the vastly different size of the reference class under that other hypothesis, but that doesn’t provide any evidence that I can update X% on, since fundamentally, I still don’t know the size of the reference class, and if I try to reason as if I did, I would be smuggling precognition into my argument.
So basically, in this case, where we don’t have even a clue, I accept “all humans who have lived up until this point, or will be born in the next year or two” as a valid reference class to reason probabilistically over (i.e. “reason just like a Frequentist would”), because I already have reasonably firm evidence that they have in fact existed, or are very likely to exist, so assuming that in the probabilistic reasoning isn’t going to mess stuff up. However, given just how uncertain I currently am about what’s going to happen more than a few years from now, since there appears likely to be a Singularity in our near future, I consider “all humans who have lived up until this point, or will ever live” as an invalid reference class that is smuggling the results of precognition in to the argument if I try to reason probabilitically over it (without Bayesianliy conditioning that reasoning separately on the different hypotheses that control the size of the reference class — and if I do that, then there’s no update to the prior). So I can’t use that as a reference class in SSA.
So that’s what I think about what Claude tells me Bostrom said: I disagree with both the positions I’m told he outlined as alternatives, but I’m closer to SSA with a causal modification. My position is the one under which the Doomsday Argument and the “no-Doomsday Argument” are both fallacies. Because they both obviously have to be fallacies.
Note that my version of the SSA isn’t actually that useful: it’s basically a way of doing an internal consistency check on the hypothesis you believe. If you do it and it says “by your hypothesis, a huge coincidence has occurred” that suggests that going and trying to think of a new hypothesis that fits all your observed facts equally well and don’t make some aspect of it a huge coincidence might be a good idea. But if, say, 30 bits of coincidence have occurred by your current hypothesis (so a one in a billion fluke), that only supports 30 bits of additional hypothesis complexity that eliminates the coincidence — which isn’t that much, a basically few words or a smallish equation. Otherwise Occam’s Razor (minimizing Kolmogorov complexity) still wins.
I find that I am still explaining things in painful detail — fundamentally, because I’m not sure what question you’re asking by raising Bostrom.
You said:
To which I agreed:
To put this yet another way:
P(there exists a 100 billionth person born| there will only ever be 101 billion people born) = 1
and
P(there exists a 100 billionth person born| there will eventually be quintillions of people born) = 1
so we can deduce exactly nothing about how many people will be born after us from the simple observation that we were born number 100 billionth (or so) and thus people #1 up to #100,000,000,000 have all existed. Under the quintillions or so hypothesis, it will later in retrospect turn out that all of us born so far were all in some sense very atypical: yes, someone had to be 100 billionth, but that’s still vastly closer to the state of the birth order than the end if the end is in the quintillions. But we don’t currently know that, and if it later turns out to be the case, so what? Someone had to be 100 billionth, whether that’s rather near the end or astoundingly near the beginning, it still occurs with probability 1 under both hypotheses, so there is no Bayesian update between the hypotheses when it happens.
Similarly, if a very shortsighted ant is walking along a ruler, and reaches the 1mm line, that tells it nothing about whether this is a 2mm-long ruler or a 10km-long ruler: it can’t see that far, so it has no evidence yet. It only shows it’s at least 1mm long, plus the small distance the ant can see (in chronological terms, as far as we can predict with any accuracy). You can’t make deductions about how much bigger the size of the reference class might be beyond the number of members you already know about.
I believe I am simply restating your argument here, since I agree with you.
Claude tells me that the essence of what you’ve been trying to ask me is:
You’re not allowed to use that fact because you don’t actually know if you’re in the 10% or the 90%. In the absence of that information, you don’t get to make a Bayesian update. To do Bayesian arguments correctly, you need to respect causality, and only use information you actually currently have access to.
Claude interpreted your question as base-rate reasoning, though when asked it admitted that you don’t use the phrase. I’m going to assume that it was correct. Base rates are normally a valid way to set your Bayesian prior. If you know “the base rate for people having disease X is Y%”, then a reasonable initial prior for whether a particular patient has that disease, in the absence of patient-specific evidence, is Y%. But for Doomsday, we have no information of the base rate of species inventing AI and surviving is. We have performed the experiment zero times so far, so we currently have no good way to set a current prior. So any argument that suggests that we do has to be a fallacy.
In particular, using what Bostrom calls the Self-Sampling Assumption using a reference class that we don’t know the size of in order to make a deduction about its size is an invalid circular argument: it’s assuming you know the answer to the question you’re trying to answer, in breach of causality. Yes, there will eventually be a last human alive, or an AI, or an alien archeologist who will know the answer, and will be able to tell, for any individual human, whether the statement “I will turn out to be in the first 10% of people who ever lived” was true or false for them, including for you and I. It’s a statement that will eventually have a truth value knowable at reasonable computational cost, but that doesn’t yet have one (at a computational cost less than running a quantum simulation of the entire Earth and everything causally connected to it faster than real-time, which as far as we know is physically impossible, and we’re certainly not in a position to do). In particular, that’s a statement that will turn out to have been true for everybody born until some point in time, and then false for everybody born after that point. So the “base rate” is initially 100%, dropping to 0% at some time that we don’t yet know. But since I don’t yet know that information, I can’t, in Bayesianism, update my priors now based on information that I don’t yet have and that will only exist (at less than vastly unreasonable computational cost that I haven’t paid) in the future, so I have no way to access yet. Bayesianism is about how to update your priors when you learn new information, and the Doomsday Argument is the Bayesian equivalent of trying to lift yourself up by your own bootstraps in the absence of any new information.
(I remain puzzled why this wasn’t obvious to Bostrom, assuming that he’s as familiar with Bayesianism as Claude makes it sound like he is. But then I’m puzzled why anyone falls for the Doomsday Argument: as soon as you notice it’s breaking causality, it seems obvious to me that it has to be a falacy, and the question then is where the flaw is. The answer is in an invalid choice of reference class. Or maybe I’m just a physicist and have had causal thinking drummed into me — though to be fair I’ve seen plenty of physicists abuse anthropic reasoning too, sometimes in acausal ways. I even wrote a joke about this.)
Was any of that a successful answer to the question you’ve been trying to ask me? Because I’m afraid that, even after rereading our conversation carefully, I have to admit I’m still unclear what you’re actually asking, or where we actually differ, if anywhere — so much so that I’m resorting to asking an LLM to tell me. Or, if none of that is an answer, then could you please accept my apologies for being confused, assume that I haven’t read Bostrom’s book on the subject, that as far as I can tell we agree with each other, that I have now (very belatedly, for which I again apologize) figured out that you are familiar with Bayesianism and that you just don’t see something that seems obvious to me about how to correctly apply it as being obvious, but that, if I still haven’t managed to answer your question despite multiple attempts, then I still have no clue what that specific thing is — and try to explain what exactly you are asking me to clarify about my position again, more slowly? It’s entirely possible that I’m the ignorant one here: I’m certainly puzzled as to what if anything we disagree about or why.
Alternatively, if you simply want to drop this rather long conversation here, then please feel entirely free. I’ve already upset you once, and I most certainly don’t want to do so again.