Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
Can Bayesianism deal with the mail-order prophet scam? If so, how?
This scam uses a segmented mailing list and a large initial population to advertise predictions so that a ‘winning’ portion of recipients receive an apparently improbable series of correct predictions. Most recipients (the ‘losers’) receive an incorrect prediction at some point.
For clarity, here is the situation:
A Bayesian with no prior knowledge of the scam receives a letter advertising stock predictions via a new proprietary quantitative model. The letter predicts that by the end of the month, stock XYZ will be up. It also states that the firm will prove their legitimacy by sending 6 more predictions (one per month) that will correctly predict whether XYZ is up or down. Sure enough, the Bayesian receives each letter, and the predictions all end up correct!
Unbeknownst to the Bayesian, 63 other Bayesians also received the first letter, 31 others received the second, 15 the third, 7 the fourth, 3 the fifth, and 1 other received the sixth letter. Our unlucky Bayesian was the only one to receive the seventh.
The question is, assuming no knowledge of the scam or communication with the “losers” (for whom a stock prediction was wrong), should the Bayesian—strictly adhering to Bayesian Epistemology—believe that the firm’s stock predictions are legit?
I don’t know how to approach this from a Bayesian perspective and would appreciate any guidance from someone who’s more familiar. My intuition is that the Bayesian should believe that the predictions are legit (which we observers know to be incorrect).
Also I feel I should disclose that I’m not a Bayesian (rather a Critical Fallibilist), and I’m looking for the best Bayesian answers to this problem (links to prior material welcome). It’s often quite hard to find the best answers to a problem from a particular school, so please bear with me if I’ve missed something obvious.
IMO there may be one Bayesian who falls for this, and there may be one frequentist who is struck by lightning twice.
depending on your prior that some unknown person is mailing you tomorrow‘s stock prices (probably very low) there is some number of correct predictions which should convince you this is true. If that prior weight is low enough (and it should be tiny), it might be that the number of correct predictions is so high that the entire world’s population of Bayesians is not enough for one Bayesian who falls for it. But if there IS one bayesian who falls for it, that’s bad luck for her, and seems like no argument against bayesianism.
I think your intuition that this is a problem comes from the idea that “oh now all the frequentists can start running this scam and money pump all the bayesians.” but if you actually do the calculation, it won’t work out that way, because the scams interfere with each other. When you receive the 101st (first) letter from a new scammer, you don’t bother to open it because your prior that it’s a scam has increased a lot; all the previous cold letters have turned out to be scams.
Already there. The advice I always give people asking about a dodgy-looking message they’ve received is “IT’S A SCAM, IT’S ALWAYS A SCAM.”
One you accept that IT’S A SCAM, IT’S ALWAYS A SCAM, you can then at your leisure speculate on how the scam works, i.e. what was the generating process that created the message, but it doesn’t really matter if you can’t work out how the scam works, because IT’S A SCAM, IT’S ALWAYS A SCAM.
I’m not actually that interested in the scam, more how Bayesianism handles the problem. If we assume the Bayesian has reasonable priors and isn’t naive, then your answer makes sense. But when we are talking about science and the frontier of knowledge, we don’t have that luxury.
I think I can demonstrate this with a much more abstract problem: how can I use Bayesianism to evaluate multiple schools of epistemology to find the best one? Can Bayesianism come to a definitive answer or will there always be a non-zero probability that any of the schools are the correct one? (assuming any of them are)
While I’m asking this to demonstrate that bad priors and naivety are the default state, I’m also genuinely interested in the answer.
The same way you would use any school of epistemology to evaluate multiple schools of epistemology (including itself). Start from where you are, for there is no other way to begin, and see where it gets you.
Okay, so what events or things do you pay attention to for updates relating to Bayesianism? Like say I’m a Bayesian with random priors, except for
P(Bayesianism)which is say 0.51 currently. What kind of things do you observe that cause updates and why/how?Sorry for the delayed replay. My comments are being held for review.
I don’t think P(my epistemology) is a practically useful concept.
Various people have suggested paradoxes of Bayesian reasoning. The few I’ve looked in detail at, I don’t think hold water. But that’s the sort of thing to study.
Does critical fallibilism have anything to say about itself? If it gives itself a full bill of health, that’s no more useful than the statement “This sentence is true”, which can consistently be assigned any truth value.
This is surprising to me—I’ve always thought that whatever our understanding of knowledge was should apply to itself. If it doesn’t, how do you know that your epistemology is right? An epistemology is the thing that’s meant to give you the answer to the question. So it seems like a problem if an epistemology can’t apply to itself.
One response might be ‘Bayesianism is about science, not philosophy’, but then on what basis are you adopting bayesianism? if that decision is supported by some other epistemology, then why not use that instead? (it seems like it might be more powerful because it can do something bayesianism can’t)
Yeah it does, though a ‘full bill of health’ might mean something different. CF says that we should use the best ideas we have access to. “Best” means that it works to achieve a goal and is unrefuted (no unanswered decisive criticisms). CF claims it has no decisive criticisms (at least insofar as the epistemology is concerned), but does not claim such criticisms are impossible (it’s fallibilist after all). CF is Elliot Temple’s school and he is very willing to debate (which means a forum discussion).
From Debate Policies Introduction, he says:
One of goals of CF-ists is to find and debate people who disagree with you because that’s the best way to get criticisms, which are an integral part of error correction and improvement.
But also the world isn’t very good for debate, at least not at a high level. It’s hard to find people and forums who will continue a discussion to it’s conclusion. So maybe CF isn’t as good as it could be if there was more debate.
If someone knew of a criticism of CF, then Elliot would want to know. If he disagreed, he’d want a public discussion about it.
I can expand on anything if you have questions.
Arguably, using Bayesianism to evaluate schools of epistemology (which may not be Bayesian) is a type error. However, I think I still endorse assigning tentative probabilities to beliefs about epistemology and updating them by Bayes rule in practice, so I think I don’t have a full answer for you (only a bunch of disorganized thoughts; it’s a deep question).
It’s not quite clear to me what you’re getting at.
If it’s a type error is that because bayesianism’s domain does not extend to all knowledge?
If bayesianism doesn’t cover all knowledge that would make it an incomplete epistemology. This is problematic for a few reasons (so I’m not sure this is what you mean); one is that it becomes unreliable for any knowledge because it cannot tell you what you are missing or how that missing part is involved in meaningful judgements.
If bayesianism does cover all knowledge than either epistemology (and bayesianism) seemingly doesn’t count as knowledge or we have a contradiction.
Or maybe I’m misunderstanding what you mean?
Personally, I’ve long thought that an epistemology should be able to evaluate itself (and other epistemic schools/ideas). This belief was quite formative for me and is one reason that I gravitated to fallibilism (broadly) since it seemed like any ‘true’ theory of epistemology would have this quality.
All bayesian answers to the original question (that I’ve seen in this thread) assume the bayesian already has good priors and isn’t naive. But that isn’t how we find ourselves in the world. We start from bad priors (random or worse) and naivety. So, while I agree real-world bayesians can avoid being scammed, I am not sure that bayesianism avoids it since the way of avoiding it is to have already obtained the knowledge you need to avoid it. I’m trying to see if there’s something more fundamental. That real-world bayesians don’t get scammed isn’t really a surprise, one doesn’t need a correct understanding of epistemology to reason about the world—e.g., we were able to do science and make progress before we understood science—but if we are concerned about whether bayesianism is correct or not, then it does matter.
I’m interested in how these kinds of issues (novel unintuitive systems) are dealt with when one doesn’t have obvious crutches like a known good state to start from.
The reason for asking about using Bayesianism to evaluate schools of epistemology is because epistemology is notoriously hard to make progress in. I was hoping that the idea we start with bad priors and naivety was more obvious in this context.
Hope that helps explain what I’m getting at.
Sorry for the delayed replay. My comments are being held for review.
epistemology is not complete as a project! I think meta-epistemology in particular is still pretty confusing. i suspect that the pragmatically right approach still involves Bayesian updating, because Bayesian updating is just correct in many situations.
I think you’re using the word “Bayesian” to mean “someone who believes all knowledge can be derived from Bayesian updating” and this is not really standard. There’s a lot of types of Bayesian, e.g. most Bayesians just favor Bayesian statistical methods. The claim that we ought to use Bayesian updating is probably consistent with many epistemologies.
Now you’re arguing that you have to learn to not trust mail scams. But this is true whether or not youre Bayesian, for standard no free lunch reasons. Bayesianism tells you how to work with the knowledge that you have. I feel like you’re being uncharitable here.
Here is my Bayesian analysis from the perspective of someone who’s received six successful predictions in a row.
Number of people in history who have received similar predictions and it was a scam: a few thousand, maybe
Number of people in history who have received similar predictions and it was legit: exactly zero, because no one who can reliably predict short-term market moves has ever in history marketed themselves this way
Therefore, I have strong Bayesian evidence that it’s a scam.
Is there a way to address it without relying on accurate priors and general knowledge of the world? Typically with science it seems like we don’t have accurate priors or very good contextual awareness. (I posted a concrete example here)
To be fair, my analysis was actually more of a frequentist analysis because all I did was count the number of occurrences of each thing without using a prior!
I think I understand why frequentist-ism is clearly separated from bayesianism broadly speaking, but it also seems to me that in the absence of anything better, taking a frequency is not a bad way to get a starting point.
Is that common/accepted as part of bayesianism? Because if it’s not, I don’t know where priors would come from besides random allocation or evenly split between all hypotheses (which is problematic because there are
Sorry for the delayed replay. My comments are being held for review.
If you have weak/uncertain priors, the thing to do is run low-cost experiments that differentiate between your different hypotheses of what’s going on.
A real cheap experiment in relation to the question “is this a scam?” is to Google to see if others have received similar letters and what their outcomes were. If it’s a scam, you’re likely to surface evidence of this, if it’s not a scam, you’re likely to surface both people saying it went well for them, and debunking sites that explain what’s going on, where the letters came from, etc. If you get no information at all based on a google search, just no results come back, someone went to a lot of effort to make that be the case, and you should be suspicious, it’s evidence something’s amiss.
In a case where you’re at the forefront of scientific discovery, there may be no cheap tests available, but you still devise tests which you predict, based on what you currently know, will go one way if a theory is correct, another way if it’s incorrect, and see what happens.
CF kind of agrees here. CF says that any contradiction between theories can be used to create meaningful hypotheses that will refute one or both theories; and CF doesn’t have a concept of weak or strong arguments, instead it uses decisive criticisms and claims are either refuted or unrefuted.
What does Bayesianism say if you have strong priors? Is there an objective breakpoint that separates weak from strong priors? (e.g., P < 0.5?)
There isn’t a set breakpoint that separates weak from strong priors, it’s a continuum from “it seems extremely unlikely that everything I know about the world is false, but it’s not technically impossible” to “it seems extremely likely that I’m typing on a keyboard right now, but there’s a tiny possibility that something else is going on, like a hallucination or me being a brain in a vat or some other possibility I haven’t thought of”.
Bayesianism says that if you have strong priors about a particular matter, you should be surprised with corresponding strength if something your priors said shouldn’t happen, happens. Some occurrences mean “I adjust what I think slightly, but this was within what I guessed might happen”, while others mean “halt and catch fire—either I should fundamentally rethink my notions of how reality works, or I’ve deeply misunderstood what just happened”.
I haven’t read about CF (I skimmed your link extremely rapidly, which doesn’t count) but it sounds pretty binary. Bayesian thinking is not that.
re: binary. Yes… well you could argue that technically it’s ternary in that ‘unsure’ is an option, but CF says there’s always a way to get from ‘unsure’ to yes/no.
It’s similar to critical rationalism (CR) but removes all degrees for beliefs. CR was already pretty light on degrees/credence/weights (pun not intended), CF removes them entirely. CF also integrates some ideas from other schools (so it’s not a pure descendant of CR).
What kind of event would cause you to update your prior about bayesianism?
Does bayesianism say anything about things that cannot be observed directly?
e.g., many moral arguments are about what is right/wrong to do, and we often don’t see consequences, and even if we do see consequences, those aren’t outputs of an experiment in the same way science experiments work. So it’s not clear to me that (from bayesian pov) we even should use our observations about the world to update our ideas about morality.
(note: we don’t need to focus on morality particularly, it was just an example I thought we’d both agree involves things we can’t observe directly)
Sorry for the delayed replay. My comments are being held for review.
initial reaction to your several replies today: I feel like writing several replies that would be quite long, but a) I don’t have time to do that within the next few days and b) I don’t want to spam you with walls of text you’re not interested in. I’ll try and refine my thoughts down to something more short and focused, but, how interested are you in reading the longer less-focused version? This is a “I wrote you a long letter because I didn’t have time to write you a short one” situation—longer is easier, but potentially less useful.
“Does Bayesianism say X” is a complex question to answer—there are a cluster of ideas that are implemented among the group here, that I personally have noticed, but I don’t have a canonical source for what “Bayesianism” says or does not say, so I’d be sharing my current impressions and understandings, and as I’ve said before, I’m not an expert Bayesian, and what I think might not be representative of the group.
As for “What kind of event would cause you to update your prior about Bayesism?”… well at the core of this set of ideas is a mathematical formula. And I understand why that formula is what it is (I walked through the reasoning once, years ago, and it made sense). So if the question was “what kind of event would make my believe Bayes’ formula is incorrect” then it would have to be the same sort of thing that would cause me to question the validity of math more generally—something that would make me think “maybe the Pythagorean theorem doesn’t reliably describe characteristics of triangles”. There are things that could do that, but it’d be a pretty fundamental questioning of the nature of reality. Or, I guess if some mathematician found some fundamental flaw in Bayes’ formula and I could walk through their reasoning? But “Bayesianism” and “Bayes’ formula” are not the same thing, and I could give up on various ideas that cluster around Bayes’ formula much more easily. If I saw this group systematically making errors in thinking that I could trace back to a Bayes-adjacent idea, I’d update my thinking based on that evidence fairly easily. What I’ve seen instead is that members of this group using this form of reasoning have reached conclusions that turned out correct, well ahead of society in general coming to the same conclusions. if that changed, I’d learn from that.
When I said “Bayesianism says”, a few messages above, I was just running with your conceptual frame for the sake of discussion—and there are certain things that are implied by Bayes’ formula which I think everyone would agree on, like “if you have a high prior on something happening that doesn’t happen as expected, that causes a big update, and similarly for low priors on something that happens”. Inputting certain numbers into Bayes’ formula means other numbers come out, and in that sense “Bayesianism says/Bayes’ formula says” consistent things. But there are other ideas that kind of come along for the ride and might be grouped under the label “Bayesianism” while being less directly connected to the formula, and it’s there where I go “I may not be the best person to comment on that/it’s a bit fuzzy/complex”.
Heh, I know what you mean.
I’d suggest picking something you think we disagree about to focus on if we want to make some progress (helps keep discussion tree width small). But you’re also welcome to write down your thoughts with less structure—I’m still interested in that and I get that it can be easier sometimes. If you do, it might be good to, at the end, point out what you think the strongest or most important point / disagreement is, and I’ll focus on that and/or suggest something of my own.
FWIW, I’m already familiar-in-passing with more modern retreats from like hardcore bayesianism. IMO it’s a bit unprincipled but necessary because bayesianism isn’t complete (eg hypothesis generation, evidence modelling, etc). One thing I don’t like about the state of bayesianism is everyone seems to have their pet theory about how to handle the problems. IMO this is a sign that bayesianism is failing (but doesn’t mean it’s irredeemable). If b-ism weren’t failing, we’d know of a more consistent integrated position that people point to and say ‘see X’s book/essays for the gold standard’.
If there is one thing in particularly that I take issue with, it’s the idea that Bayesianism is the logic of science. I disagree with that wholeheartedly. The problem is the leap from applying a statistics method epistemologically. Bayes has no epistemic insights that aren’t covered elsewhere, and the domains where it works aren’t useful for getting to the truth and understanding the world proper. It doesn’t help us to explain how and why scientific progress happens and is possible.
Edit/Addendum: to (loosely and reductively) analogize to pythagoras: bayesianism is to bayes what pyramid power is to pythagoras.
This isn’t exactly a fair analogy, but the point I want to make is that the belief that bayes’ theorem applies to epistemological matters is a massive leap. I can see how it’s tempting and it’s much easier to see how intelligent people would fall into the trap. I don’t mean to be insulting or offensive btw, and I’m sorry if I am, but I don’t want to lie about what I think and am a little short on time.
Bayesian is about having priors and updating them. If your prior is that the efficient market theory is true and that there are some people who run scams via unsolicited email exist, getting 6 letters is no strong reason to update towards the company sending the unsolicited email being legit.
It seems like a consistent element of everyone’s explanations so far has been that the bayesian has reasonable priors and is not naive. I’ll respond to everyone at once here (hope that’s okay) to keep things organized. cc @michaeldickens, @cole-wyeth, @gavin-runeblade, @thomas-castriensis (thanks everyone for the explanations so far).
edit: IDK how to mention/tag people :(
One thing I notice, the Bayesian needs a prior for ‘unknown scam’ but shouldn’t they also consider the inverse? Like an unknown bank error in your favor type situation. I suppose that if the prior for that is small enough it doesn’t matter so it’s ignored as insignificant. Is the significance a matter of the ratio between priors of unknowns? Like it’s way more likely to be an unknown scam than an unknown bank error, so we ignore the bank error.
Something bigger doesn’t quite sit right with me: I don’t think having priors and not being naive is our default state for knowledge frontiers (hence the initial framing and restrictions on knowledge of scams). Rather, the default is that we are missing priors (or they’re wildly inaccurate) and we are naive to the ways in which we are naive (there is a lot that we don’t know we don’t know).
Here’s a concrete example: consider a Bayesian in 1901 (before general relativity). Almost all measurements of planetary motion agreed with Newton’s universal gravity (UG) and Mercury’s orbit was our main (only?) counterexample. We had multiple (incorrect) theories about how Mercury’s orbit might be perturbed by this body or that. The most recently discovered planet at the time (Neptune) was predicted via UG (0.125 of all known planets).
My understanding is that the Bayesian will, upon seeing measurements of all the planets’ orbits, update based on successful predictions. Most of the predictions are correct for UG alone, and all of them are correct for UG + New Mystery Planet. Let’s call these hypotheses
Now, based on the reasoning used (if I understand it correctly), the Bayesian already has reasonable priors (at least they believe so), and the general relativity hypothesis (playing the role of the scam hypothesis) is already accounted for in the Bayesian’s calculations and updates. It doesn’t matter that the Bayesian is unaware of how it works or not. [Unsure] The probability of GR being true is independent of whether the Bayesian knows about it or not [1] . Let’s call all the other hypotheses (excluding any UG hypotheses) just
Okay so in case of observing something uncontroversial, like Mars data,
Then, when observing Mercury, we see
Have I gone wrong somewhere? I’m not confident that I’m at the right midstate here. If there isn’t a problem with my logic then this result feels intuitively wrong because we’re getting more confident in
Also it seems like
that’s how it seems it must be to me. I can’t quite put my finger on it, but it feels wrong that learning about an idea independently of all observations of data somehow updates things retroactively.
Bayesianism has no rules for what someone priors should be. It has rules about how to progress from a state of having priors.
There’s a reason that Einstein did not get his Nobel Prize for the special theory of relativity. At the time the prize was given, the Nobel Prize committee did not believe that the predictions about Mercuries orbit were strong enough evidence for the special theory of relativity to give him the Nobel Prize for it. Besides Mercuries orbit there was also the Michelson-Morley experiment.
To the extend that you were taught in school that the Michelson-Morley and Mercuries orbit provided definite evidence for the special theory of relativity, that’s a retrospective accounting from people who already knew it to be true and not the perspective from the physicists that gave Einstein his Nobel Prize.
Whether or not physicists at the time should have updated more strongly into the direction of seeing special relativity as proven depends a lot about what you believe of the merits of alternative explanations for the observations and how well those fitted the data and how likely you consider the measurement for Mercuries orbit to be correct.
In Bayesanism probabilities are not independent of the model of the observer the same way that frequentism has a notion of observer independent probabilities.
The general mechanism with the mail order scam you talked about is called survivorship bias. If you take the question of how high the existential risk of being nuclear war happens to be, this matters. If you just observe that we now have nuclear weapons for a long time, it might be wrong to update with each passing year into a lower chance of nuclear war because you would not be around to observe reality in case everyone got killed by nuclear war. That’s why we need to look at Petrov and Arkhipov to get an understanding about near misses and we treat both of them as heroes on LessWrong.
Sure, but that’s not exactly what I’m asking about.
From your perspective, you have a prior
P(bayesianism). I’ll call thisP(B)for short. There’s also some priorP(not bayesianism)that covers all other epistemologies, both those which you know about and those you do not. DoesP(B)change:when you learn of other epistemologies? (I think this breaks up a bayesian’s hypothesis space but there’s no new data)
when you learn about other epistemologies? (Like details about how they work)
when you learn that you had an obstructive misunderstanding about other epistemologies? (Like, you thought it worked like X but actually it was Y)
other events?
You can replace P(B) with P(GR) if you like and ‘epistemologies’ with ‘theories of gravity’. The underlying thing I’m unsure about is where the update to the prior happens and why. If it is when you learn about the new theory, then it seems like an admission that the P(other) option is just outright wrong, so working backwards I guess bayesianism stays consistent by not updating the prior in response to learning a new idea.
Sorry for the delayed replay. My comments are being held for review.
I think you need to make a clearer distinction between a hypothetical perfect bayesian reasoner, who would know all possible hypotheses from the beginning and only narrow them down through observation, and the things humans do to try to approximate that, which will sometimes involve going back and changing the prior when a new hypothesis has been thought of.
Keep in mind that these “probabilities” are subjective assessments of probability based on an individual’s prior knowledge, not facts about reality. Two Bayesians with different prior experience may disagree about how probable something is (/seems to them), but reality will not disagree or debate with itself about the truth of the matter, or assign probability to different possibilities (mumble mumble I don’t really understand quantum mechanics and am pretending it doesn’t matter for the purpose of this conversation).
Whether or not General Relativity is true is unaffected by any probabilities any Bayesian may put on its truth or falsehood when reasoning about the evidence they’ve seen so far. But whether or not a particular Bayesian finds General Relativity to be probably true, is definitely affected by whether they know about it or not. Keep a clear distinction in your mind between “the probability a Bayesian reasoner assigns to some fact being true” and “whether or not that fact is true in reality”—these are not the same.
This is a “map vs. territory” distinction. Bayesian probabilities go into a mental model of how the world works (map), while how the world actually works is separate from the map.
Doesn’t that mean we should expect that Bayesians often disagree and they have no way to resolve it except consulting reality (i.e., an experiment)?
If that’s the case, why bother with Bayesianism at all? It seems like any situation where people needed to agree on the truth of something, Bayesianism wouldn’t help:
They already agree (‘scientific consensus’ and similar situations) -- Bayesianism doesn’t offer anything here.
One person disagrees, everyone else agrees—without more data no progress is possible, so either we know a decisive experiment that will resolve the issue (in which case we don’t need Bayesianism), or we don’t, in which case Bayesianism can’t help.
Lots of disagreement—similar to above but more chaotic.
Also say there is an experiment, is there any standard or agreement among bayesians about how to weight credence? (when should it be weak or strong? etc) Because if there isn’t, they might not even be able to agree on what experiment to do or if it will matter.
The FDA just decided to allow Bayesian statistics to be used to argue for drug approval. If a drug company wants approval they will need to state their priors and then how they updated based on the studies they ran. If the FDA does not agree, they might say “Well, I don’t think those priors are reasonable, please use X as priors”. I expect that the exact rules of how the process goes will evolve with time.
On another practical application space, you have superforcasting where credence calibration is important. You make progress by believing people with better Brier’s scores (or log loss). If some scientists in a field can make predictions about experimental outcomes with better Brier’s scores, than they seem to understand more about the domain and should be trusted.
Are you responding to this? (which I said above)
If so, I think we might be talking cross-purposes. I’m asking about bayesian epistemology particularly, not statistics. (Nor about estimations based on other people’s beliefs since that doesn’t seem like an epistemology except maybe JTB with more steps)
If not, I’m not sure what your point is particularly.
Sorry for the delayed replay. My comments are being held for review.
The question “How do we know which drugs work and are beneficial to patients?” is an applied epistemology question. Looking at how it gets answered by a sophisticated system tells you how epistemology actually works in practice instead of how philosophers think it’s supposed to work in their ivory tower. If you use Bayesian statistics you want an epistemology behind that use that guides you in how you use the statistics to reason.
Superforcasting is much more about Bayesian epistemology than about Bayesian statistics. You have Superforcasters who would they they are Bayesians but can’t write down Bayes theorem.
You seem to have some idea that epistemology is supposed to be “objective”. It’s supposed to give you the answer from God’s view. A lot of Western science is build around wanting to reach God’s view. The problem is that God doesn’t exist. According to Nietzsche, he’s dying. Bayesian epistemology is an epistomology without God, which means that you have to deal with your beliefs and other peoples beliefs.
The reason to bother with Bayesianism is not because it helps you to see the world from God’s view but because it has practical utility in applied epistemology with FDA drug approval and Superforcasting being two examples.
Objective in a sense, but I’m not sure how I’ve given you the ‘God’s view’ impression. I think epistemology should be objective in that it should work universally via the same rules, and that people can discuss both ideas and the world (evidence) in such a way to reach agreement in an objective sense. But they can be wrong, there’s no infallible method of getting to the truth. They can also objectively agree on each other’s subjective states.
Putting aside superforecasting because I don’t know much about it, using bayesian statistics for statistical analysis is fine.
But that’s not what’s happening on LW. When people on LW talk about their priors and updating them, they’re not talking about bayesian statistics, they’re talking about epistemology, about what ideas are true. I think those are fundamentally different things and they work in different ways (and it seems like you do, too). I’m here because LW is the largest bayesian forum (or if not the largest it’s better than reddit for discussion, point is, it’s the best option for talking to bayesians).
The idea that we should apply bayesian statistics to epistemic tasks is what I’m interested in discussing.
If superforecasting is important to discuss, can you link me to something that represents what you mean by it?
If you use frequentist statistics you can just claim that your data follows a normal distribution (which it objectively most likely isn’t, even the archtypical example of height violates a normal distribution because there are more people with dwarfism than the normal distribution assumes). On the other hand, to use Bayesian statistics you do need to decide on priors which does involve subjective decisions.
The fact that you aren’t explicitly thinking of God doesn’t mean that the idea of objectivity does not come out of a Christian scientific tradition which had as a key motivation trying to see things from God’s view. Your ideas of how you think, it should work have that theistic origin.
Bayesianism as discussed on LessWrong is about how it would be good for an agent to reason and that’s a different goal. Eliezer was interested in it because he wants to know how what’s true about how agents effectively reason to be able to say things about superintelligence.
Philip E. Tetlock separately was interested in the epistemic task about how to reason about what to believe. After the Iraq war the US military thought they had problems with epistomology as shown by the fact that they got the WMD question so horribly wrong. Out of that there was an IARPA tournament where Tetlock’s team won. Tetlock runs GJOpen which does solve epistemic problems for entities that want probability for certain events happening. He got some grant money from OpenPhil. There’s also Metaculus that comes out of rationalist sphere. His book Superforcasting is a good resource if you want to understand how the kind of Bayesianism where people don’t explicitly use statistics works in practice.
I’m not sure what the best shorter source is but maybe https://www.gjopen.com/training/
Short answer: Yes.
Longer answer: Two Bayesians who start out with the same prior probabilities, and see the same evidence, should update their posterior probabilities in the same way, and so their mental models should stay consistent with each other. Two Bayesians who start out with different prior probabilities, but see the same evidence, should update their posterior probabilities in ways that are predictable to each other, and in line with the evidence—that is, if one reasoner (A)‘s prior probability that (for example) General Relativity is true was high, while another (B)’s was low, then when an experiment is run which provides evidence for general relativity, A’s estimates of General Relativity’s likelihood of being true will change less than B’s (because B’s priors were more wrong), but both will update in a direction and to an extent that is predictable to either of them. As they see more and more of the same evidence, their models of the world should converge.
This is all assuming an ideal Bayesian reasoner with practically-unlimited computing power who doesn’t cheat or decide not to reason according to Bayesian rules when it becomes inconvenient, and humans don’t meet those constraints. But, there’s math to say how much you should update given particular evidence. So:
Yep. “How to weight credence” is a bit unclearly stated, but there’s Bayes’ formula, which tells you how to update your probabilities based on evidence, and that might be what you’re getting at?
Which is (one reason) why bother with Bayesianism at all. It’s a method of approaching consensus when working under uncertainty. It’s kind of an “agreeing to the rules of the game” situation, where “the rules” are a mathematical equation that says how probabilities must change when people are disagreeing (and “must” here carries the same level of mathematical strength as saying “2+2 must equal 4″, it’s not a thing that was decided by committee) - if for example you say it’s 95% unlikely/5% likely that something will happen under your idea of how the world works, and then it happens, if you’re playing fair, you make a big update, and if you put numbers on it, Bayes’ rule tells you what your new numbers should be. If you don’t like the new numbers, you have to either acknowledge that what you said your priors were was incorrect, or that what you said your likelihood estimates of different outcomes were was incorrect—so either you retroactively revise how you used to think the world works, or you retroactively revise what you thought would happen and how confident you were, both of which are kind of awkward and embarrassing. And the people you’re disagreeing with, if you don’t make an appropriately sized update given what you told them your priors and likelihood estimates were, can point this out as a fact. And if you’re not very confident, or you don’t think particular evidence should carry much weight, you can express that in a way that makes it clear how much you’re going to update based on whatever evidence you see, before you see it, so it isn’t like:
“I think x is definitely wrong, and y will provide strong evidence”
″OK, so y didn’t go how I thought, now I think x is only almost certain to be false”
It’s like:
“I think X is a% likely to be false, and Y experiment will turn out the way I expect with b% likelihood as a result.”
“Oh. Ok, well, I guess now I’m down to c% likelihood that X is false. Shoot.”
And instead of being like “it’s not fair that you moved from definitely to almost certain based on something you said would provide strong evidence but didn’t go the way you expected”, the reaction is “yep, that math is correct, you updated how you should have given your priors”. And before you get to that point, pre-experiement, you can argue over whether b% likelihood is reasonable, where it’s hard to argue about the correct meaning of the word “strong”.
And once you get really familiar with doing this (I’m still not great at it), you know intuitively how much putting X% probability on a particular outcome means you’re going to have to change your views if it doesn’t happen, and you become appropriately cautious (“calibrated”) in your estimates, and your saying things in probabilities conveys a lot of information to other people who are also familiar with talking this way.
All of that is in idealized theory among people who are quite smart and can do lots of calculation in their heads. Lots of people also LARP it and use Bayesian-sounding words without actually having the deeper intuitive understanding of what what they’re saying means.
Okay.
FWIW, CF says that two people should be able to use discussion to resolve most disagreements. Some disagreements do require experiments, but only when we have multiple contradictory theories that otherwise agree. (e.g., UG and GR mostly agree so it’s reasonable that two scientists might disagree on which is correct but they should be able to agree on a decisive experiment.) When such discussions fail there are ways to deal with that, too (so you can go address the reason for failure then resume the original discussion).
Doesn’t that require some assumptions about their hypothesis spaces? You assume that B’s priors are more wrong, but we don’t know that from just saying that
I agree the update can be predictable to one another if they specify and communicate the necessary constants / weightings / etc. re specification and communication: For something that seems important to bayesians I’m surprised that I don’t see more of that.
I’m not convinced of this but I agree it seems that intuitively, most of the time at least, they should.
This surprised me to read and I have some immediate questions.
When is Bayesian reasoning inconvenient?
Do you think there’s other valid ways to reason?
If so, why do we need bayesianism? Is Bayesianism incomplete?
Or if not, how would an ideal Bayesian reasoner ever be fooled into using bad/broken/misleading methods of reasoning?
Are there other more energy efficient ways of reasoning to get to the same answers reliably? Why use bayesianism at all in that case?
Is cheating ever consistent with bayesianism?
Sure, but so is “do whatever Max says”. Consensus on its own has nothing to do with truth. Now, I guess you mean something a little more specific but I don’t want to put words in your mouth.
Here’s an example of how Bayesianism fails to produce consensus (or it seems like that to me):
Two aliens (A1, A2) who do not know about relativity are communicating. A1 observes an event E1 happens before E2, but A2 observes E2 before E1. They will each update in different directions.
Is there a Bayesian way to resolve this?
How does Bayesianism deal with confidence? Do all priors have errors attached? If so, how are errors handled in the maths?
Also, if the error bars are large enough, is convergence still guaranteed? Or what if our errors are off due to things like overconfidence?
I’m not sure how this would help Bayesians, what does a successful discussion of this kind look like?
The reason I’m not sure is unanswered questions like what are they observing to cause updates? and which priors are being updated?
Sorry for the delayed replay. My comments are being held for review.
even if we assume GR is absolutely 100% correct, and IRL we know that GR isn’t 100% correct.
I think the missing step is that you’re updating
When new evidence comes in that falsifies NMP, P(O) jumps up to 0.5.
Okay I see, thanks.
Is that because we only have
Also, I guess the
Also, do you know how things change after GR is published? (assuming no new data in the mean time)
I don’t have a reason for setting them equal, no. The prior probabilities could be arbitrarily split between the remaining options.
Yes, that’s correct. If we were to keep experimenting and observing, we would find some data that would have essentially 0 likelihood showing up under
That last question is trickier. If there’s no new data either way, but it predicts reality better than most hypotheses in
Then you can compare which specific predictions
I’m not sure this is true in general. Sometimes we only figure out what data to look for to disprove theory 1 after someone comes up with theory 2. So before that, a bayesian might be gathering lots of data and updating theory 1′s prior towards P=1, but they’d be being mislead with no way to know they were being mislead. But also, where does theory 2 come from? Either it was something the bayesian could have thought up themselves (in which case why were they updating in the wrong direction?), or bayesianism is incomplete and theory 2 was generated in a non-bayesian way (and in this case it’s hard to see bayesianism as anything but inferior to this other method; at the very least the sum of both methods is definitely better than bayesianism alone).
There are an infinite number of those, though. Putting that aside, there are other mathematical constructions distinct from GR that are in
And it seems like in order to split
But UG_other makes all possible predictions (since it’s a group of many hypotheses). Or close to ‘all’, at least.
Oh and I think I just saw an issue with your original reasoning:
The evidence that falsified NMP also falsified some hypotheses of O (but not GR). So
Sorry for the delayed replay. My comments are being held for review.
(This is not intended to be “the” Bayesian answer, just my unsophisticated first reaction.)
Even if our unlucky Bayesian doesn’t know about this particular type of scam, shouldn’t we still assume that he believes that
various tricks can be played on people, including him, and
scammers contacting people with investment-related claims is not rare, and
genuine oracles (i.e. accurate stock predictors) contacting people out of the blue happens very rarely if ever?
In that case he would still conclude that it is more likely that he is being fooled (although he doesn’t know how) than that he’s been contacted by a genuine oracle.
I’m not an expert Bayesian, and it’s not part of my identity which I would feel the need to defend by going “here’s why I wouldn’t get scammed”, but I know how I would answer from a “modify your expectations in light of new evidence” lens, which I understand to be the core of Bayesianism if put into plain English.
The key thing is, what are your priors?
If you were a very naive Bayesian reasoner, say a 5 year old of average intelligence, and your experience was extremely sheltered, skewed towards a very kind world where everyone was always nice to you and you weren’t really aware that scams were a thing that happens sometimes, and you didn’t know anything about how stock prices worked, you might be taken in. Because your probability that someone really could predict the way these messages indicated would be “I dunno, seems unlikely, but maybe?”
But as an adult human with the priors I have, here’s how I’d think of it if I received such a letter. All numbers are roundish and hand-wavy.
Probability it’s a scam, excluding marketing spam, donation requests, and surveys, given it’s an unsolicited message in any communication medium, selected randomly: I dunno, 70%? There’s a reason I don’t pick up calls from people who aren’t in my contact list any more, if someone I don’t know wants to reach me, they can leave a message and let me think before responding.
Probability it’s a scam, given the message is unsolicited and it’s from someone I don’t know, a business I don’t do business with, or is anonymous: 80%.
Probability it’s a scam, given the message is unsolicited, from someone I don’t know, and it’s even slightly odd in any way: 90%.
Probability it’s a scam, given the message is unsolicited, from someone I don’t know, it’s odd, and it involves me spending money to make more money later: Basically 100%. At this point, I don’t care what it says, it’s a scam and it’s getting deleted or recycled. It’s not worth my time or mental effort to evaluate the claims it contains.
Additionally and separately, but potentially relevant given the content of the message: Probability I do not have an anonymous benefactor who wants me to be rich for the sole reason that that would make them happy or discharge a duty they feel they have: 100%. I mean, not literally mathematically 100%, but close enough, I’m as sure of this as I am of almost anything, aside from things I can verify with my senses directly like “I have 5 fingers on each hand”.
Given those priors, unlike those of a sheltered 5 year old, it’s hard for a scam to get through.
If I got an unsolicited message from someone I don’t know making a stock prediction, I’d be like “that’s weird, it’s almost certainly a scam”. If they wanted me to spend money, I’d be like “definitely a scam” and throw it out, and have forgotten about it one month later.
If I got a second one, I’d check online to see what the nature of the scam was, because it is somewhat odd to get mail scams, those cost a dollar per message to run, and given my field of work friends and family sometimes check with me to see if things are legit, so I’d want to know what the deal was. I hadn’t heard of the Mail-Order Prophet scam before, so if I was sent a piece of mail trying to draw me in before having read this on LessWrong, I’d be like “neat, new category of scam!”
A similar situation, to demonstrate that I’m not BSing about how I’d react to unsolicited nice things happening from unexpected benefactors: One time, I got a piece of mail saying there was a package for me at the nearby gas station. Which was super weird, I didn’t know gas stations took packages. But OK. So I go to the gas station, and the guy behind the desk was like “were you expecting a TV?”, to which the response was “No”. But, there was a TV there for me, new in box. I was very confused about what the scam was in this case, but I was like “well OK, I guess I’ll take it”. And I looked for contact information or some other indication of who had sent this, and the only thing was a tech support/setup help number. So I called that, and was like “so this seems really scammy, what’s the deal?” And they explained that these TVs were sent to people who opened new bank accounts at a local bank, as a part of a promotional offer. And I had in fact opened a new bank account, because I needed to, several weeks prior, and nobody had told me about the promotional offer. So I was still sufficiently concerned to call my bank and confirm, rather than trusting the word of the person on the other end of the contact information on the unsolicited TV’s box. The bank did confirm, and only then did I plug it in and set it up.
Backing selection effects out of data is a notoriously expensive operation without guarantees of convergence to the true distribution afaik.
That seems reasonable, but how can the Bayesian tell in the scam example? (if there’s a selection effect at play or not) What about in the real world? We do have selection effects like this in nature (eg evolution / natural selection), so it seems like Bayesianism should be able to handle it.
The other replies gave you good examples of how to resolve this. Let me take a stab at your mistake. From a high level, you are assuming the information contained in the scam is the only information the Bayesian has available to use.
As shown a Bayesian has probably got priors about the way the market works, the way people advertise, the existence and nature of scams, etc. The information in the predictions is applied against those priors, not just against itself.
If someone is able to consistently predict the market, they don’t need the sort of investors they could get from mass-mailing people. The only conceivable reason they would be sending stuff to random strangers (and not like, hedge funds, banks, or billionaires) is if they’re trying to scam you.
Maybe if you looked at just the predictions, and didn’t consider the context in which someone would be sending accurate market predictions to you, then you could get tricked. In context though, there’s no plausible scenario where someone intelligent (superhuman) enough to make accurate stock picks consistently is marketing their services through the mail. They would have to be both smart enough to succeed, and dumb enough to not have access to better sources of funding, and dumb enough to not realize that their 10% weekly return strategy invalidates the need for investment in the first place. This sort of person is incompatible with the real world—intelligence is general enough.
It would be like if someone claiming to be Donald Trump starting texting you to tell you about his next Tariff. Even if your Trump got it right 6 times in a row, it’s not plausible he would be texting a random citizen his intentions. Maybe it warrants further attention since it is definitely some sort of signal, but it doesn’t warrant belief.
Whatever one’s epistemology, there is a fundamental question that must be asked of any observation: what was the generating process that produced it? Thus armed, one may meet such threats “innocent as doves and wise as serpents”.
Even if one has never heard of this scam, every worldly wise person knows that beyond the firewall is a mob frantically clamouring to steal from you.
The version of this scam that I’ve heard of involves the “prophet” starting to ask for money in exchange for their predictions. At that point, we have to ask, “How much?”
How much is the information worth, given the track record? And how much do we suspect (for instance) we’re getting illicitly leaked insider info?
How much value is there to the “prophet” in causing us to believe a particular thing — not “how much is your info worth to me?” but “how much is my belief worth to you?”
The likelihood function is a modeling choice. You are free to choose one that assigns a lower P(Not Scam|N Correct Predictions) than would be warranted for an unbiased model.
For this scam, even if they mailed letters to everyone on the planet, you could calculate the maximum number of guaranteed correct predictions they could have achieved if they are running this scam and were scamming a given fraction of the population. Then you could, for example, construct a model that assigns arbitrarily low probability to that number of letters and treats the later predictions as meaningful evidence.
Something else that is relevant to real-life Bayesians occurs to me. “Strictly adhering to Bayesian epistemology” is doing some work here. And in real life, if my reasoning or math leads me off a cliff/to some absurd conclusion, I have to put some weight on the possibility I’ve made an error somewhere, which I haven’t yet found. An idealized Bayesian reasoner could reason perfectly every time and be certain that they had done so, but I am not an idealized Bayesian reasoner, and I know that about myself. So how it would work is, if some long chain of reasoning or math leads me to a very surprising or unusual conclusion, I don’t throw away the possibility that the surprising conclusion is right, and that might even be my highest-probability guess, but I still leave open the possibility I’ve simply made an error. And so I might go “well this is what the math says, but I’m not going to rely on it” for anything that involves making a really important decision.
Isn’t this just solved with priors about you making a mistake? If you arrive at 1=2 or some other contradiction, it makes sense that such a prior would increase which in turn updates priors about whether the next action should be error checking, etc.
Even if I’m wrong about the bayesian way to handle this kind of case, this kind of thing doesn’t feel generally or broadly incompatible with bayesianism.
For what it’s worth, I think an epistemology needs to work in the real world—a noisy, dirty, perplexing place—in part because it is used by fallible people! If an epistemology is only reliable by invoking a perfect oracle or deduction engine or the like, then I think the argument is an admission that the epistemology can’t do error correction, in which case the epistemology would be incomplete at best and broken at worst.
Sorry for the delayed replay. My comments are being held for review.
Bayesian epistemology typically works in the framework of an existing hypothesis space, with a prior over that space, which is then updated. In addition to updating your credences about the possibilities in the space, you can also reformulate your hypothesis space itself, e.g., because you become aware of new possibilities (like the existence of scammers), or because you want to carve the world into different concepts due to some ontological shift. I think the Bayesian should just be allowed to reformulate their hypothesis space and reform their prior to get out of this.
Okay that sounds reasonable (to me, a non-bayesian) but where do the new hypotheses come from, or the ideas for how to reformulate the space? If they came from Bayesianism, why is reformulation ever necessary?
Humans are not ideal Bayesian reasoning engines with unbounded ability to track all possible hypotheses.
In principle an ideal Bayesian reasoner could deduce the hypotheses of life, resource competition, strategies for obtaining resources, and the possible existence of scams from basically nothing (in addition to myriad other hypotheses). If they’re starting from the existence of markets as implied by being able to understand the letter at all, they could get to scams almost instantly.
So if we’re not ideal bayesian reasoners, how do we make progress? It seems like things need to come from outside bayesianism somehow. Or is it just a lottery that we can’t ever do better at except by having more people actively do bayesian reasoning about stuff? If there is something outside bayesian reasoning/epistemology and new ideas come from there, then it seems like it does things bayesianism does not, or is better at it. And such a thing would either contradict bayesianism at worst or make it (partially) obsolete at best.
Sorry for the delayed replay. My comments are being held for review.
There aren’t just two possibilities “ideal bayesian reasoning” and “useless rubbish”. There is a huge range of heuristics, ad-hoc models, evolved instincts, and everything else in the mix. These are all ‘outside bayesianism’ and while the collection is almost certainly worse than ideal bayesian reasoning, they are not useless.
That also doesn’t mean that we can only improve by having more people actively do bayesian reasoning about stuff, though there are certainly many cases where people would be better off actively doing bayesian reasoning.
There are many ways to improve incredibly complex systems such as human minds and their interactions. It’s far from certain that applying more bayesian reasoning is the best way. We are definitely not capable of reaching the ideal, and will have to settle for something imperfect. Maybe there is a better approximation than “try bayesian reasoning as far as our limited human brains can handle”, maybe there is not.
The main point is that we don’t know anything better, and pretty much everything else that we do know looks worse. However, there is a lot that we don’t know, and far more that we don’t even know that we don’t know.
One thing that is pretty clear is that an ideal bayesian reasoner would distribute some probability across all non-self-contradictory hypotheses that you can express in text of bounded length. There are only finitely many of them, so a failure to include some of them would be a pretty major departure from idealness.
I agree with lots of what you said, but, to focus:
This, I think, is wrong. Not a little bit wrong, but outright wrong, and has been wrong for decades. If you want to believe and repeat this, I can provide some standards I think you should meet before being so confident. I think you should stop claiming it (unless you can answer about why other contenders are wrong, which would be big news philosophically if you could).
That’s a big claim on my part, so I will make a point and then give you some alternatives:
Claim: There is no one who can put forward a specific, integrated version of Bayesianism that will: take responsibility for defending it, put the best and strongest arguments forward, and debate with any philosopher who disagrees.
(Integrated here means it comes with all necessary parts for it to work; e.g., if one needs a model for hypothesis generation, then at the very least there are known fleshed out ways of doing that. Also, debates don’t need to happen twice, we can simply point to an argument made in the ongoing (academic and non-academic) literature and check whether the argument is resolved or not.)
Alternatives:
Critical Rationalism (Karl Popper). CR gave us the basis of modern science (last 100yrs or so) so should be taken as a serious option. Popper tried to engage with bayesians and inductivists but some of his criticisms never received answers.
Critical Fallibilism (Elliot Temple). CF is descended from CR and some other philosophies. It has original and important contributions from Elliot which solve some problems with CR, like eliminating credences completely (Popper tried and failed to get rid of ‘strong’ and ‘weak’ arguments fully, which is one of CF’s criticisms of CR).
Elliot has been critical of LessWrong for a decade plus FWIW and even though points like the one I made above (not original to me) have been brought up to LW for years, there is little-to-no engagement.
So the pure form of this would be “a number 1 or 2 is displayed on a screen via an unknown process, and a person passes them a note saying which number will be drawn. This happens 6 times in a row”. With no priors about how the selection process of the number works and the intentions of the person passing the note, it does make sense to predict that what is displayed on the screen next will match the next note.
Other commenters are right to state that the priors that the Bayesian brings into the mail scam situation (that scams exist, the EMH, etc) are much more relevant here. Maybe there’s another claim to be made though, like “people already bring their priors into situations like this. Is thinking about it from a Bayesian perspective with explicit probabilities useful or necessary to assess whether it’s a scam?” To that, I would say no.
Yeah okay, thanks. I mention in my broad reply here that I am not sure about bringing in the priors because that’s not how nature works. (I agree that it’s the sensible thing for someone IRL to do, but it seems epistemically unreliable and not the situation we find ourselves in generally.)