[Unsure] The probability of GR being true is independent of whether the Bayesian knows about it or not[1]
Keep in mind that these “probabilities” are subjective assessments of probability based on an individual’s prior knowledge, not facts about reality. Two Bayesians with different prior experience may disagree about how probable something is (/seems to them), but reality will not disagree or debate with itself about the truth of the matter, or assign probability to different possibilities (mumble mumble I don’t really understand quantum mechanics and am pretending it doesn’t matter for the purpose of this conversation).
Whether or not General Relativity is true is unaffected by any probabilities any Bayesian may put on its truth or falsehood when reasoning about the evidence they’ve seen so far. But whether or not a particular Bayesian finds General Relativity to be probably true, is definitely affected by whether they know about it or not. Keep a clear distinction in your mind between “the probability a Bayesian reasoner assigns to some fact being true” and “whether or not that fact is true in reality”—these are not the same.
This is a “map vs. territory” distinction. Bayesian probabilities go into a mental model of how the world works (map), while how the world actually works is separate from the map.
Keep in mind that these “probabilities” are subjective assessments of probability based on an individual’s prior knowledge, not facts about reality. Two Bayesians with different prior experience may disagree about how probable something is (/seems to them),
Doesn’t that mean we should expect that Bayesians often disagree and they have no way to resolve it except consulting reality (i.e., an experiment)?
If that’s the case, why bother with Bayesianism at all? It seems like any situation where people needed to agree on the truth of something, Bayesianism wouldn’t help:
They already agree (‘scientific consensus’ and similar situations) -- Bayesianism doesn’t offer anything here.
One person disagrees, everyone else agrees—without more data no progress is possible, so either we know a decisive experiment that will resolve the issue (in which case we don’t need Bayesianism), or we don’t, in which case Bayesianism can’t help.
Lots of disagreement—similar to above but more chaotic.
Also say there is an experiment, is there any standard or agreement among bayesians about how to weight credence? (when should it be weak or strong? etc) Because if there isn’t, they might not even be able to agree on what experiment to do or if it will matter.
The FDA just decided to allow Bayesian statistics to be used to argue for drug approval. If a drug company wants approval they will need to state their priors and then how they updated based on the studies they ran. If the FDA does not agree, they might say “Well, I don’t think those priors are reasonable, please use X as priors”. I expect that the exact rules of how the process goes will evolve with time.
On another practical application space, you have superforcasting where credence calibration is important. You make progress by believing people with better Brier’s scores (or log loss). If some scientists in a field can make predictions about experimental outcomes with better Brier’s scores, than they seem to understand more about the domain and should be trusted.
If so, I think we might be talking cross-purposes. I’m asking about bayesian epistemology particularly, not statistics. (Nor about estimations based on other people’s beliefs since that doesn’t seem like an epistemology except maybe JTB with more steps)
If not, I’m not sure what your point is particularly.
Sorry for the delayed replay. My comments are being held for review.
The question “How do we know which drugs work and are beneficial to patients?” is an applied epistemology question. Looking at how it gets answered by a sophisticated system tells you how epistemology actually works in practice instead of how philosophers think it’s supposed to work in their ivory tower. If you use Bayesian statistics you want an epistemology behind that use that guides you in how you use the statistics to reason.
Superforcasting is much more about Bayesian epistemology than about Bayesian statistics. You have Superforcasters who would they they are Bayesians but can’t write down Bayes theorem.
You seem to have some idea that epistemology is supposed to be “objective”. It’s supposed to give you the answer from God’s view. A lot of Western science is build around wanting to reach God’s view. The problem is that God doesn’t exist. According to Nietzsche, he’s dying. Bayesian epistemology is an epistomology without God, which means that you have to deal with your beliefs and other peoples beliefs.
The reason to bother with Bayesianism is not because it helps you to see the world from God’s view but because it has practical utility in applied epistemology with FDA drug approval and Superforcasting being two examples.
You seem to have some idea that epistemology is supposed to be “objective”. It’s supposed to give you the answer from God’s view.
Objective in a sense, but I’m not sure how I’ve given you the ‘God’s view’ impression. I think epistemology should be objective in that it should work universally via the same rules, and that people can discuss both ideas and the world (evidence) in such a way to reach agreement in an objective sense. But they can be wrong, there’s no infallible method of getting to the truth. They can also objectively agree on each other’s subjective states.
The reason to bother with Bayesianism is not because it helps you to see the world from God’s view but because it has practical utility in applied epistemology with FDA drug approval and Superforcasting being two examples.
Putting aside superforecasting because I don’t know much about it, using bayesian statistics for statistical analysis is fine.
But that’s not what’s happening on LW. When people on LW talk about their priors and updating them, they’re not talking about bayesian statistics, they’re talking about epistemology, about what ideas are true. I think those are fundamentally different things and they work in different ways (and it seems like you do, too). I’m here because LW is the largest bayesian forum (or if not the largest it’s better than reddit for discussion, point is, it’s the best option for talking to bayesians).
The idea that we should apply bayesian statistics to epistemic tasks is what I’m interested in discussing.
If superforecasting is important to discuss, can you link me to something that represents what you mean by it?
If you use frequentist statistics you can just claim that your data follows a normal distribution (which it objectively most likely isn’t, even the archtypical example of height violates a normal distribution because there are more people with dwarfism than the normal distribution assumes). On the other hand, to use Bayesian statistics you do need to decide on priors which does involve subjective decisions.
Objective in a sense, but I’m not sure how I’ve given you the ‘God’s view’ impression. I think epistemology should be objective in that it should work universally via the same rules, and that people can discuss both ideas and the world (evidence) in such a way to reach agreement in an objective sense.
The fact that you aren’t explicitly thinking of God doesn’t mean that the idea of objectivity does not come out of a Christian scientific tradition which had as a key motivation trying to see things from God’s view. Your ideas of how you think, it should work have that theistic origin.
Bayesianism as discussed on LessWrong is about how it would be good for an agent to reason and that’s a different goal. Eliezer was interested in it because he wants to know how what’s true about how agents effectively reason to be able to say things about superintelligence.
Philip E. Tetlock separately was interested in the epistemic task about how to reason about what to believe. After the Iraq war the US military thought they had problems with epistomology as shown by the fact that they got the WMD question so horribly wrong. Out of that there was an IARPA tournament where Tetlock’s team won. Tetlock runs GJOpen which does solve epistemic problems for entities that want probability for certain events happening. He got some grant money from OpenPhil. There’s also Metaculus that comes out of rationalist sphere. His book Superforcastingis a good resource if you want to understand how the kind of Bayesianism where people don’t explicitly use statistics works in practice.
Doesn’t that mean we should expect that Bayesians often disagree and they have no way to resolve it except consulting reality (i.e., an experiment)?
Short answer: Yes.
Longer answer: Two Bayesians who start out with the same prior probabilities, and see the same evidence, should update their posterior probabilities in the same way, and so their mental models should stay consistent with each other. Two Bayesians who start out with different prior probabilities, but see the same evidence, should update their posterior probabilities in ways that are predictable to each other, and in line with the evidence—that is, if one reasoner (A)‘s prior probability that (for example) General Relativity is true was high, while another (B)’s was low, then when an experiment is run which provides evidence for general relativity, A’s estimates of General Relativity’s likelihood of being true will change less than B’s (because B’s priors were more wrong), but both will update in a direction and to an extent that is predictable to either of them. As they see more and more of the same evidence, their models of the world should converge.
This is all assuming an ideal Bayesian reasoner with practically-unlimited computing power who doesn’t cheat or decide not to reason according to Bayesian rules when it becomes inconvenient, and humans don’t meet those constraints. But, there’s math to say how much you should update given particular evidence. So:
Also say there is an experiment, is there any standard or agreement among bayesians about how to weight credence?
Yep. “How to weight credence” is a bit unclearly stated, but there’s Bayes’ formula, which tells you how to update your probabilities based on evidence, and that might be what you’re getting at?
Which is (one reason) why bother with Bayesianism at all. It’s a method of approaching consensus when working under uncertainty. It’s kind of an “agreeing to the rules of the game” situation, where “the rules” are a mathematical equation that says how probabilities must change when people are disagreeing (and “must” here carries the same level of mathematical strength as saying “2+2 must equal 4″, it’s not a thing that was decided by committee) - if for example you say it’s 95% unlikely/5% likely that something will happen under your idea of how the world works, and then it happens, if you’re playing fair, you make a big update, and if you put numbers on it, Bayes’ rule tells you what your new numbers should be. If you don’t like the new numbers, you have to either acknowledge that what you said your priors were was incorrect, or that what you said your likelihood estimates of different outcomes were was incorrect—so either you retroactively revise how you used to think the world works, or you retroactively revise what you thought would happen and how confident you were, both of which are kind of awkward and embarrassing. And the people you’re disagreeing with, if you don’t make an appropriately sized update given what you told them your priors and likelihood estimates were, can point this out as a fact. And if you’re not very confident, or you don’t think particular evidence should carry much weight, you can express that in a way that makes it clear how much you’re going to update based on whatever evidence you see, before you see it, so it isn’t like:
“I think x is definitely wrong, and y will provide strong evidence” ″OK, so y didn’t go how I thought, now I think x is only almost certain to be false”
It’s like:
“I think X is a% likely to be false, and Y experiment will turn out the way I expect with b% likelihood as a result.”
“Oh. Ok, well, I guess now I’m down to c% likelihood that X is false. Shoot.”
And instead of being like “it’s not fair that you moved from definitely to almost certain based on something you said would provide strong evidence but didn’t go the way you expected”, the reaction is “yep, that math is correct, you updated how you should have given your priors”. And before you get to that point, pre-experiement, you can argue over whether b% likelihood is reasonable, where it’s hard to argue about the correct meaning of the word “strong”.
And once you get really familiar with doing this (I’m still not great at it), you know intuitively how much putting X% probability on a particular outcome means you’re going to have to change your views if it doesn’t happen, and you become appropriately cautious (“calibrated”) in your estimates, and your saying things in probabilities conveys a lot of information to other people who are also familiar with talking this way.
All of that is in idealized theory among people who are quite smart and can do lots of calculation in their heads. Lots of people also LARP it and use Bayesian-sounding words without actually having the deeper intuitive understanding of what what they’re saying means.
Doesn’t that mean we should expect that Bayesians often disagree and they have no way to resolve it except consulting reality (i.e., an experiment)?
Short answer: Yes.
Okay.
FWIW, CF says that two people should be able to use discussion to resolve most disagreements. Some disagreements do require experiments, but only when we have multiple contradictory theories that otherwise agree. (e.g., UG and GR mostly agree so it’s reasonable that two scientists might disagree on which is correct but they should be able to agree on a decisive experiment.) When such discussions fail there are ways to deal with that, too (so you can go address the reason for failure then resume the original discussion).
[...] that is, if one reasoner (A)‘s prior probability that (for example) General Relativity is true was high, while another (B)’s was low, then when an experiment is run which provides evidence for general relativity, A’s estimates of General Relativity’s likelihood of being true will change less than B’s (because B’s priors were more wrong) [...]
Doesn’t that require some assumptions about their hypothesis spaces? You assume that B’s priors are more wrong, but we don’t know that from just saying that [1]. We need to know about all their other priors, too.
I agree the update can be predictable to one another if they specify and communicate the necessary constants / weightings / etc. re specification and communication: For something that seems important to bayesians I’m surprised that I don’t see more of that.
As they see more and more of the same evidence, their models of the world should converge.
I’m not convinced of this but I agree it seems that intuitively, most of the time at least, they should.
This is all assuming an ideal Bayesian reasoner with practically-unlimited computing power who doesn’t cheat or decide not to reason according to Bayesian rules when it becomes inconvenient.
This surprised me to read and I have some immediate questions.
When is Bayesian reasoning inconvenient?
Do you think there’s other valid ways to reason?
If so, why do we need bayesianism? Is Bayesianism incomplete?
Or if not, how would an ideal Bayesian reasoner ever be fooled into using bad/broken/misleading methods of reasoning?
Are there other more energy efficient ways of reasoning to get to the same answers reliably? Why use bayesianism at all in that case?
Is cheating ever consistent with bayesianism?
Which is (one reason) why bother with Bayesianism at all. It’s a method of approaching consensus when working under uncertainty.
Sure, but so is “do whatever Max says”. Consensus on its own has nothing to do with truth. Now, I guess you mean something a little more specific but I don’t want to put words in your mouth.
Here’s an example of how Bayesianism fails to produce consensus (or it seems like that to me):
Two aliens (A1, A2) who do not know about relativity are communicating. A1 observes an event E1 happens before E2, but A2 observes E2 before E1. They will each update in different directions.
Is there a Bayesian way to resolve this?
[...] if you’re not very confident, [...]
How does Bayesianism deal with confidence? Do all priors have errors attached? If so, how are errors handled in the maths?
Also, if the error bars are large enough, is convergence still guaranteed? Or what if our errors are off due to things like overconfidence?
[...] you can argue over whether b% likelihood is reasonable [...]
I’m not sure how this would help Bayesians, what does a successful discussion of this kind look like?
The reason I’m not sure is unanswered questions like what are they observing to cause updates? and which priors are being updated?
Sorry for the delayed replay. My comments are being held for review.
Keep in mind that these “probabilities” are subjective assessments of probability based on an individual’s prior knowledge, not facts about reality. Two Bayesians with different prior experience may disagree about how probable something is (/seems to them), but reality will not disagree or debate with itself about the truth of the matter, or assign probability to different possibilities (mumble mumble I don’t really understand quantum mechanics and am pretending it doesn’t matter for the purpose of this conversation).
Whether or not General Relativity is true is unaffected by any probabilities any Bayesian may put on its truth or falsehood when reasoning about the evidence they’ve seen so far. But whether or not a particular Bayesian finds General Relativity to be probably true, is definitely affected by whether they know about it or not. Keep a clear distinction in your mind between “the probability a Bayesian reasoner assigns to some fact being true” and “whether or not that fact is true in reality”—these are not the same.
This is a “map vs. territory” distinction. Bayesian probabilities go into a mental model of how the world works (map), while how the world actually works is separate from the map.
Doesn’t that mean we should expect that Bayesians often disagree and they have no way to resolve it except consulting reality (i.e., an experiment)?
If that’s the case, why bother with Bayesianism at all? It seems like any situation where people needed to agree on the truth of something, Bayesianism wouldn’t help:
They already agree (‘scientific consensus’ and similar situations) -- Bayesianism doesn’t offer anything here.
One person disagrees, everyone else agrees—without more data no progress is possible, so either we know a decisive experiment that will resolve the issue (in which case we don’t need Bayesianism), or we don’t, in which case Bayesianism can’t help.
Lots of disagreement—similar to above but more chaotic.
Also say there is an experiment, is there any standard or agreement among bayesians about how to weight credence? (when should it be weak or strong? etc) Because if there isn’t, they might not even be able to agree on what experiment to do or if it will matter.
The FDA just decided to allow Bayesian statistics to be used to argue for drug approval. If a drug company wants approval they will need to state their priors and then how they updated based on the studies they ran. If the FDA does not agree, they might say “Well, I don’t think those priors are reasonable, please use X as priors”. I expect that the exact rules of how the process goes will evolve with time.
On another practical application space, you have superforcasting where credence calibration is important. You make progress by believing people with better Brier’s scores (or log loss). If some scientists in a field can make predictions about experimental outcomes with better Brier’s scores, than they seem to understand more about the domain and should be trusted.
Are you responding to this? (which I said above)
If so, I think we might be talking cross-purposes. I’m asking about bayesian epistemology particularly, not statistics. (Nor about estimations based on other people’s beliefs since that doesn’t seem like an epistemology except maybe JTB with more steps)
If not, I’m not sure what your point is particularly.
Sorry for the delayed replay. My comments are being held for review.
The question “How do we know which drugs work and are beneficial to patients?” is an applied epistemology question. Looking at how it gets answered by a sophisticated system tells you how epistemology actually works in practice instead of how philosophers think it’s supposed to work in their ivory tower. If you use Bayesian statistics you want an epistemology behind that use that guides you in how you use the statistics to reason.
Superforcasting is much more about Bayesian epistemology than about Bayesian statistics. You have Superforcasters who would they they are Bayesians but can’t write down Bayes theorem.
You seem to have some idea that epistemology is supposed to be “objective”. It’s supposed to give you the answer from God’s view. A lot of Western science is build around wanting to reach God’s view. The problem is that God doesn’t exist. According to Nietzsche, he’s dying. Bayesian epistemology is an epistomology without God, which means that you have to deal with your beliefs and other peoples beliefs.
The reason to bother with Bayesianism is not because it helps you to see the world from God’s view but because it has practical utility in applied epistemology with FDA drug approval and Superforcasting being two examples.
Objective in a sense, but I’m not sure how I’ve given you the ‘God’s view’ impression. I think epistemology should be objective in that it should work universally via the same rules, and that people can discuss both ideas and the world (evidence) in such a way to reach agreement in an objective sense. But they can be wrong, there’s no infallible method of getting to the truth. They can also objectively agree on each other’s subjective states.
Putting aside superforecasting because I don’t know much about it, using bayesian statistics for statistical analysis is fine.
But that’s not what’s happening on LW. When people on LW talk about their priors and updating them, they’re not talking about bayesian statistics, they’re talking about epistemology, about what ideas are true. I think those are fundamentally different things and they work in different ways (and it seems like you do, too). I’m here because LW is the largest bayesian forum (or if not the largest it’s better than reddit for discussion, point is, it’s the best option for talking to bayesians).
The idea that we should apply bayesian statistics to epistemic tasks is what I’m interested in discussing.
If superforecasting is important to discuss, can you link me to something that represents what you mean by it?
If you use frequentist statistics you can just claim that your data follows a normal distribution (which it objectively most likely isn’t, even the archtypical example of height violates a normal distribution because there are more people with dwarfism than the normal distribution assumes). On the other hand, to use Bayesian statistics you do need to decide on priors which does involve subjective decisions.
The fact that you aren’t explicitly thinking of God doesn’t mean that the idea of objectivity does not come out of a Christian scientific tradition which had as a key motivation trying to see things from God’s view. Your ideas of how you think, it should work have that theistic origin.
Bayesianism as discussed on LessWrong is about how it would be good for an agent to reason and that’s a different goal. Eliezer was interested in it because he wants to know how what’s true about how agents effectively reason to be able to say things about superintelligence.
Philip E. Tetlock separately was interested in the epistemic task about how to reason about what to believe. After the Iraq war the US military thought they had problems with epistomology as shown by the fact that they got the WMD question so horribly wrong. Out of that there was an IARPA tournament where Tetlock’s team won. Tetlock runs GJOpen which does solve epistemic problems for entities that want probability for certain events happening. He got some grant money from OpenPhil. There’s also Metaculus that comes out of rationalist sphere. His book Superforcasting is a good resource if you want to understand how the kind of Bayesianism where people don’t explicitly use statistics works in practice.
I’m not sure what the best shorter source is but maybe https://www.gjopen.com/training/
Short answer: Yes.
Longer answer: Two Bayesians who start out with the same prior probabilities, and see the same evidence, should update their posterior probabilities in the same way, and so their mental models should stay consistent with each other. Two Bayesians who start out with different prior probabilities, but see the same evidence, should update their posterior probabilities in ways that are predictable to each other, and in line with the evidence—that is, if one reasoner (A)‘s prior probability that (for example) General Relativity is true was high, while another (B)’s was low, then when an experiment is run which provides evidence for general relativity, A’s estimates of General Relativity’s likelihood of being true will change less than B’s (because B’s priors were more wrong), but both will update in a direction and to an extent that is predictable to either of them. As they see more and more of the same evidence, their models of the world should converge.
This is all assuming an ideal Bayesian reasoner with practically-unlimited computing power who doesn’t cheat or decide not to reason according to Bayesian rules when it becomes inconvenient, and humans don’t meet those constraints. But, there’s math to say how much you should update given particular evidence. So:
Yep. “How to weight credence” is a bit unclearly stated, but there’s Bayes’ formula, which tells you how to update your probabilities based on evidence, and that might be what you’re getting at?
Which is (one reason) why bother with Bayesianism at all. It’s a method of approaching consensus when working under uncertainty. It’s kind of an “agreeing to the rules of the game” situation, where “the rules” are a mathematical equation that says how probabilities must change when people are disagreeing (and “must” here carries the same level of mathematical strength as saying “2+2 must equal 4″, it’s not a thing that was decided by committee) - if for example you say it’s 95% unlikely/5% likely that something will happen under your idea of how the world works, and then it happens, if you’re playing fair, you make a big update, and if you put numbers on it, Bayes’ rule tells you what your new numbers should be. If you don’t like the new numbers, you have to either acknowledge that what you said your priors were was incorrect, or that what you said your likelihood estimates of different outcomes were was incorrect—so either you retroactively revise how you used to think the world works, or you retroactively revise what you thought would happen and how confident you were, both of which are kind of awkward and embarrassing. And the people you’re disagreeing with, if you don’t make an appropriately sized update given what you told them your priors and likelihood estimates were, can point this out as a fact. And if you’re not very confident, or you don’t think particular evidence should carry much weight, you can express that in a way that makes it clear how much you’re going to update based on whatever evidence you see, before you see it, so it isn’t like:
“I think x is definitely wrong, and y will provide strong evidence”
″OK, so y didn’t go how I thought, now I think x is only almost certain to be false”
It’s like:
“I think X is a% likely to be false, and Y experiment will turn out the way I expect with b% likelihood as a result.”
“Oh. Ok, well, I guess now I’m down to c% likelihood that X is false. Shoot.”
And instead of being like “it’s not fair that you moved from definitely to almost certain based on something you said would provide strong evidence but didn’t go the way you expected”, the reaction is “yep, that math is correct, you updated how you should have given your priors”. And before you get to that point, pre-experiement, you can argue over whether b% likelihood is reasonable, where it’s hard to argue about the correct meaning of the word “strong”.
And once you get really familiar with doing this (I’m still not great at it), you know intuitively how much putting X% probability on a particular outcome means you’re going to have to change your views if it doesn’t happen, and you become appropriately cautious (“calibrated”) in your estimates, and your saying things in probabilities conveys a lot of information to other people who are also familiar with talking this way.
All of that is in idealized theory among people who are quite smart and can do lots of calculation in their heads. Lots of people also LARP it and use Bayesian-sounding words without actually having the deeper intuitive understanding of what what they’re saying means.
Okay.
FWIW, CF says that two people should be able to use discussion to resolve most disagreements. Some disagreements do require experiments, but only when we have multiple contradictory theories that otherwise agree. (e.g., UG and GR mostly agree so it’s reasonable that two scientists might disagree on which is correct but they should be able to agree on a decisive experiment.) When such discussions fail there are ways to deal with that, too (so you can go address the reason for failure then resume the original discussion).
Doesn’t that require some assumptions about their hypothesis spaces? You assume that B’s priors are more wrong, but we don’t know that from just saying that
I agree the update can be predictable to one another if they specify and communicate the necessary constants / weightings / etc. re specification and communication: For something that seems important to bayesians I’m surprised that I don’t see more of that.
I’m not convinced of this but I agree it seems that intuitively, most of the time at least, they should.
This surprised me to read and I have some immediate questions.
When is Bayesian reasoning inconvenient?
Do you think there’s other valid ways to reason?
If so, why do we need bayesianism? Is Bayesianism incomplete?
Or if not, how would an ideal Bayesian reasoner ever be fooled into using bad/broken/misleading methods of reasoning?
Are there other more energy efficient ways of reasoning to get to the same answers reliably? Why use bayesianism at all in that case?
Is cheating ever consistent with bayesianism?
Sure, but so is “do whatever Max says”. Consensus on its own has nothing to do with truth. Now, I guess you mean something a little more specific but I don’t want to put words in your mouth.
Here’s an example of how Bayesianism fails to produce consensus (or it seems like that to me):
Two aliens (A1, A2) who do not know about relativity are communicating. A1 observes an event E1 happens before E2, but A2 observes E2 before E1. They will each update in different directions.
Is there a Bayesian way to resolve this?
How does Bayesianism deal with confidence? Do all priors have errors attached? If so, how are errors handled in the maths?
Also, if the error bars are large enough, is convergence still guaranteed? Or what if our errors are off due to things like overconfidence?
I’m not sure how this would help Bayesians, what does a successful discussion of this kind look like?
The reason I’m not sure is unanswered questions like what are they observing to cause updates? and which priors are being updated?
Sorry for the delayed replay. My comments are being held for review.
even if we assume GR is absolutely 100% correct, and IRL we know that GR isn’t 100% correct.