As a response to the literal comment at top-of-thread, this is clearly reasonable. But I think Eliezer is correctly invoking some important subtext here, which your comment doesn’t properly answer. (I think this because I often make a similar move to the one Eliezer is making, and have only understood within the past couple years what’s load-bearing about it.)
Specifically, there’s an important difference between:
“<person> was wrong about <argument/prediction/etc>, so we should update downward on deferring to their arguments/predictions/etc”, vs
“<person> was wrong about <argument/prediction/etc>, in a way which seems blindingly obvious when we actually think about it, and so is strong evidence that <person> has some systematic problem in the methods they’re using to think (as opposed to just being unlucky with this one argument/prediction/etc)”
Eliezer isn’t saying the first one, he’s saying the second one, and then following it up with a specific model of what is wrong with the thinking-methods in question. He’s invoking bio anchors and that Carlsmith report as examples of systematically terrible thinking, i.e. thinking which is in some sense “obviously” wrong when one is not trying-on-some-level to avoid real-thinking about it, and he’s specifically pointing to desire-to-appear-moderate as the likely primary factor which drove that systematically terrible thinking. He’s not invoking them merely as examples of people being wrong.
“<person> was wrong about <argument/prediction/etc>, in a way which seems blindingly obvious when we actually think about it, and so is strong evidence that <person> has some systematic problem in the methods they’re using to think (as opposed to just being unlucky with this one argument/prediction/etc)”
I think “this was blindingly obvious when we actually think about it”, is not socially admissible evidence, because of hindsight bias.
I thought about a lot of this stuff before 2020. For the most part, I didn’t reach definitive conclusions about a lot of it. In retrospect, a lot of the the conclusions that I did provisionally accept, I, in retrospect, think I was overconfident about, given the epistemic warrant.
Was I doing “actual thing”? No, probably not, or at least not by many relevant standards. Could I have done better? Surely, but not by recourse to magical “just think better” cognition.
The fact remains that It Was Not Obvious To Me.
Others may claim that it was obvious to them, and they might be right—maybe it was obvious to them.
If a person declared operationalized-enough-to-be-gradable prediction before the event was settled, well, then I can update that their worldview made correct predictions.
But if they additionally say that it was obvious and we all should have been able to tell, well, that they say that doesn’t add any additional evidential weight? A person saying “it was blindingly obvious” doesn’t particularly distinguish the world where it actually was obvious, and I would have been able to tell if I had done Actual Thinking, and the world where it was a confusing question about the future that was hard to call in advance, and they happened to get this one right.
If you can show me how it’s obvious, such that it does in fact become obvious to me, that’s a different story.[1] But even then, the time to show that something is obvious is before reality reveals the answer.
It’s fine, I guess, for Eliezer and John to assert that some things were actually obvious, and we all should have been able to tell.
But I (and almost everyone else who didn’t call it as obvious in advance), should pay attention to the correct prediction, and ignore the assertion that it was obvious.
And, to be clear, both Eliezer and John have put in enormous levels of effort into trying to do that kind of communication. I can’t fault you for not attempting to show what you think you know.
Feels like there’s some kind of frame-error here, like you’re complaining that the move in question isn’t using a particular interface, but the move isn’t intended to use that interface in the first place? Can’t quite put my finger on it, but I’ll try to gesture in the right direction.
Consider ye olde philosophers who liked to throw around syllogisms. You and I can look at many of those syllogisms and be like “that’s cute and clever and does not bind to reality at all, that’s not how real-thinking works”. But if we’d been around at the time, very plausibly we would not be able to recognize the failure; maybe we would not have been able to predict in advance that many of the philosophers’ clever syllogisms totally fail to bind to reality.
Nonetheless, it is still useful and instructive to look at those syllogisms and say “look, these things obviously-in-some-sense do not bind to reality, they are not real-thinking, and therefore they are strong evidence that there is something systematically wrong with the thinking-methods of those philosophers”. (Eliezer would probably reflexively follow that up with “so I should figure out what systematic thinking errors plagued those seemingly-bright philosophers, and caused them to deceive themselves with syllogisms, in order to avoid those errors myself”.)
And if there’s some modern-day philosopher standing nearby saying that in fact syllogisms totally do bind to reality… then yeah, this whole move isn’t really a response to them. That’s not really what it’s intended for. But even if one’s goal is to respond to that philosopher, it’s probably still a useful first step to figure out what systematic thinking error causes them to not notice that many of their syllogisms totally fail to bind to reality.
So I guess maybe… Eliezer’s imagined audience here is someone who has already noticed that bio anchors and the Carlsmith thing fail to bind to reality, but you’re criticizing it for not instead responding to a hypothetical audience who thinks that the reports maybe do bind to reality?
So I guess maybe… Eliezer’s imagined audience here is someone who has already noticed that bio anchors and the Carlsmith thing fail to bind to reality, but you’re criticizing it for not instead responding to a hypothetical audience who thinks that the reports maybe do bind to reality?
I almost added a sentence at the end of my comment to the effect of…
“Either someone did that X was blindly obvious, in which case they don’t need to be told, or it wasn’t blindingly obvious to them, and they should should pay attention to the correct prediction, and ignore the assertion that it was obvious. In either case...the statement isn’t doing anything?”
Who are statements like these for? Is it for the people who thought that things were obvious to find and identify each other?
To gesture at a concern I have (which I think is probably orthogonal to what you’re pointing at):
On a first pass, the only people who might be influenced by statements like that are being influenced epistemically illegitimately.
Like, I’m imagining a person, Bob, who heard all the arguments at the time and did not feel confident enough to make a specific prediction. But then we all get to wait a few years and see how (some of the questions, though not most of them) actually played out, and then Eliezer or whoever says “not only was I right, it was blindingly obvious that I was right, and we all should have known all along!”
This is in practice received by Bob as almost an invitation to rewrite history and hindsight bias about what happened. It’s very natural to agree with Eliezer (or whoever) that, “yeah, it was obvious all along.” [1]
And that’s really sus! Bob didn’t get new information or think of new considerations that caused the confusing question to go from confusing to obvious. He just learned the answer!
He should be reminding himself that he didn’t in fact make an advance prediction, and remembering that at the time, it seemed like a confusing hard-to-call question, and analyzing what kinds of general thinking patterns would have allowed him to correctly call this one in advance.
I think when Eliezer gets irate and at people for what he considers their cognitive distortions:
It doesn’t convince the people he’s ostensibly arguing against, because those people don’t share his premises. They often disagree with him, on the object level, about whether the specific conclusions under discussion have been falsified.
(eg Ryan saying he doesn’t think bio ancors was unreasonable, in this thread, or Paul disagreeing with Eliezer claims ~”that people like Paul are surprised by how the world actually plays out.”)
It doesn’t convince the tiny number of people who could see for themselves that those ways of thinking were blindingly obvious (and/or have a shared error pattern with Eliezer, that cause them to be making the same mistake).
(eg John Wentworth)
It does sweep up some social-ideologically doomer-y people into feeling more confidence for their doomerism and related beliefs, both by social proof (Eliezer is so confident and assertive, which makes me feel more comfortable asserting high P(doom)s), and because Eliezer’s setting a frame in which he’s right, and people doing Real Thinking(TM), can see that he’s right, and anyone who doesn’t get it is blinded by frustrating biases.
(eg “Bob”, though I’m thinking of a few specific people.)
It alienates a bunch of onlookers, both people who think that Eliezer is wrong / making a mistake, and people who are agnostic.
In all cases, this seems either unproductive or counterproductive.
Like, there’s some extra psychological omph of just how right Eliezer (or whoever) was and how wrong the other parties were. You get to be on the side of the people who were right all along, against the oppressive forces of OpenPhil’s powerful distortionary forces / the power of modest epistemology / whatever. There’s some story that the irateness invites onlookers like Bob to participate in.
Ok, I think one of the biggest disconnects here is that Eliezer is currently talking in hindsight about what we should learn from past events, and this is and should often be different from what most people could have learned at the time. Again, consider the syllogism example: just because you or I might have been fooled by it at the time does not mean we can’t learn from the obvious-in-some-sense foolishness after the fact. The relevant kind of “obviousness” needs to include obviousness in hindsight for the move Eliezer is making to work, not necessarily obviousness in advance, though it does also need to “obvious” in advance in a different sense (more on that below).
Short handle: “It seems obvious in hindsight that <X> was foolish (not merely a sensible-but-incorrect prediction from insufficient data); why wasn’t that obvious at the time, and what pattern do I need to be on the watch for to make it obvious in the future?”
Eliezer’s application of that pattern to the case at hand goes:
It seems obvious-in-some-sense in hindsight that bio anchors and the Carlsmith thing were foolish, i.e. one can read them and go “man this does seem kind of silly”.
Insofar as that wasn’t obvious at the time, it’s largely because people were selecting for moderate-sounding conclusions. (That’s not the only generalizable pattern which played a role here, but it’s an important one.)
So in the future, I should be on the lookout for the pattern of selecting for moderate-sounding conclusions.
I think an important gear here is that things can be obvious-in-hindsight, but not in advance, in a way which isn’t really a Bayesian update on new evidence and therefore doesn’t strictly follow prediction rules.
Toy example:
Someone publishes a proof of a mathematical conjecture, which enters canon as a theorem.
Some years later, another person stumbles on a counterexample.
Surprised mathematicians go back over the old proof, and indeed find a load-bearing error. Turns out the proof was wrong!
The key point here is that the error was an error of reasoning, not an error of insufficient evidence or anything like that. The error was “obvious” in some sense in advance; a mathematician who’d squinted at the right part of the proof could have spotted it. Yet in practice, it was discovered by evidence arriving, rather than by someone squinting at the proof.
Note that this toy example is exactly the sort where the right primary move to make afterwards is to say “the error is obvious in hindsight, and was obvious-in-some-sense beforehand, even if nobody noticed it. Why the failure, and how do we avoid that in the future?”.
This is very much the thing Eliezer is doing here. He’s (he claims) pointing to a failure of reasoning, not of insufficient evidence. For many people, the arrival of more recent evidence has probably made it more obvious that there was a reasoning failure, and those people are the audience who (hopefully) get value from the move Eliezer made—hopefully they will be able to spot such silly patterns better in the future.
If someone predicts in advance that something is obviously false, and then you come to believe that it’s false, then you should update not just towards thought processes which would have predicted that the thing is false, but also towards thought processes which would have predicted that the thing is obviously false. (Conversely, if they predict that it’s obviously false, and it turns out to be true, you should update more strongly against their thought processes than if they’d just predicted it was false.)
IIRC Eliezer’s objection to bioanchors can be reasonably interpreted as an advance prediction that “it’s obviously false”, though to be confident I’d need to reread his original post (which I can’t be bothered to do right now).
But I (and almost everyone else who didn’t call it as obvious in advance), should pay attention to the correct prediction, and ignore the assertion that it was obvious.
I think this is wrong. The scenarios where this outcome was easily predicted given the right heuristics and the scenarios where this was surprising to every side of the debate are quite different. Knowing who had predictors that worked in this scenario is useful evidence, especially when the debate was about which frames for thinking about things and selecting heuristics were useful.
Or, to put this in simpler but somewhat imprecise terms: This was not obvious to you because you were thinking about things the wrong way. You didn’t know which way to think about things at the time because you lacked information about which predicted things better. You now have evidence about which ways work better, and can copy heuristics from people who were less surprised.
As a response to the literal comment at top-of-thread, this is clearly reasonable. But I think Eliezer is correctly invoking some important subtext here, which your comment doesn’t properly answer. (I think this because I often make a similar move to the one Eliezer is making, and have only understood within the past couple years what’s load-bearing about it.)
Specifically, there’s an important difference between:
“<person> was wrong about <argument/prediction/etc>, so we should update downward on deferring to their arguments/predictions/etc”, vs
“<person> was wrong about <argument/prediction/etc>, in a way which seems blindingly obvious when we actually think about it, and so is strong evidence that <person> has some systematic problem in the methods they’re using to think (as opposed to just being unlucky with this one argument/prediction/etc)”
Eliezer isn’t saying the first one, he’s saying the second one, and then following it up with a specific model of what is wrong with the thinking-methods in question. He’s invoking bio anchors and that Carlsmith report as examples of systematically terrible thinking, i.e. thinking which is in some sense “obviously” wrong when one is not trying-on-some-level to avoid real-thinking about it, and he’s specifically pointing to desire-to-appear-moderate as the likely primary factor which drove that systematically terrible thinking. He’s not invoking them merely as examples of people being wrong.
I think “this was blindingly obvious when we actually think about it”, is not socially admissible evidence, because of hindsight bias.
I thought about a lot of this stuff before 2020. For the most part, I didn’t reach definitive conclusions about a lot of it. In retrospect, a lot of the the conclusions that I did provisionally accept, I, in retrospect, think I was overconfident about, given the epistemic warrant.
Was I doing “actual thing”? No, probably not, or at least not by many relevant standards. Could I have done better? Surely, but not by recourse to magical “just think better” cognition.
The fact remains that It Was Not Obvious To Me.
Others may claim that it was obvious to them, and they might be right—maybe it was obvious to them.
If a person declared operationalized-enough-to-be-gradable prediction before the event was settled, well, then I can update that their worldview made correct predictions.
But if they additionally say that it was obvious and we all should have been able to tell, well, that they say that doesn’t add any additional evidential weight? A person saying “it was blindingly obvious” doesn’t particularly distinguish the world where it actually was obvious, and I would have been able to tell if I had done Actual Thinking, and the world where it was a confusing question about the future that was hard to call in advance, and they happened to get this one right.
If you can show me how it’s obvious, such that it does in fact become obvious to me, that’s a different story.[1] But even then, the time to show that something is obvious is before reality reveals the answer.
It’s fine, I guess, for Eliezer and John to assert that some things were actually obvious, and we all should have been able to tell.
But I (and almost everyone else who didn’t call it as obvious in advance), should pay attention to the correct prediction, and ignore the assertion that it was obvious.
And, to be clear, both Eliezer and John have put in enormous levels of effort into trying to do that kind of communication. I can’t fault you for not attempting to show what you think you know.
Feels like there’s some kind of frame-error here, like you’re complaining that the move in question isn’t using a particular interface, but the move isn’t intended to use that interface in the first place? Can’t quite put my finger on it, but I’ll try to gesture in the right direction.
Consider ye olde philosophers who liked to throw around syllogisms. You and I can look at many of those syllogisms and be like “that’s cute and clever and does not bind to reality at all, that’s not how real-thinking works”. But if we’d been around at the time, very plausibly we would not be able to recognize the failure; maybe we would not have been able to predict in advance that many of the philosophers’ clever syllogisms totally fail to bind to reality.
Nonetheless, it is still useful and instructive to look at those syllogisms and say “look, these things obviously-in-some-sense do not bind to reality, they are not real-thinking, and therefore they are strong evidence that there is something systematically wrong with the thinking-methods of those philosophers”. (Eliezer would probably reflexively follow that up with “so I should figure out what systematic thinking errors plagued those seemingly-bright philosophers, and caused them to deceive themselves with syllogisms, in order to avoid those errors myself”.)
And if there’s some modern-day philosopher standing nearby saying that in fact syllogisms totally do bind to reality… then yeah, this whole move isn’t really a response to them. That’s not really what it’s intended for. But even if one’s goal is to respond to that philosopher, it’s probably still a useful first step to figure out what systematic thinking error causes them to not notice that many of their syllogisms totally fail to bind to reality.
So I guess maybe… Eliezer’s imagined audience here is someone who has already noticed that bio anchors and the Carlsmith thing fail to bind to reality, but you’re criticizing it for not instead responding to a hypothetical audience who thinks that the reports maybe do bind to reality?
I almost added a sentence at the end of my comment to the effect of…
“Either someone did that X was blindly obvious, in which case they don’t need to be told, or it wasn’t blindingly obvious to them, and they should should pay attention to the correct prediction, and ignore the assertion that it was obvious. In either case...the statement isn’t doing anything?”
Who are statements like these for? Is it for the people who thought that things were obvious to find and identify each other?
To gesture at a concern I have (which I think is probably orthogonal to what you’re pointing at):
On a first pass, the only people who might be influenced by statements like that are being influenced epistemically illegitimately.
Like, I’m imagining a person, Bob, who heard all the arguments at the time and did not feel confident enough to make a specific prediction. But then we all get to wait a few years and see how (some of the questions, though not most of them) actually played out, and then Eliezer or whoever says “not only was I right, it was blindingly obvious that I was right, and we all should have known all along!”
This is in practice received by Bob as almost an invitation to rewrite history and hindsight bias about what happened. It’s very natural to agree with Eliezer (or whoever) that, “yeah, it was obvious all along.” [1]
And that’s really sus! Bob didn’t get new information or think of new considerations that caused the confusing question to go from confusing to obvious. He just learned the answer!
He should be reminding himself that he didn’t in fact make an advance prediction, and remembering that at the time, it seemed like a confusing hard-to-call question, and analyzing what kinds of general thinking patterns would have allowed him to correctly call this one in advance.
I think when Eliezer gets irate and at people for what he considers their cognitive distortions:
It doesn’t convince the people he’s ostensibly arguing against, because those people don’t share his premises. They often disagree with him, on the object level, about whether the specific conclusions under discussion have been falsified.
(eg Ryan saying he doesn’t think bio ancors was unreasonable, in this thread, or Paul disagreeing with Eliezer claims ~”that people like Paul are surprised by how the world actually plays out.”)
It doesn’t convince the tiny number of people who could see for themselves that those ways of thinking were blindingly obvious (and/or have a shared error pattern with Eliezer, that cause them to be making the same mistake).
(eg John Wentworth)
It does sweep up some social-ideologically doomer-y people into feeling more confidence for their doomerism and related beliefs, both by social proof (Eliezer is so confident and assertive, which makes me feel more comfortable asserting high P(doom)s), and because Eliezer’s setting a frame in which he’s right, and people doing Real Thinking(TM), can see that he’s right, and anyone who doesn’t get it is blinded by frustrating biases.
(eg “Bob”, though I’m thinking of a few specific people.)
It alienates a bunch of onlookers, both people who think that Eliezer is wrong / making a mistake, and people who are agnostic.
In all cases, this seems either unproductive or counterproductive.
Like, there’s some extra psychological omph of just how right Eliezer (or whoever) was and how wrong the other parties were. You get to be on the side of the people who were right all along, against the oppressive forces of OpenPhil’s powerful distortionary forces / the power of modest epistemology / whatever. There’s some story that the irateness invites onlookers like Bob to participate in.
Ok, I think one of the biggest disconnects here is that Eliezer is currently talking in hindsight about what we should learn from past events, and this is and should often be different from what most people could have learned at the time. Again, consider the syllogism example: just because you or I might have been fooled by it at the time does not mean we can’t learn from the obvious-in-some-sense foolishness after the fact. The relevant kind of “obviousness” needs to include obviousness in hindsight for the move Eliezer is making to work, not necessarily obviousness in advance, though it does also need to “obvious” in advance in a different sense (more on that below).
Short handle: “It seems obvious in hindsight that <X> was foolish (not merely a sensible-but-incorrect prediction from insufficient data); why wasn’t that obvious at the time, and what pattern do I need to be on the watch for to make it obvious in the future?”
Eliezer’s application of that pattern to the case at hand goes:
It seems obvious-in-some-sense in hindsight that bio anchors and the Carlsmith thing were foolish, i.e. one can read them and go “man this does seem kind of silly”.
Insofar as that wasn’t obvious at the time, it’s largely because people were selecting for moderate-sounding conclusions. (That’s not the only generalizable pattern which played a role here, but it’s an important one.)
So in the future, I should be on the lookout for the pattern of selecting for moderate-sounding conclusions.
I think an important gear here is that things can be obvious-in-hindsight, but not in advance, in a way which isn’t really a Bayesian update on new evidence and therefore doesn’t strictly follow prediction rules.
Toy example:
Someone publishes a proof of a mathematical conjecture, which enters canon as a theorem.
Some years later, another person stumbles on a counterexample.
Surprised mathematicians go back over the old proof, and indeed find a load-bearing error. Turns out the proof was wrong!
The key point here is that the error was an error of reasoning, not an error of insufficient evidence or anything like that. The error was “obvious” in some sense in advance; a mathematician who’d squinted at the right part of the proof could have spotted it. Yet in practice, it was discovered by evidence arriving, rather than by someone squinting at the proof.
Note that this toy example is exactly the sort where the right primary move to make afterwards is to say “the error is obvious in hindsight, and was obvious-in-some-sense beforehand, even if nobody noticed it. Why the failure, and how do we avoid that in the future?”.
This is very much the thing Eliezer is doing here. He’s (he claims) pointing to a failure of reasoning, not of insufficient evidence. For many people, the arrival of more recent evidence has probably made it more obvious that there was a reasoning failure, and those people are the audience who (hopefully) get value from the move Eliezer made—hopefully they will be able to spot such silly patterns better in the future.
If someone predicts in advance that something is obviously false, and then you come to believe that it’s false, then you should update not just towards thought processes which would have predicted that the thing is false, but also towards thought processes which would have predicted that the thing is obviously false. (Conversely, if they predict that it’s obviously false, and it turns out to be true, you should update more strongly against their thought processes than if they’d just predicted it was false.)
IIRC Eliezer’s objection to bioanchors can be reasonably interpreted as an advance prediction that “it’s obviously false”, though to be confident I’d need to reread his original post (which I can’t be bothered to do right now).
I think this is wrong. The scenarios where this outcome was easily predicted given the right heuristics and the scenarios where this was surprising to every side of the debate are quite different. Knowing who had predictors that worked in this scenario is useful evidence, especially when the debate was about which frames for thinking about things and selecting heuristics were useful.
Or, to put this in simpler but somewhat imprecise terms: This was not obvious to you because you were thinking about things the wrong way. You didn’t know which way to think about things at the time because you lacked information about which predicted things better. You now have evidence about which ways work better, and can copy heuristics from people who were less surprised.