What lesson am I supposed to learn from “That Alien Message”? It’s a work of fiction. You do not generalize from fictional evidence. Maybe I should write a story about how slow a takeoff would be given the massive inefficiencies of present technology, all the trivial and mundane ways an AI in the midst of a takeoff would get tripped up and caught, and all the different ways redundant detection mechanisms, honey pots, and fail safe contraptions would prevent existential risk scenarios? But such a work of fiction would be just as invalid as evidence.
Ok, I’m still confused as to many of my questions, but let me see if this bit sounds right: the only parameter via which something like “That Alien Message” could become more persuasive to you is by being less fictional. Fictional accounts of anything will NEVER cause you to update your beliefs. Does that sound right?
If that’s right, then I want to suggest why such things should sometimes be persuasive. A perfect Bayesian reasoner with finite computational ability operates not just with uncertainty about the outside world, but also with logical uncertainty as to consequences of their beliefs. So as humans, we operate with at least that difficulty when dealing with our own beliefs. In practice we deal with much much much worse.
I believe the correct form of the deduction your trying to make is “don’t add a fictional story to the reference class of a real analogue for purposes of figuring your beliefs”, and I agree. However, there are other ways a fictional story can be persuasive and should (in my view) cause you to update your beliefs:
It illustrates a new correct deduction which you weren’t aware of before, whose consequences you then begin working out.
It reminds you of a real experience you’ve had, which was not present in your mind before, whose existence then figures into your reference classes.
It changes your emotional attitude toward something, indirectly changing your beliefs by causing you to reflect on that thing differently in the future.
Some of these are subject to biases which would need correcting to move toward better reasoning, but I perceive you as claiming that these should have no impact, ever. Am I interpreting that correctly(I’m going to guess that I’m not somewhere), and if so why do you think that?
I think it’s a pretty big assumption to assume that fictional stories typically do those things correctly. Fictional stories are, after all, produced by people with agendas. If the proportion of fictional stories with plausible but incorrect deductions, reminders, or reflections is big enough, even your ability to figure out which ones are correct might not make it worthwhile to use fiction this way.
(Consider an extreme case where you can correctly assess 95% of the time whether a fictional deduction, reminder, or reflection is correct, but they are incorrect at a 99% rate. You’d have about a 4⁄5 chance of being wrong if you update based on fiction.)
Agreed; you’d have to figure all of that out separately. For what it’s worth, given the selection of fictional stories I’m usually exposed to and decide to read, I think they’re generally positive value (though probably not the best in terms of opportunity cost.)
If a story or thought experiment prompts you to think of some existing data you hadn’t paid attention to, and realize that data was not anticipated by your present beliefs, then that data acts as evidence for updating beliefs. The story or thought experiment was merely a reference used to call attention to that data.
“Changing your emotional attitude” as far as I can tell is actually cache-flushing. It does not change your underlying beliefs, it just trains your emotional response to align with those beliefs, eliminating inconsistencies in thought.
I’m not sure where “that alien message” is supposed to lie in either of those two categories. It makes no reference to actual experimental data which I may not have been paying attention to, nor do I detect any inconsistency it is unraveling. Rather, it makes a ton of assumptions and then runs with those assumptions, when in fact those assumptions were not valid in the first place. It’s a cathedral built on sand.
Basically agreed on paragraph 1, but I do want to suggest that then we not say “I will never update on fictional stories.” Taken naively, you then might avoid fictional stories because they’re useless (“I never update on them!”), when of course they might be super useful if they cause you to pull up relevant experiences quite often.
I’ll make an example of how “That Alien Message” could do for me what I illustrated in my 1st bullet point. I think, “Oh, it seems very unlikely that an AI could break out of a box, you just have this shutoff switch and watch it closely and …”. Then That Alien Message suggests the thought experiment of “instead of generalizing over all AI, instead imagine just a highly specific type of AI that may or may not reasonably come to exist: a bunch of really sped up, smart humans.” Then it sparks my thinking for what a bunch of really sped up, smart humans could accomplish with even narrow channels of communication. Then I think “actually, though I’ve seen no new instances of the AI reference class in reality, I now reason differently about how a possible AI could behave since that class (of possibilities) includes the the thing I just thought of. Until I get a lot more information about how that class behaves in reality, I’m going to be a lot more cautious.”
By picking out a specific possible example, it illustrates that my thinking around the possible AI reference class wasn’t expansive enough. This could help break through, for example, an accessibility heuristic: when I think of a random AI, I think of my very concrete vision of how such an AI would behave, instead of really trying to think about what could lie in that huge space.
Perhaps you are already appropriately cautious, and this story sparks/sparked no new thoughts in you, or you have a good reason to believe that communities of sped up humans or anything at least as powerful are excluded from the reference space, or the reference space you care about is narrower, but it seemed like you were making a stronger statement that such stories will never have any impact on you.
Creative / clever thinking is good. It’s where new ideas come from. Practicing creative thinking by reading interesting stories is not a waste of time. Updating based on creative / clever thoughts, on the other hand, is a mistake. The one almost-exception I can think of is “X is impossible!” where a clever plan for doing X, even if not actually implemented, suffices as weak evidence that “X is impossible!” is false. Or rather, it should propagate up your belief hierarchy and make you revisit why you thought X was impossible in the first place. Because the two remaining options are: (1) you were mistaken about the plausibility of X, or (2) this clever new hypothetical is not so clever—it rests on hidden assumptions that turn out to be false. Either way you are stuck testing your own assumptions and/or the hypothesis’ assumptions before making that ruling.
The trouble is, most people tend to just assume (1). I don’t know if there is a name for this heuristic, but it does lead to bias.
Your arguments rests on trying to be clever, which Mark rejected as a means of gathering knowledge.
Do you have empiric evidence that there are cases where people did well by updating after reading fictional stories?
Are there any studies that suggest that people who update through fictional stories do better?
Studies, no. I can’t imagine studies existing today that resolve this, which of course is a huge failure of imagination: that’s a really good thing to think about. For anything high enough level, I expect to run into problems with “do better”, such as “do better at predicting the behavior of AGI” being an accessible category. I would be very excited if there were nearby categories that we could get our hands on though; I expect this is similar to the problem of developing and testing a notion of “Rationality quotient” and proving it’s effectiveness.
I’m not sure where you’re referring to with Mark rejecting cleverness as a way of gathering knowledge, but I think we may be arguing about what the human equivalent of logical uncertainty looks like? What’s the difference in this case between “cleverness” and “thinking”? (Also could you point me to the place you were talking about?)
I guess I usually think of cleverness with the negative connotation being “thinking in too much detail with a blind spot”. So could you say which part of my response you think is bad thinking, or what you instead mean by cleverness?
I see. I do think you can update based on thinking; the human analogue of the logical uncertainty I was talking about. As an aspiring mathematician, this is what I think the practice of mathematics looks, for instance.
I understand the objection that this process may fail in real life, or lead to worse outcomes, since our models aren’t purely formal and our reasoning isn’t either purely deductive or optimally Bayesian. It looks like some others have made some great comments to this article also discussing that.
I’m just confused about which thinking you’re considering bad. I’m sure I’m not understanding, because it sounds to me like “the thinking which is thinking, and not direct empirical observation.” There’s got to be some level above direct empirical observation, or you’re just an observational rock. The internal process you have which at any level approximates Bayesian reasoning is a combination of your unconscious processing and your conscious thinking.
I’m used to people picking apart arguments. I’m used to heuristics that say, “hey you’ve gone too far with abstract thinking here, and here’s an empirical way to settle it, or here’s an argument for why your abstract thinking has gone too far and you should wait for empirical evidence or do X to seek some out” But I’m not used to “your mistake was abstract thinking at all; you can do nothing but empirically observe to gain a new state of understanding”, at least with regard to things like this. I feel like I’m caricaturing, but there’s a big blank when I try to figure out what else is being said.
There are two ways you can to do reasoning.
1) You build a theory about Bayesian updating and how it should work.
2) You run studies of how humans reasons and when they reason successfully. You identify when and how human reason correctly.
If I would argue that taking a specific drug helps you with illness X, the only argument you would likely accept is an empiric study. That’s independent with whether or not you can find a flaw in casual reason of why I think drug X should help with an illness. At least if you believe in evidence-based medicine.
The reason is that in the past theory based arguments often turned out to be wrong in the field of medicine.
We don’t live in a time where we don’t have anyone doing decision science. Whether or not people are simply blinded by fiction or whether it helps reasoning is an empirical question.
Ok, I think see the core of what you’re talking about, especially “Whether or not people are simply blinded by fiction or whether it helps reasoning is an empirical question.” This sounds like an outside view versus inside view distinction: I’ve been focused on “What should my inside view look like” and using outside view tools to modify that when possible (such as knowledge of a bias from decision science.) I think you and maybe Mark are trying to say “the inside view is useless or counter-productive here; only the outside view will be of any use” so that in the absence of outside view evidence, we should simply not attempt to reason further unless it’s a super-clear case, like Mark illustrates in his other comment.
My intuition is that this is incorrect, but it reminds me of the Hanson-Yudkowsky debates on outside vs. weak inside view, and I think I don’t have a strong enough grasp to clarify my intuition sufficiently right now. I’m going to try and pay serious attention to this issue in the future though, and would appreciate if you have any references that you think might clarify.
It’s not only outside vs. inside view. It’s knowing things is really hard. Humans are by nature overconfident. Life isn’t fair. The fact that empiric evidence is hard to get doesn’t make theoretical reasoning about the issue any more likely to be correct.
I rather trust a doctor with medical experience (has an inside view) to translate empirical studies in a way that applies directly to me than someone who reasons simply based on reading the study and who has no medical experience.
I do sin from time from time and act overconfident. But that doesn’t mean it’s right. Skepticism is a virtue. I like Foersters book “Truth is the invention of a liar” (unfortunately that book is in German, and I haven’t read other writing by him).
It doesn’t really gives answers but it makes the unknowing more graspable.
It’d be invalid as evidence, but it might still give a felt sense of your ideas that helps appreciate them. Discussions of AI, like discussions of aliens, have always been drawing on fiction at least for illustration. I for one would love to see that story.
What lesson am I supposed to learn from “That Alien Message”? It’s a work of fiction. You do not generalize from fictional evidence. Maybe I should write a story about how slow a takeoff would be given the massive inefficiencies of present technology, all the trivial and mundane ways an AI in the midst of a takeoff would get tripped up and caught, and all the different ways redundant detection mechanisms, honey pots, and fail safe contraptions would prevent existential risk scenarios? But such a work of fiction would be just as invalid as evidence.
Ok, I’m still confused as to many of my questions, but let me see if this bit sounds right: the only parameter via which something like “That Alien Message” could become more persuasive to you is by being less fictional. Fictional accounts of anything will NEVER cause you to update your beliefs. Does that sound right?
If that’s right, then I want to suggest why such things should sometimes be persuasive. A perfect Bayesian reasoner with finite computational ability operates not just with uncertainty about the outside world, but also with logical uncertainty as to consequences of their beliefs. So as humans, we operate with at least that difficulty when dealing with our own beliefs. In practice we deal with much much much worse.
I believe the correct form of the deduction your trying to make is “don’t add a fictional story to the reference class of a real analogue for purposes of figuring your beliefs”, and I agree. However, there are other ways a fictional story can be persuasive and should (in my view) cause you to update your beliefs:
It illustrates a new correct deduction which you weren’t aware of before, whose consequences you then begin working out.
It reminds you of a real experience you’ve had, which was not present in your mind before, whose existence then figures into your reference classes.
It changes your emotional attitude toward something, indirectly changing your beliefs by causing you to reflect on that thing differently in the future.
Some of these are subject to biases which would need correcting to move toward better reasoning, but I perceive you as claiming that these should have no impact, ever. Am I interpreting that correctly(I’m going to guess that I’m not somewhere), and if so why do you think that?
I think it’s a pretty big assumption to assume that fictional stories typically do those things correctly. Fictional stories are, after all, produced by people with agendas. If the proportion of fictional stories with plausible but incorrect deductions, reminders, or reflections is big enough, even your ability to figure out which ones are correct might not make it worthwhile to use fiction this way.
(Consider an extreme case where you can correctly assess 95% of the time whether a fictional deduction, reminder, or reflection is correct, but they are incorrect at a 99% rate. You’d have about a 4⁄5 chance of being wrong if you update based on fiction.)
Agreed; you’d have to figure all of that out separately. For what it’s worth, given the selection of fictional stories I’m usually exposed to and decide to read, I think they’re generally positive value (though probably not the best in terms of opportunity cost.)
If a story or thought experiment prompts you to think of some existing data you hadn’t paid attention to, and realize that data was not anticipated by your present beliefs, then that data acts as evidence for updating beliefs. The story or thought experiment was merely a reference used to call attention to that data.
“Changing your emotional attitude” as far as I can tell is actually cache-flushing. It does not change your underlying beliefs, it just trains your emotional response to align with those beliefs, eliminating inconsistencies in thought.
I’m not sure where “that alien message” is supposed to lie in either of those two categories. It makes no reference to actual experimental data which I may not have been paying attention to, nor do I detect any inconsistency it is unraveling. Rather, it makes a ton of assumptions and then runs with those assumptions, when in fact those assumptions were not valid in the first place. It’s a cathedral built on sand.
Basically agreed on paragraph 1, but I do want to suggest that then we not say “I will never update on fictional stories.” Taken naively, you then might avoid fictional stories because they’re useless (“I never update on them!”), when of course they might be super useful if they cause you to pull up relevant experiences quite often.
I’ll make an example of how “That Alien Message” could do for me what I illustrated in my 1st bullet point. I think, “Oh, it seems very unlikely that an AI could break out of a box, you just have this shutoff switch and watch it closely and …”. Then That Alien Message suggests the thought experiment of “instead of generalizing over all AI, instead imagine just a highly specific type of AI that may or may not reasonably come to exist: a bunch of really sped up, smart humans.” Then it sparks my thinking for what a bunch of really sped up, smart humans could accomplish with even narrow channels of communication. Then I think “actually, though I’ve seen no new instances of the AI reference class in reality, I now reason differently about how a possible AI could behave since that class (of possibilities) includes the the thing I just thought of. Until I get a lot more information about how that class behaves in reality, I’m going to be a lot more cautious.”
By picking out a specific possible example, it illustrates that my thinking around the possible AI reference class wasn’t expansive enough. This could help break through, for example, an accessibility heuristic: when I think of a random AI, I think of my very concrete vision of how such an AI would behave, instead of really trying to think about what could lie in that huge space.
Perhaps you are already appropriately cautious, and this story sparks/sparked no new thoughts in you, or you have a good reason to believe that communities of sped up humans or anything at least as powerful are excluded from the reference space, or the reference space you care about is narrower, but it seemed like you were making a stronger statement that such stories will never have any impact on you.
Creative / clever thinking is good. It’s where new ideas come from. Practicing creative thinking by reading interesting stories is not a waste of time. Updating based on creative / clever thoughts, on the other hand, is a mistake. The one almost-exception I can think of is “X is impossible!” where a clever plan for doing X, even if not actually implemented, suffices as weak evidence that “X is impossible!” is false. Or rather, it should propagate up your belief hierarchy and make you revisit why you thought X was impossible in the first place. Because the two remaining options are: (1) you were mistaken about the plausibility of X, or (2) this clever new hypothetical is not so clever—it rests on hidden assumptions that turn out to be false. Either way you are stuck testing your own assumptions and/or the hypothesis’ assumptions before making that ruling.
The trouble is, most people tend to just assume (1). I don’t know if there is a name for this heuristic, but it does lead to bias.
Your arguments rests on trying to be clever, which Mark rejected as a means of gathering knowledge.
Do you have empiric evidence that there are cases where people did well by updating after reading fictional stories? Are there any studies that suggest that people who update through fictional stories do better?
This seems promising!
Studies, no. I can’t imagine studies existing today that resolve this, which of course is a huge failure of imagination: that’s a really good thing to think about. For anything high enough level, I expect to run into problems with “do better”, such as “do better at predicting the behavior of AGI” being an accessible category. I would be very excited if there were nearby categories that we could get our hands on though; I expect this is similar to the problem of developing and testing a notion of “Rationality quotient” and proving it’s effectiveness.
I’m not sure where you’re referring to with Mark rejecting cleverness as a way of gathering knowledge, but I think we may be arguing about what the human equivalent of logical uncertainty looks like? What’s the difference in this case between “cleverness” and “thinking”? (Also could you point me to the place you were talking about?)
I guess I usually think of cleverness with the negative connotation being “thinking in too much detail with a blind spot”. So could you say which part of my response you think is bad thinking, or what you instead mean by cleverness?
It’s detached from empirical observation. It rests on the assumption that one can gather knowledge by reasoning itself (i.e. being clever).
I see. I do think you can update based on thinking; the human analogue of the logical uncertainty I was talking about. As an aspiring mathematician, this is what I think the practice of mathematics looks, for instance.
I understand the objection that this process may fail in real life, or lead to worse outcomes, since our models aren’t purely formal and our reasoning isn’t either purely deductive or optimally Bayesian. It looks like some others have made some great comments to this article also discussing that.
I’m just confused about which thinking you’re considering bad. I’m sure I’m not understanding, because it sounds to me like “the thinking which is thinking, and not direct empirical observation.” There’s got to be some level above direct empirical observation, or you’re just an observational rock. The internal process you have which at any level approximates Bayesian reasoning is a combination of your unconscious processing and your conscious thinking.
I’m used to people picking apart arguments. I’m used to heuristics that say, “hey you’ve gone too far with abstract thinking here, and here’s an empirical way to settle it, or here’s an argument for why your abstract thinking has gone too far and you should wait for empirical evidence or do X to seek some out” But I’m not used to “your mistake was abstract thinking at all; you can do nothing but empirically observe to gain a new state of understanding”, at least with regard to things like this. I feel like I’m caricaturing, but there’s a big blank when I try to figure out what else is being said.
There are two ways you can to do reasoning. 1) You build a theory about Bayesian updating and how it should work. 2) You run studies of how humans reasons and when they reason successfully. You identify when and how human reason correctly.
If I would argue that taking a specific drug helps you with illness X, the only argument you would likely accept is an empiric study. That’s independent with whether or not you can find a flaw in casual reason of why I think drug X should help with an illness. At least if you believe in evidence-based medicine. The reason is that in the past theory based arguments often turned out to be wrong in the field of medicine.
We don’t live in a time where we don’t have anyone doing decision science. Whether or not people are simply blinded by fiction or whether it helps reasoning is an empirical question.
Ok, I think see the core of what you’re talking about, especially “Whether or not people are simply blinded by fiction or whether it helps reasoning is an empirical question.” This sounds like an outside view versus inside view distinction: I’ve been focused on “What should my inside view look like” and using outside view tools to modify that when possible (such as knowledge of a bias from decision science.) I think you and maybe Mark are trying to say “the inside view is useless or counter-productive here; only the outside view will be of any use” so that in the absence of outside view evidence, we should simply not attempt to reason further unless it’s a super-clear case, like Mark illustrates in his other comment.
My intuition is that this is incorrect, but it reminds me of the Hanson-Yudkowsky debates on outside vs. weak inside view, and I think I don’t have a strong enough grasp to clarify my intuition sufficiently right now. I’m going to try and pay serious attention to this issue in the future though, and would appreciate if you have any references that you think might clarify.
It’s not only outside vs. inside view. It’s knowing things is really hard. Humans are by nature overconfident. Life isn’t fair. The fact that empiric evidence is hard to get doesn’t make theoretical reasoning about the issue any more likely to be correct.
I rather trust a doctor with medical experience (has an inside view) to translate empirical studies in a way that applies directly to me than someone who reasons simply based on reading the study and who has no medical experience.
I do sin from time from time and act overconfident. But that doesn’t mean it’s right. Skepticism is a virtue. I like Foersters book “Truth is the invention of a liar” (unfortunately that book is in German, and I haven’t read other writing by him). It doesn’t really gives answers but it makes the unknowing more graspable.
It’d be invalid as evidence, but it might still give a felt sense of your ideas that helps appreciate them. Discussions of AI, like discussions of aliens, have always been drawing on fiction at least for illustration. I for one would love to see that story.