Bostrom’s original definition of “infohazard” covers scenarios of the form “someone would like to suppress this information” in addition to “if you learn this information it will harm you”.
Scenarios where “someone would like to suppress this information” are much more common than “self-infohazard” scenarios.
When someone wants to suppress information (such as a cult leader who’s secretly abusive), it is in their interest to make others believe that it’s a “self-infohazard” (including with arguments like “They’re not ready to learn how I’ve been abusing them; it would shatter their worldview”), or possibly a “socio-infohazard” (where it’s in people’s collective interests to not know the leader’s crimes—”It would fracture the community”, “Having everyone listen to me brings order and unity and you don’t want to ruin that”, etc.).
I would add that it’s probably best if our vocabulary choices don’t make it easy for bad actors to make superficially-plausible claims that suppressing the information they want suppressed is good and virtuous. I would say that the word “infohazard” itself sounds to me (and, I suspect, to the naive layperson) like the “directly hazardous to me if I learn it” sense.
Therefore, it’s probably a bad thing if serious people believe that “infohazard” technically means Bostrom’s maximally-broad original definition, because that makes it harder for serious people to protest when e.g. someone says “The fact that some people in our organization screwed up X is an infohazard, so we shouldn’t publicly mention it”. Accepting that definition is like creating a motte-and-bailey term. (Incidentally: Learning that someone has done something bad is usually at least slightly unpleasant—more so if it’s a partner or leader—and therefore it’s ~always possible to make at least a slight case that “knowledge of my bad behavior” is a self-infohazard.)
I would suggest the following rule as an antidote: Anytime someone says the unqualified term “infohazard”, it means “information that I want suppressed”—i.e. we should be suspicious of their motives. If they have good reasons to suppress information, they should have to state them upfront. More specific terms might be “social infohazard”, “existential infohazard”, “cognitive infohazard”, etc. I’ll also note Eliezer’s suggestion of “secret”:
We already have a word for information that agent A would rather have B not know, because B’s knowledge of it benefits B but harms A; that word is ‘secret’.
By the way, I’m kind of weirded out by the idea that we need short terms that mean “information that should be suppressed” (i.e. in terms of Huffman coding, “we should use a short word for it” = “it is very common”), and furthermore that it’s rationalists who are trying to come up with such words. I think it’s ultimately for innocent reasons—that Bostrom picked “information hazards” for the title of his paper, and people made the obvious catchy portmanteau; still, I don’t want to push the language in that direction.
If you really want to create widespread awareness of the broad definition, the thing to do would be to use the term in all the ways you currently wouldn’t.
E.g. “The murderer realized his phone’s GPS history posed a significant infohazard, as it could be used to connect him to the crime.”
By the way, I’m kind of weirded out by the idea that we need short terms that mean “information that should be suppressed” (i.e. in terms of Huffman coding, “we should use a short word for it” = “it is very common”), and furthermore that it’s rationalists who are trying to come up with such words. I think it’s ultimately because Bostrom picked “information hazards” for the title of his paper, and people made the obvious catchy portmanteau; but I don’t want to push the language in that direction.
I largely agree with you. Having a richer vocabulary would be helpful for thinking about problems of this theme with more nuance, if the participants used that rich vocabulary accurately and with goodwill. I also think that defining new words to label these nuanced distinctions can be helpful to motivate more sophisticated thinking. But when we try to reason about concrete problems using this terminology and conceptual scheme, we ought to taboo our words and show why a given piece of information is hazardous to some person or group.
I’m skeptical that the use of these short phrases implies that rationalists have overly normalized speech suppression (if that’s what you mean by your Huffman coding argument). Copywriters pre-emptively shorten novel words and phrases to make them catchy, or to give them the appearance of colloquialism and popularity. Since the rationalist community is primarily blog-based, I see these shortenings as part of a general trend toward “readability,” not as a symptom of rationalists being over-steeped in “infohazard” concepts.
I’m skeptical that the use of these short phrases implies that rationalists have overly normalized speech suppression (if that’s what you mean by your Huffman coding argument).
Edited to hopefully clarify. I do believe that the reasons for it are innocent, but it still feels uncomfortable, and, to the extent that it’s under our control, I would like to reduce it.
That sounded like you said it’s bad if serious people accept a definition of “infohazard” that allows it to be legitimately used by corrupt people trying to keep their corruption secret for purely selfish reasons, but then in the next paragraph you proposed a definition that allows it to be used that way?
The really bad thing is if corrupt people get a motte-and-bailey, where they get to use the term “infohazard” unchallenged in describing why their corruption should be kept secret, and casual observers assume it means “a thing everyone agrees should be kept secret”. I’m recommending spreading the meme that the bare word “infohazard” carries a strong connotation of “a thing I want kept secret for nefarious reasons and I want to trick you all into going along with it”. I think, if the meme is widely spread, it should fix the issue.
Your earlier comment sounded to me like you were framing the problem as “the word has these connotations for typical people, and the problem is that serious people have a different definition and aren’t willing to call out bad actors who are relying on the connotations to carry their arguments.”
That framing naturally suggests a solution of “serious people need to either use different definitions or have different standard for when to call people out.”
Now it seems like you’re framing the problem as “the word’s going to be used by corrupt people, and the problem is that typical people assign connotations to the word that make the corrupt person’s argument more persuasive.”
I dislike the second framing for a couple reasons:
The first framing suggests we need to change the explicit understanding of serious people; the second that we need to change the implicit understanding of typical people. Of those two, changing the first thing seems massively more feasible to me.
You are evoking scenarios where the bad guy says a word that is understood to mean “spreading this is bad for me.” I think this is an unrealistic scenario, and you should instead be evoking a scenario where the bad guy switches to a word that is still widely understood to mean “spreading this is bad for the collective”, but where serious people no longer think that it technically could mean something else. (The bad guy will switch to whatever word is currently most favorable for them, not stick to a single word while we change the connotations out from under them.)
I don’t think the problem is motte-and-bailey per se; to me, that term implies a trick that relies upon a single audience member subconsciously applying different definitions to different parts of the argument. But it sounds to me like you’re describing a problem where one part of the audience is persuaded, because they apply the narrower definition and aren’t knowledgeable enough to object, while another part of the audience is NOT persuaded, but fails to object, because they apply the broader definition. (No single audience member is applying multiple definitions.)
If the second group actually did object, then hypothetically the speaker could turn this into a motte-and-bailey by defending their arguments under the broader definition. But I don’t think that’s much of a practical risk, in this case. To actually execute that motte-and-bailey, you’d need to at some point say something like “spreading this info is bad for me [and therefore it counts as an infohazard]”, and I think that sound bite would lose you so many rhetorical points among people-you-could-potentially-trick-with-it that it wouldn’t typically be worth it.
I do hope that “spreading the meme that “infohazard” probably means “info I selfishly want to suppress”″ will cause serious people to more readily notice and raise objections when someone is using it to gloss over corruption. I guess I didn’t specify that, but I believe that would be the primary means by which it would help in the short term. So I think we don’t actually disagree here? (In the longer term, I do suspect that either the meme would spread to ordinary people, or the term “infohazard” would fall into disuse.)
So the motte-and-bailey doctrine is when you make a bold, controversial statement. Then when somebody challenges you, you retreat to an obvious, uncontroversial statement, and say that was what you meant all along, so you’re clearly right and they’re silly for challenging you. Then when the argument is over you go back to making the bold, controversial statement.
It does have an element of conditionally retreating depending on whether you’re challenged. Also, I think the original context is people making bold claims on the internet, which probably means the audience is many people, and there’s often a comments section where objections might be made or not made. The case of persuading a single person to accept a claim via one definition, then telling them the claim implies something else via a different definition—I would use different words for that, perhaps “equivocating” (Wiki agrees, although the article body references motte-and-bailey as one use case) or “Trojan horse argument”.
The ideal user of a motte-and-bailey hopes that, most of the time, they won’t be challenged; when they are challenged, it does become less convincing. The motte needs to be something they can at least “fight to a standstill” defending; if it is, then this discourages challengers. I expect we agree on this.
I would agree that “It’s an infohazard because revealing it hurts me” is generally not a good motte. However, there’s still a selection of other justifications to retreat to, some of which might be hard to disprove objectively, which suffices for the “argue to a standstill” purpose. If necessary, for someone who cares primarily about their social capital and doesn’t absolutely need to win the argument, it might even be a motte to say “I wasn’t claiming that the downsides of revealing the truth definitely outweigh the values of truth and justice; I just meant that there are significant downsides”.
Let’s take the example of “It’s an infohazard for churchgoers to learn that Priest Bob had molested some children 20 years ago.” The bailey would be “I’m claiming that many of our churchgoers would be heartbroken, would lose faith in God, etc., and since no one is challenging me on this you should update towards thinking the churchgoers are so fragile this is a worthwhile tradeoff.” One motte would be “Well, it would clearly cause emotional distress to lots of people, and we should think carefully before releasing it. (Definitely we should hesitate long enough for objectors like you to leave the room, so I can repeat my original argument to a more naive audience.)”
(Incidentally, I would admit that people didn’t need the word “infohazard” to make arguments like the above. But having the word would probably make their job easier, give them a layer of plausible deniability.)
I disagree that the word “infohazard” makes it easier to use arguments like the ones in your final example. If we had a word that was universally acknowledged to mean “information whose dissemination causes communal harm”, they could make precisely the same argument using that word, and I don’t see how the argument would be weakened.
And...I guess I’m confused about your strategy of spreading your proposed meme to serious people. If the goal is to provide the serious people a basis upon which to object, this strikes me as a terrible basis; “your word choice implies you are probably corrupt” is an unpersuasive counter-argument. If the goal is to make the serious people notice at all that the argument is objectionable, then that seems like a fragile and convoluted way of doing that—making people notice that an argument might be flawed, based on easily-changeable word choice, rather than an actual logical flaw. Maybe I’m still not understanding the proposed mechanism of action?
Related: “Infohazard” is a predominantly conflict-theoretic concept
Paraphrasing of some of the things Jessica says:
Bostrom’s original definition of “infohazard” covers scenarios of the form “someone would like to suppress this information” in addition to “if you learn this information it will harm you”.
Scenarios where “someone would like to suppress this information” are much more common than “self-infohazard” scenarios.
When someone wants to suppress information (such as a cult leader who’s secretly abusive), it is in their interest to make others believe that it’s a “self-infohazard” (including with arguments like “They’re not ready to learn how I’ve been abusing them; it would shatter their worldview”), or possibly a “socio-infohazard” (where it’s in people’s collective interests to not know the leader’s crimes—”It would fracture the community”, “Having everyone listen to me brings order and unity and you don’t want to ruin that”, etc.).
I would add that it’s probably best if our vocabulary choices don’t make it easy for bad actors to make superficially-plausible claims that suppressing the information they want suppressed is good and virtuous. I would say that the word “infohazard” itself sounds to me (and, I suspect, to the naive layperson) like the “directly hazardous to me if I learn it” sense.
Therefore, it’s probably a bad thing if serious people believe that “infohazard” technically means Bostrom’s maximally-broad original definition, because that makes it harder for serious people to protest when e.g. someone says “The fact that some people in our organization screwed up X is an infohazard, so we shouldn’t publicly mention it”. Accepting that definition is like creating a motte-and-bailey term. (Incidentally: Learning that someone has done something bad is usually at least slightly unpleasant—more so if it’s a partner or leader—and therefore it’s ~always possible to make at least a slight case that “knowledge of my bad behavior” is a self-infohazard.)
I would suggest the following rule as an antidote: Anytime someone says the unqualified term “infohazard”, it means “information that I want suppressed”—i.e. we should be suspicious of their motives. If they have good reasons to suppress information, they should have to state them upfront. More specific terms might be “social infohazard”, “existential infohazard”, “cognitive infohazard”, etc. I’ll also note Eliezer’s suggestion of “secret”:
By the way, I’m kind of weirded out by the idea that we need short terms that mean “information that should be suppressed” (i.e. in terms of Huffman coding, “we should use a short word for it” = “it is very common”), and furthermore that it’s rationalists who are trying to come up with such words. I think it’s ultimately for innocent reasons—that Bostrom picked “information hazards” for the title of his paper, and people made the obvious catchy portmanteau; still, I don’t want to push the language in that direction.
If you really want to create widespread awareness of the broad definition, the thing to do would be to use the term in all the ways you currently wouldn’t.
E.g. “The murderer realized his phone’s GPS history posed a significant infohazard, as it could be used to connect him to the crime.”
I largely agree with you. Having a richer vocabulary would be helpful for thinking about problems of this theme with more nuance, if the participants used that rich vocabulary accurately and with goodwill. I also think that defining new words to label these nuanced distinctions can be helpful to motivate more sophisticated thinking. But when we try to reason about concrete problems using this terminology and conceptual scheme, we ought to taboo our words and show why a given piece of information is hazardous to some person or group.
I’m skeptical that the use of these short phrases implies that rationalists have overly normalized speech suppression (if that’s what you mean by your Huffman coding argument). Copywriters pre-emptively shorten novel words and phrases to make them catchy, or to give them the appearance of colloquialism and popularity. Since the rationalist community is primarily blog-based, I see these shortenings as part of a general trend toward “readability,” not as a symptom of rationalists being over-steeped in “infohazard” concepts.
Edited to hopefully clarify. I do believe that the reasons for it are innocent, but it still feels uncomfortable, and, to the extent that it’s under our control, I would like to reduce it.
That sounded like you said it’s bad if serious people accept a definition of “infohazard” that allows it to be legitimately used by corrupt people trying to keep their corruption secret for purely selfish reasons, but then in the next paragraph you proposed a definition that allows it to be used that way?
The really bad thing is if corrupt people get a motte-and-bailey, where they get to use the term “infohazard” unchallenged in describing why their corruption should be kept secret, and casual observers assume it means “a thing everyone agrees should be kept secret”. I’m recommending spreading the meme that the bare word “infohazard” carries a strong connotation of “a thing I want kept secret for nefarious reasons and I want to trick you all into going along with it”. I think, if the meme is widely spread, it should fix the issue.
Your earlier comment sounded to me like you were framing the problem as “the word has these connotations for typical people, and the problem is that serious people have a different definition and aren’t willing to call out bad actors who are relying on the connotations to carry their arguments.”
That framing naturally suggests a solution of “serious people need to either use different definitions or have different standard for when to call people out.”
Now it seems like you’re framing the problem as “the word’s going to be used by corrupt people, and the problem is that typical people assign connotations to the word that make the corrupt person’s argument more persuasive.”
I dislike the second framing for a couple reasons:
The first framing suggests we need to change the explicit understanding of serious people; the second that we need to change the implicit understanding of typical people. Of those two, changing the first thing seems massively more feasible to me.
You are evoking scenarios where the bad guy says a word that is understood to mean “spreading this is bad for me.” I think this is an unrealistic scenario, and you should instead be evoking a scenario where the bad guy switches to a word that is still widely understood to mean “spreading this is bad for the collective”, but where serious people no longer think that it technically could mean something else. (The bad guy will switch to whatever word is currently most favorable for them, not stick to a single word while we change the connotations out from under them.)
I don’t think the problem is motte-and-bailey per se; to me, that term implies a trick that relies upon a single audience member subconsciously applying different definitions to different parts of the argument. But it sounds to me like you’re describing a problem where one part of the audience is persuaded, because they apply the narrower definition and aren’t knowledgeable enough to object, while another part of the audience is NOT persuaded, but fails to object, because they apply the broader definition. (No single audience member is applying multiple definitions.)
If the second group actually did object, then hypothetically the speaker could turn this into a motte-and-bailey by defending their arguments under the broader definition. But I don’t think that’s much of a practical risk, in this case. To actually execute that motte-and-bailey, you’d need to at some point say something like “spreading this info is bad for me [and therefore it counts as an infohazard]”, and I think that sound bite would lose you so many rhetorical points among people-you-could-potentially-trick-with-it that it wouldn’t typically be worth it.
I do hope that “spreading the meme that “infohazard” probably means “info I selfishly want to suppress”″ will cause serious people to more readily notice and raise objections when someone is using it to gloss over corruption. I guess I didn’t specify that, but I believe that would be the primary means by which it would help in the short term. So I think we don’t actually disagree here? (In the longer term, I do suspect that either the meme would spread to ordinary people, or the term “infohazard” would fall into disuse.)
Regarding motte-and-bailey—I disagree about the definition. Quoting one of Scott’s essays on the subject:
It does have an element of conditionally retreating depending on whether you’re challenged. Also, I think the original context is people making bold claims on the internet, which probably means the audience is many people, and there’s often a comments section where objections might be made or not made. The case of persuading a single person to accept a claim via one definition, then telling them the claim implies something else via a different definition—I would use different words for that, perhaps “equivocating” (Wiki agrees, although the article body references motte-and-bailey as one use case) or “Trojan horse argument”.
The ideal user of a motte-and-bailey hopes that, most of the time, they won’t be challenged; when they are challenged, it does become less convincing. The motte needs to be something they can at least “fight to a standstill” defending; if it is, then this discourages challengers. I expect we agree on this.
I would agree that “It’s an infohazard because revealing it hurts me” is generally not a good motte. However, there’s still a selection of other justifications to retreat to, some of which might be hard to disprove objectively, which suffices for the “argue to a standstill” purpose. If necessary, for someone who cares primarily about their social capital and doesn’t absolutely need to win the argument, it might even be a motte to say “I wasn’t claiming that the downsides of revealing the truth definitely outweigh the values of truth and justice; I just meant that there are significant downsides”.
Let’s take the example of “It’s an infohazard for churchgoers to learn that Priest Bob had molested some children 20 years ago.” The bailey would be “I’m claiming that many of our churchgoers would be heartbroken, would lose faith in God, etc., and since no one is challenging me on this you should update towards thinking the churchgoers are so fragile this is a worthwhile tradeoff.” One motte would be “Well, it would clearly cause emotional distress to lots of people, and we should think carefully before releasing it. (Definitely we should hesitate long enough for objectors like you to leave the room, so I can repeat my original argument to a more naive audience.)”
(Incidentally, I would admit that people didn’t need the word “infohazard” to make arguments like the above. But having the word would probably make their job easier, give them a layer of plausible deniability.)
I disagree that the word “infohazard” makes it easier to use arguments like the ones in your final example. If we had a word that was universally acknowledged to mean “information whose dissemination causes communal harm”, they could make precisely the same argument using that word, and I don’t see how the argument would be weakened.
And...I guess I’m confused about your strategy of spreading your proposed meme to serious people. If the goal is to provide the serious people a basis upon which to object, this strikes me as a terrible basis; “your word choice implies you are probably corrupt” is an unpersuasive counter-argument. If the goal is to make the serious people notice at all that the argument is objectionable, then that seems like a fragile and convoluted way of doing that—making people notice that an argument might be flawed, based on easily-changeable word choice, rather than an actual logical flaw. Maybe I’m still not understanding the proposed mechanism of action?