So probably [exaggerating near-team non-existential AI risks] is a brilliant rhetorical strategy with no downsides. But it still gives me a visceral “ick” reaction to associate with something that might not be accurate.
Specifically, the “If we get a reputation as the people who fall for every panic about AI [...] will we eventually cry wolf one too many times and lose our credibility before crunch time?” argument is about being honest so as to be trusted by others. But another reason to be honest is so that other people can have the benefits of accurate information. If you simply report the evidence and arguments that actually convinced you, then your audience can combine the information you’re giving them with everything else they know, and make an informed decision for themselves.
This generalizes far beyond the case of AI. Take the “you can save a live for $3000” claim. How sure are you that that’s actually true? If it’s not true, that would be a huge problem not just because it’s not representative of the weird things EA insiders are thinking about, but because it would be causing people to spend a lot of money on the basis of false information.
I’m wondering if the last paragraph was a mistake on my part—whether I should have picked a different example. The parent seems likely to have played a causal role in catalyzing new discussion on “A Drowning Child Is Hard to Find”, but I’m much less interested in litigating the matter of cost-effectiveness numbers (which I know very little about) than I am in the principle that we want to have (or build the capacity to have, if we don’t currently have that capacity) systematically truth-tracking intellectual discussions, rather than accepting allegedly-small distortions for instrumental marketing reasons of the form, “This argument isn’t quite right, but it’s close enough, and the correct version would scare away powerful and influential people from our very important cause.” (As it is written of the fifth virtue, “The part of yourself that distorts what you say to others also distorts your own thoughts.”)
(cross posted from the Slate Star comment section)
Listen to that “ick” reaction, Scott! That’s evolution’s way of telling you about all the downsides you’re not currently seeing!
Specifically, the “If we get a reputation as the people who fall for every panic about AI [...] will we eventually cry wolf one too many times and lose our credibility before crunch time?” argument is about being honest so as to be trusted by others. But another reason to be honest is so that other people can have the benefits of accurate information. If you simply report the evidence and arguments that actually convinced you, then your audience can combine the information you’re giving them with everything else they know, and make an informed decision for themselves.
This generalizes far beyond the case of AI. Take the “you can save a live for $3000” claim. How sure are you that that’s actually true? If it’s not true, that would be a huge problem not just because it’s not representative of the weird things EA insiders are thinking about, but because it would be causing people to spend a lot of money on the basis of false information.
I’m wondering if the last paragraph was a mistake on my part—whether I should have picked a different example. The parent seems likely to have played a causal role in catalyzing new discussion on “A Drowning Child Is Hard to Find”, but I’m much less interested in litigating the matter of cost-effectiveness numbers (which I know very little about) than I am in the principle that we want to have (or build the capacity to have, if we don’t currently have that capacity) systematically truth-tracking intellectual discussions, rather than accepting allegedly-small distortions for instrumental marketing reasons of the form, “This argument isn’t quite right, but it’s close enough, and the correct version would scare away powerful and influential people from our very important cause.” (As it is written of the fifth virtue, “The part of yourself that distorts what you say to others also distorts your own thoughts.”)