Assume, for a moment, that scaring people into action is the correct response to imminent doom.
[…]
Note that this is all conditioned on scaring people into action being the the best action conditioned on imminent doom, which I am unsure about.
Noted. FWIW I’m seriously skeptical of whether it’s true. For two main reasons:
The kind of “imminent doom” we’re talking about is years away. People can’t stay actively scared and useful on timescales that long. I think they go numb and adjust instead.
Fear distorts thinking. Saying “But it’s dire!” as an argument for scaring people creates a memetic evolutionary incentive to make that claim. You can severely warp the epistemic commons if the norms around pulling the fire alarm let it happen too easily.
With that said: I think if we were pretty confident that AI doom were coming in the next year, then freaking people out about it might make a lot of sense.
But I think the bar for making that call should be very high. Otherwise bad memes eat us all.
we could possibly use other events to get quicker results.
Yep, sounds like that’d be the wholesome and good-faith thing to do at this point, best as I can tell.
you don’t give much evidence that this particular emotional mechanism is involved as opposed to others.
True. I was trying to suggest a gearsy model of how impressions of doom might get exaggerated and warp perceptions. I’ve seen stuff very much like the trauma model play out best as I can tell. So it seemed like a plausible model here. It’s my go-to default.
If you or someone else would care to spell out a different gearsy model, and if we can distinguish between them, I’d be quite happy to hear it and look at the truth.
In particular you gesture at:
I think that the desire to be part of a secret, specially enlightened in-group is likely a big factor as well.
Intuitively that seems plausible to me. At a glance I think it’d put a different set of incentives on memetic evolution, but that they’d still warp shared vision away from truth.
I’m running out of reply steam, so I’m not going to go much deeper than I just have here. But I want to note that I think your idea makes some sense, and I bet we could tell empirically to what extent it vs. the trauma thing is happening.
The main thing I keep seeing missed when coming up with experiments is, we’re talking about subjective structures. Which means that we’re inside the things we’re trying to analyze. Which means that the scissors we construct to distinguish between worlds have to account for that self-referential element.
I’m reminded of how a bunch of people jumped in on an experiment Scott Alexander spelled out in “The Apologist and the Revolutionary” where dropping ice cold water in one ear had been reported to sort of disable the internal rationalizer for a little while, causing patients to rescind some absurd beliefs for a few minutes — and then later deny that they had rescinded them once the rationalizer came back online. Eliezer in particular went to do the experiment and came back saying that he hadn’t changed his mind about anything notable. Folk had to point out that that’s precisely what the experimental results would have predicted he’d say.
People make this kind of logic error with subjective science all the time. You really do have to account for the fact that you are inside the structure you’re trying to explore, and think through the implications, in order for your experiments to matter.
So, yeah, if we can account for that, then I bet we can tell to what extent this secret in-group drive is playing a role vs. something like the trauma model I currently use as my default go-to.
Noted. FWIW I’m seriously skeptical of whether it’s true. For two main reasons:
The kind of “imminent doom” we’re talking about is years away. People can’t stay actively scared and useful on timescales that long. I think they go numb and adjust instead.
Fear distorts thinking. Saying “But it’s dire!” as an argument for scaring people creates a memetic evolutionary incentive to make that claim. You can severely warp the epistemic commons if the norms around pulling the fire alarm let it happen too easily.
With that said: I think if we were pretty confident that AI doom were coming in the next year, then freaking people out about it might make a lot of sense.
But I think the bar for making that call should be very high. Otherwise bad memes eat us all.
Yep, sounds like that’d be the wholesome and good-faith thing to do at this point, best as I can tell.
True. I was trying to suggest a gearsy model of how impressions of doom might get exaggerated and warp perceptions. I’ve seen stuff very much like the trauma model play out best as I can tell. So it seemed like a plausible model here. It’s my go-to default.
If you or someone else would care to spell out a different gearsy model, and if we can distinguish between them, I’d be quite happy to hear it and look at the truth.
In particular you gesture at:
Intuitively that seems plausible to me. At a glance I think it’d put a different set of incentives on memetic evolution, but that they’d still warp shared vision away from truth.
I’m running out of reply steam, so I’m not going to go much deeper than I just have here. But I want to note that I think your idea makes some sense, and I bet we could tell empirically to what extent it vs. the trauma thing is happening.
The main thing I keep seeing missed when coming up with experiments is, we’re talking about subjective structures. Which means that we’re inside the things we’re trying to analyze. Which means that the scissors we construct to distinguish between worlds have to account for that self-referential element.
I’m reminded of how a bunch of people jumped in on an experiment Scott Alexander spelled out in “The Apologist and the Revolutionary” where dropping ice cold water in one ear had been reported to sort of disable the internal rationalizer for a little while, causing patients to rescind some absurd beliefs for a few minutes — and then later deny that they had rescinded them once the rationalizer came back online. Eliezer in particular went to do the experiment and came back saying that he hadn’t changed his mind about anything notable. Folk had to point out that that’s precisely what the experimental results would have predicted he’d say.
People make this kind of logic error with subjective science all the time. You really do have to account for the fact that you are inside the structure you’re trying to explore, and think through the implications, in order for your experiments to matter.
So, yeah, if we can account for that, then I bet we can tell to what extent this secret in-group drive is playing a role vs. something like the trauma model I currently use as my default go-to.