But I don’t expect anything like that. I expect that folk will keep making predictions, and they’ll keep shifting, and they’ll keep being a bit hard to pin down, and everything will keep seeming more dire in a Shepard tone[1] kind of way. Some advances in AI will happen, and the narratives will shift, and it’ll always seem like those narratives are shifting for perfectly sensible reasons.
I 100% agree that this is what we would expect in worlds where AI keeps advancing but never becomes really dangerous, and am in favor of operationalizing this with a mutually agreed-upon time to reevaluate the possibility of emotional biases.
However, I strongly disagree with the suggestion of 2028. Assume, for a moment, that scaring people into action is the correct response to imminent doom. Then unless we are extremely confident that there is no imminent doom, we should still be scaring people. Scaring people is bad and we want to avoid it, but we want to avoid extinction a whole lot more. Given that, I think it makes the most sense to go with something like a 99th percentile date.
I get awfully suspicious if there’s literally no way to tell for at least a decade. That starts to sound an awful lot like astrology.
The community’s aggregate 99th percentile date for truly manifest doom does seem to be at least that far away, but we could possibly use other events to get quicker results.
Note that this is all conditioned on scaring people into action being the the best action conditioned on imminent doom, which I am unsure about.
Cool. I’m not sure what to do with that, but acknowledged.
Sorry, it’s a bad habit of mine to unproductively say whether I agree or disagree with something without giving reasoning. The reason I am unconvinced is that, although it does seem to fit at first glance, you don’t give much evidence that this particular emotional mechanism is involved as opposed to others. For example, I think that the desire to be part of a secret, specially enlightened in-group is likely a big factor as well.
Assume, for a moment, that scaring people into action is the correct response to imminent doom.
[…]
Note that this is all conditioned on scaring people into action being the the best action conditioned on imminent doom, which I am unsure about.
Noted. FWIW I’m seriously skeptical of whether it’s true. For two main reasons:
The kind of “imminent doom” we’re talking about is years away. People can’t stay actively scared and useful on timescales that long. I think they go numb and adjust instead.
Fear distorts thinking. Saying “But it’s dire!” as an argument for scaring people creates a memetic evolutionary incentive to make that claim. You can severely warp the epistemic commons if the norms around pulling the fire alarm let it happen too easily.
With that said: I think if we were pretty confident that AI doom were coming in the next year, then freaking people out about it might make a lot of sense.
But I think the bar for making that call should be very high. Otherwise bad memes eat us all.
we could possibly use other events to get quicker results.
Yep, sounds like that’d be the wholesome and good-faith thing to do at this point, best as I can tell.
you don’t give much evidence that this particular emotional mechanism is involved as opposed to others.
True. I was trying to suggest a gearsy model of how impressions of doom might get exaggerated and warp perceptions. I’ve seen stuff very much like the trauma model play out best as I can tell. So it seemed like a plausible model here. It’s my go-to default.
If you or someone else would care to spell out a different gearsy model, and if we can distinguish between them, I’d be quite happy to hear it and look at the truth.
In particular you gesture at:
I think that the desire to be part of a secret, specially enlightened in-group is likely a big factor as well.
Intuitively that seems plausible to me. At a glance I think it’d put a different set of incentives on memetic evolution, but that they’d still warp shared vision away from truth.
I’m running out of reply steam, so I’m not going to go much deeper than I just have here. But I want to note that I think your idea makes some sense, and I bet we could tell empirically to what extent it vs. the trauma thing is happening.
The main thing I keep seeing missed when coming up with experiments is, we’re talking about subjective structures. Which means that we’re inside the things we’re trying to analyze. Which means that the scissors we construct to distinguish between worlds have to account for that self-referential element.
I’m reminded of how a bunch of people jumped in on an experiment Scott Alexander spelled out in “The Apologist and the Revolutionary” where dropping ice cold water in one ear had been reported to sort of disable the internal rationalizer for a little while, causing patients to rescind some absurd beliefs for a few minutes — and then later deny that they had rescinded them once the rationalizer came back online. Eliezer in particular went to do the experiment and came back saying that he hadn’t changed his mind about anything notable. Folk had to point out that that’s precisely what the experimental results would have predicted he’d say.
People make this kind of logic error with subjective science all the time. You really do have to account for the fact that you are inside the structure you’re trying to explore, and think through the implications, in order for your experiments to matter.
So, yeah, if we can account for that, then I bet we can tell to what extent this secret in-group drive is playing a role vs. something like the trauma model I currently use as my default go-to.
Then unless we are *extremely confident *that there is no imminent doom, we should still be scaring people. Scaring people is bad and we want to avoid it, but we want to avoid extinction a whole lot more. Given that, I think it makes the most sense to go with something like a 99th percentile date.
This sort of reasoning leads to Pascal’s Mugging and lots of weirdness like concern for the welfare of insects or electrons.
We should not be acting on the basis of extremely unlikely events because the events lead to a very large change in utility.
This just changes the quantile you should use. I already tried to adjust for this in the ballpark number I gave — I in fact think existential doom is much more than 99 times worse than scaring people. Even if you disagree and say it’s not worth trying to avoid an outcome because it has only a 1% probability, you’d probably agree it’s worth it at 50%, and it seems to me the median timeline is still way past 2028.
I 100% agree that this is what we would expect in worlds where AI keeps advancing but never becomes really dangerous, and am in favor of operationalizing this with a mutually agreed-upon time to reevaluate the possibility of emotional biases.
However, I strongly disagree with the suggestion of 2028. Assume, for a moment, that scaring people into action is the correct response to imminent doom. Then unless we are extremely confident that there is no imminent doom, we should still be scaring people. Scaring people is bad and we want to avoid it, but we want to avoid extinction a whole lot more. Given that, I think it makes the most sense to go with something like a 99th percentile date.
The community’s aggregate 99th percentile date for truly manifest doom does seem to be at least that far away, but we could possibly use other events to get quicker results.
Note that this is all conditioned on scaring people into action being the the best action conditioned on imminent doom, which I am unsure about.
Sorry, it’s a bad habit of mine to unproductively say whether I agree or disagree with something without giving reasoning. The reason I am unconvinced is that, although it does seem to fit at first glance, you don’t give much evidence that this particular emotional mechanism is involved as opposed to others. For example, I think that the desire to be part of a secret, specially enlightened in-group is likely a big factor as well.
I think this is a great metaphor, by the way.
Noted. FWIW I’m seriously skeptical of whether it’s true. For two main reasons:
The kind of “imminent doom” we’re talking about is years away. People can’t stay actively scared and useful on timescales that long. I think they go numb and adjust instead.
Fear distorts thinking. Saying “But it’s dire!” as an argument for scaring people creates a memetic evolutionary incentive to make that claim. You can severely warp the epistemic commons if the norms around pulling the fire alarm let it happen too easily.
With that said: I think if we were pretty confident that AI doom were coming in the next year, then freaking people out about it might make a lot of sense.
But I think the bar for making that call should be very high. Otherwise bad memes eat us all.
Yep, sounds like that’d be the wholesome and good-faith thing to do at this point, best as I can tell.
True. I was trying to suggest a gearsy model of how impressions of doom might get exaggerated and warp perceptions. I’ve seen stuff very much like the trauma model play out best as I can tell. So it seemed like a plausible model here. It’s my go-to default.
If you or someone else would care to spell out a different gearsy model, and if we can distinguish between them, I’d be quite happy to hear it and look at the truth.
In particular you gesture at:
Intuitively that seems plausible to me. At a glance I think it’d put a different set of incentives on memetic evolution, but that they’d still warp shared vision away from truth.
I’m running out of reply steam, so I’m not going to go much deeper than I just have here. But I want to note that I think your idea makes some sense, and I bet we could tell empirically to what extent it vs. the trauma thing is happening.
The main thing I keep seeing missed when coming up with experiments is, we’re talking about subjective structures. Which means that we’re inside the things we’re trying to analyze. Which means that the scissors we construct to distinguish between worlds have to account for that self-referential element.
I’m reminded of how a bunch of people jumped in on an experiment Scott Alexander spelled out in “The Apologist and the Revolutionary” where dropping ice cold water in one ear had been reported to sort of disable the internal rationalizer for a little while, causing patients to rescind some absurd beliefs for a few minutes — and then later deny that they had rescinded them once the rationalizer came back online. Eliezer in particular went to do the experiment and came back saying that he hadn’t changed his mind about anything notable. Folk had to point out that that’s precisely what the experimental results would have predicted he’d say.
People make this kind of logic error with subjective science all the time. You really do have to account for the fact that you are inside the structure you’re trying to explore, and think through the implications, in order for your experiments to matter.
So, yeah, if we can account for that, then I bet we can tell to what extent this secret in-group drive is playing a role vs. something like the trauma model I currently use as my default go-to.
This sort of reasoning leads to Pascal’s Mugging and lots of weirdness like concern for the welfare of insects or electrons.
We should not be acting on the basis of extremely unlikely events because the events lead to a very large change in utility.
This just changes the quantile you should use. I already tried to adjust for this in the ballpark number I gave — I in fact think existential doom is much more than 99 times worse than scaring people. Even if you disagree and say it’s not worth trying to avoid an outcome because it has only a 1% probability, you’d probably agree it’s worth it at 50%, and it seems to me the median timeline is still way past 2028.