I agree that @Valentine’s specific model here is unlikely to fit the data well here, but to be charitable to Valentine/steelman the post, the better nearby argument is that hypotheses where astronomical value, in either negative or positive directions are memetically fit and very importantly believed by lots of people and lots of people take serious actions that are later revealed to be mostly mistakes because the hypothesis of doom/salvation by something had gotten too high a probability inside their brains relative to an omniscient observer.
Another way to say it is that the doom/salvation hypotheses aren’t purely believed because they have evidence for the hypothesis directly.
This is a necessary consequence of humans needing to make expected utility decisions all the time, combined with both their values/utility functions mostly not falling in value fast enough with increasing resources to avoid the conclusion that unboundedly valuable states exist for a human, and the humans being bounded reasoners/performing bounded rationality that means they cannot distinguish probabilities between say 1 in a million and 0 finely.
However, another partial explanation comes from @Nate Showell, where pessimism is used as a coping mechanism to not deal with personal scale problems, and in particular believing that the world is doomed from something is a good excuse to not deal with stuff like doing the dishes or cleaning your bedroom, and it’s psychologically appealing to have a hypothesis that means you don’t have to do any mundane work to solve the problem:
And this is obviously problematic for anyone working on getting the public to believe an existential risk is real, if there is in fact real evidence something poses an x-risk.
Here, an underrated cure by Valentine is to focus on the object level, and to focus as much on empirical research as possible, because this way you have to engage with mundane work.
Another useful solution is to have a social life that is separate from the scientific community working on the claimed x-risk.
“it’s psychologically appealing to have a hypothesis that means you don’t have to do any mundane work”
I don’t doubt that something like inverse bike-shedding can be a driving force for some individuals to focus on the field of AI safety. I highly doubt it is explanatory for the field and the associated risk predictions to exist in the first place, or that its validity should be questioned on such grounds, but this seems to happen in the article if I’m not entirely misreading it. From my point of view, there is already an overemphasis on psychological factors in the broader debate and it would be desirable to get back to the object level, be it with theoretical or empirical research, which both have their value. This latter aspect seems to lead to a partial agreement here, even though there’s more than one path to arrive at it.
I highly doubt it is explanatory for the field and the associated risk predictions to exist in the first place, or that its validity should be questioned on such grounds, but this seems to happen in the article if I’m not entirely misreading it.
Not entirely. It’s a bit of a misreading. In this case I think the bit matters though.
(And it’s an understandable bit! It’s a subtle point I find I have a hard time communicating clearly.)
I’m trying to say two things:
There sure do seem to be some bad psychological influences going on.
It’s harder to tell what’s real when you have sufficiently bad psychological influences going on.
I think some people, such as you, are reacting really strongly to that second point. Like I’m taking a stand for AI risk being a non-issue and saying it’s all psychological projection.
I’m saying that nonzero, but close to zero. It’s a more plausible hypothesis to me than I think it is to this community. But that’s not because I’m going through the arguments that AI risk is real and finding refutations. It’s because I’ve seen some shockingly basic things turn out to be psychological projection, and I don’t think Less Wrong collectively understands that projection really can be that deep. I just don’t see it accounted for in the arguments for doom.
But that’s not the central point I’m trying to make. My point is more that I think the probability of doom is significantly elevated as a result of how memetic evolution works — and, stupidly, I think that makes doom more likely as a result of the “Don’t hit the tree” phenomenon.
And maybe even more centrally, you cannot know how elevated the probability is until you seriously check for memetic probability boosters. And even then, how you check needs to account for those memetic influences.
I’m not trying to say that AI safety shouldn’t exist as a field though.
From my point of view, there is already an overemphasis on psychological factors in the broader debate and it would be desirable to get back to the object level
Wow, you and I sure must be seeing different parts of the debate! I approximately only hear people talking about the object level. That’s part of my concern.
I mean, I see some folk doing hot takes on Twitter about psychological angles. But most of those strike me as more like pot shots and less like attempts to engage in a dialogue.
This was a great steelmanning, and is exactly the kind of thing I hope people will do in contact with what I offer. Even though I don’t agree with every detail, I feel received and like the thing I care about is being well enough held. Thank you.
I agree that @Valentine’s specific model here is unlikely to fit the data well here, but to be charitable to Valentine/steelman the post, the better nearby argument is that hypotheses where astronomical value, in either negative or positive directions are memetically fit and very importantly believed by lots of people and lots of people take serious actions that are later revealed to be mostly mistakes because the hypothesis of doom/salvation by something had gotten too high a probability inside their brains relative to an omniscient observer.
Another way to say it is that the doom/salvation hypotheses aren’t purely believed because they have evidence for the hypothesis directly.
This is a necessary consequence of humans needing to make expected utility decisions all the time, combined with both their values/utility functions mostly not falling in value fast enough with increasing resources to avoid the conclusion that unboundedly valuable states exist for a human, and the humans being bounded reasoners/performing bounded rationality that means they cannot distinguish probabilities between say 1 in a million and 0 finely.
However, another partial explanation comes from @Nate Showell, where pessimism is used as a coping mechanism to not deal with personal scale problems, and in particular believing that the world is doomed from something is a good excuse to not deal with stuff like doing the dishes or cleaning your bedroom, and it’s psychologically appealing to have a hypothesis that means you don’t have to do any mundane work to solve the problem:
https://www.lesswrong.com/posts/D4eZF6FAZhrW4KaGG/consider-chilling-out-in-2028#5748siHvi8YZLFZih
And this is obviously problematic for anyone working on getting the public to believe an existential risk is real, if there is in fact real evidence something poses an x-risk.
Here, an underrated cure by Valentine is to focus on the object level, and to focus as much on empirical research as possible, because this way you have to engage with mundane work.
Another useful solution is to have a social life that is separate from the scientific community working on the claimed x-risk.
“it’s psychologically appealing to have a hypothesis that means you don’t have to do any mundane work”
I don’t doubt that something like inverse bike-shedding can be a driving force for some individuals to focus on the field of AI safety. I highly doubt it is explanatory for the field and the associated risk predictions to exist in the first place, or that its validity should be questioned on such grounds, but this seems to happen in the article if I’m not entirely misreading it. From my point of view, there is already an overemphasis on psychological factors in the broader debate and it would be desirable to get back to the object level, be it with theoretical or empirical research, which both have their value. This latter aspect seems to lead to a partial agreement here, even though there’s more than one path to arrive at it.
Not entirely. It’s a bit of a misreading. In this case I think the bit matters though.
(And it’s an understandable bit! It’s a subtle point I find I have a hard time communicating clearly.)
I’m trying to say two things:
There sure do seem to be some bad psychological influences going on.
It’s harder to tell what’s real when you have sufficiently bad psychological influences going on.
I think some people, such as you, are reacting really strongly to that second point. Like I’m taking a stand for AI risk being a non-issue and saying it’s all psychological projection.
I’m saying that nonzero, but close to zero. It’s a more plausible hypothesis to me than I think it is to this community. But that’s not because I’m going through the arguments that AI risk is real and finding refutations. It’s because I’ve seen some shockingly basic things turn out to be psychological projection, and I don’t think Less Wrong collectively understands that projection really can be that deep. I just don’t see it accounted for in the arguments for doom.
But that’s not the central point I’m trying to make. My point is more that I think the probability of doom is significantly elevated as a result of how memetic evolution works — and, stupidly, I think that makes doom more likely as a result of the “Don’t hit the tree” phenomenon.
And maybe even more centrally, you cannot know how elevated the probability is until you seriously check for memetic probability boosters. And even then, how you check needs to account for those memetic influences.
I’m not trying to say that AI safety shouldn’t exist as a field though.
Wow, you and I sure must be seeing different parts of the debate! I approximately only hear people talking about the object level. That’s part of my concern.
I mean, I see some folk doing hot takes on Twitter about psychological angles. But most of those strike me as more like pot shots and less like attempts to engage in a dialogue.
This was a great steelmanning, and is exactly the kind of thing I hope people will do in contact with what I offer. Even though I don’t agree with every detail, I feel received and like the thing I care about is being well enough held. Thank you.