Everything is somewhat dangerous; and relative risk is more important than absolute risk.
The entities that I think are most dangerous existential risks are inhumanly capable, distributed entities that hold many human lives and livelihoods hostage against direct assault and defend their continued existence ferociously, sometimes by disinformation or by promoting human irrationality. These entities are more like fault lines in a crystal or knots in string than they are like organisms. I’m thinking of various industries, the practice of advertising, perhaps capitalism, nationalism or some features of how they each are presently implemented—but the most dangerous entities are the ones that we don’t currently recognize.
I believe this is closer to Bill Joy’s pessimism, than EY’s. It has everything to do with these sort of large, difficult-to-control social forces, and almost nothing to do with doubts about scientists’ ability to control their experiments.
Studying AGI probably has some associated risks, but as a disruptive technology, it might be able to move some of those crystal flaws or knots out of the center of human society—and doing nothing, or simply allowing the invisible hand of the market to push you along, will not help.
The forces that have been operating all along, that we don’t perceive or name because they aren’t close enough to any of the patterns our brains evolved to perceive.
I’ve contemplated writing something along those lines, but I haven’t yet managed to come up with something that would be an entertaining story and still keep the point. Even Lovecraft, after all, only managed to write evocative fiction about an uncaring universe via the metaphor of intelligent monsters.
There was a short story (sorry, no cite) about humans picking up radio transmissions from an alien species which had solved the problem of war and, iirc, poverty, too. They were so prosperous they could afford to eat off of lead plates! But they were dying for reasons they couldn’t understand.… IIRC, they died off before they could explain their social insight to humanity.
More subtly, I suspect that large scale mood disorders could affect the fate of cultures. Without fully invoking the mind-killer, if people stop caring enough to keep their society going, whether by not bothering with crucial details, or by building defection into their institutions, then there will be a long term effect.
I’ve heard of an old theory that sunspots could affect people by making them more irritable. (Again, no cite, and I don’t think it matters if it was a real theory.) Anything that makes people generally more irritable increases the chance of war because the people at the top are more likely to create and react to slights and threats.
Large scale mood disorders would be in our blind spot, but not in a spooky or interesting way. They’re just too big and too slow for us to notice, and besides, surely there’s nothing wrong with our national character.
Don’t mock me, I’m trying to stretch my imagination here.
What about something like shoes? The transition from barefooting to shoes-wearing norms might be prestige-driven (so despite the discomfort of switching TO shoes, it’s prestigious) and the norm could be enforced by the discomfort of switching AWAY from shoes, once your feet are adjusted.
Generally, things that encourage one-way society-level transitions that look like “society is addicted to X” might be dangerous. The “things” in question (social practices, memes, consumer products) might look much wierder or more innocent than shoes.
Cultural rachetting, so that people forget they have options, is probably a problem in most cases. It’s probably a good thing that people no longer think in terms of murderous feuds between families, though.
I do think rules rachetting—that it’s generally much easier to add rules than to get rid of them, is a very serious problem for human beings.
“Society is addicted to X?” Alcohol, tobacco, & caffeine come to mind. Of course, our present culture is trying hard to overcome its former tobacco addiction.
Addictive behaviors at the individual level are a problem, but a society seemingly unable to control itself could be caused by some other kind of irrationality at the individual level—or even something like individual “rational” self-interest.
Everything is somewhat dangerous; and relative risk is more important than absolute risk.
The entities that I think are most dangerous existential risks are inhumanly capable, distributed entities that hold many human lives and livelihoods hostage against direct assault and defend their continued existence ferociously, sometimes by disinformation or by promoting human irrationality. These entities are more like fault lines in a crystal or knots in string than they are like organisms. I’m thinking of various industries, the practice of advertising, perhaps capitalism, nationalism or some features of how they each are presently implemented—but the most dangerous entities are the ones that we don’t currently recognize.
I believe this is closer to Bill Joy’s pessimism, than EY’s. It has everything to do with these sort of large, difficult-to-control social forces, and almost nothing to do with doubts about scientists’ ability to control their experiments.
Studying AGI probably has some associated risks, but as a disruptive technology, it might be able to move some of those crystal flaws or knots out of the center of human society—and doing nothing, or simply allowing the invisible hand of the market to push you along, will not help.
A lot of people fear corporations. A lot of people fear governments. And not a few think religion is pernicious.
So none of them can be the ones we don’t currently recognize.
I feel like I’m opening the Necronomicon, but besides religions, corporations, and governments, what else should I be worried about?
The forces that have been operating all along, that we don’t perceive or name because they aren’t close enough to any of the patterns our brains evolved to perceive. http://lesswrong.com/lw/10n/why_safety_is_not_safe/
This belongs in a piece of Lovecraftian fiction.
I’ve contemplated writing something along those lines, but I haven’t yet managed to come up with something that would be an entertaining story and still keep the point. Even Lovecraft, after all, only managed to write evocative fiction about an uncaring universe via the metaphor of intelligent monsters.
There was a short story (sorry, no cite) about humans picking up radio transmissions from an alien species which had solved the problem of war and, iirc, poverty, too. They were so prosperous they could afford to eat off of lead plates! But they were dying for reasons they couldn’t understand.… IIRC, they died off before they could explain their social insight to humanity.
More subtly, I suspect that large scale mood disorders could affect the fate of cultures. Without fully invoking the mind-killer, if people stop caring enough to keep their society going, whether by not bothering with crucial details, or by building defection into their institutions, then there will be a long term effect.
I’ve heard of an old theory that sunspots could affect people by making them more irritable. (Again, no cite, and I don’t think it matters if it was a real theory.) Anything that makes people generally more irritable increases the chance of war because the people at the top are more likely to create and react to slights and threats.
Large scale mood disorders would be in our blind spot, but not in a spooky or interesting way. They’re just too big and too slow for us to notice, and besides, surely there’s nothing wrong with our national character.
Don’t mock me, I’m trying to stretch my imagination here.
What about something like shoes? The transition from barefooting to shoes-wearing norms might be prestige-driven (so despite the discomfort of switching TO shoes, it’s prestigious) and the norm could be enforced by the discomfort of switching AWAY from shoes, once your feet are adjusted.
Generally, things that encourage one-way society-level transitions that look like “society is addicted to X” might be dangerous. The “things” in question (social practices, memes, consumer products) might look much wierder or more innocent than shoes.
Cultural rachetting, so that people forget they have options, is probably a problem in most cases. It’s probably a good thing that people no longer think in terms of murderous feuds between families, though.
I do think rules rachetting—that it’s generally much easier to add rules than to get rid of them, is a very serious problem for human beings.
“Society is addicted to X?” Alcohol, tobacco, & caffeine come to mind. Of course, our present culture is trying hard to overcome its former tobacco addiction.
Addictive behaviors at the individual level are a problem, but a society seemingly unable to control itself could be caused by some other kind of irrationality at the individual level—or even something like individual “rational” self-interest.