Half-rationalists: people who pick up on enough memes to be really dangerous, but not on enough to realise that what they’re doing might be foolish. For example, building an AI without adding the friendliness features.
Not everything is about AI and existential risk. . We already have a section in the sequences about how knowing about cognitive biases can hurt you. It seems unlikely that anyone is going to get the knowledge base to build an AGI from simply being exposed to a few memes here. If there’s one thing that we’ve seen from AI research in the last fifty years is that strong AI is really, really hard.
Rationalists with bad goals: Someone could rationally set about trying to destroy humanity, just for the lulz.
This seems extremely unlikely. Some humans like making things bad for other people. Those people don’t generally want to destroy the world, because the world is where their toys are. Moreover, destroying humanity is something that takes a lot of effort. Barring making bad AGI, getting hold of a large nuclear arsenal, engineering a deadly virus, or making very nasty nanotech, humans don’t have many options for any of these. All of these are very tough. And people who are doing scientific research are generally (although certainly not always) people who aren’t getting much recognition and are doing it because they want to learn and help humanity. The people likely to even want to cause large scale destruction don’t have much overlap with the people who have the capability. The only possible exceptions to this might be some religious and nationalist fanatics in some countries, but that’s not a problem of rationality, and even they can’t trigger existential risk events.
Dangerous information discovered: e.g. the rationalist community develops a Theory of Everything that reveals a recipe for a physics disaster (e.g. a cheap way to turn the Earth into a block hole). A non-rationalist decides to exploit this.
This isn’t a rationalist-community worry. This is a general worry. As technology improves, people individually have more destructive power. That’s a problem completely disconnected from rationalists. Even if such improved rationality did lead to massive tech leaps, it is rarely general theories that immediately give nasty weapons, but rather sophisticated and fairly complicated applications of them along with a lot of engineering. In 1939 the basic theory for atomic weapons existed, but they were developed in secret.
It seems unlikely that anyone is going to get the knowledge base to build an AGI from simply being exposed to a few memes here
Agreed; I was remarking on the danger of being exposed to a few memes on the Uber Less Wrong that we seek to become. Memes which we may have designed to be very accessible and enticing to lay readers.
Those people don’t generally want to destroy the world, because the world is where their toys are
With a population of seven billion, it’s hard not to commit a weak version of the typical mind fallacy and assume you can assume anything about people’s motivations.
As technology improves, people individual have more destructive power
OK; so we should regard as risky any technology or social structure which we expect to significantly advance the rate of technological progress.
With a population of seven billion, it’s hard not to commit a weak version of the typical mind fallacy and assume you can assume anything about people’s motivations.
Diverse as we all are, humans occupy but a tiny pin-point in the mind-design space of all possible intelligent agents. That our values are mostly aligned is one reason that there are so many opportunities for the positive-sum exchanges that have fueled the economic growth of the past several hundred years. I rarely encounter anyone that really truly desires a future that I find horrible (though plenty claim to want such futures). On the other hand, plenty of people go about achieving their goals in all sorts of awfully inefficient ways.
Improving the world’s rationality to the point that people can actually achieve their goals seems like the least of our worries.
Not everything is about AI and existential risk. . We already have a section in the sequences about how knowing about cognitive biases can hurt you. It seems unlikely that anyone is going to get the knowledge base to build an AGI from simply being exposed to a few memes here. If there’s one thing that we’ve seen from AI research in the last fifty years is that strong AI is really, really hard.
This seems extremely unlikely. Some humans like making things bad for other people. Those people don’t generally want to destroy the world, because the world is where their toys are. Moreover, destroying humanity is something that takes a lot of effort. Barring making bad AGI, getting hold of a large nuclear arsenal, engineering a deadly virus, or making very nasty nanotech, humans don’t have many options for any of these. All of these are very tough. And people who are doing scientific research are generally (although certainly not always) people who aren’t getting much recognition and are doing it because they want to learn and help humanity. The people likely to even want to cause large scale destruction don’t have much overlap with the people who have the capability. The only possible exceptions to this might be some religious and nationalist fanatics in some countries, but that’s not a problem of rationality, and even they can’t trigger existential risk events.
This isn’t a rationalist-community worry. This is a general worry. As technology improves, people individually have more destructive power. That’s a problem completely disconnected from rationalists. Even if such improved rationality did lead to massive tech leaps, it is rarely general theories that immediately give nasty weapons, but rather sophisticated and fairly complicated applications of them along with a lot of engineering. In 1939 the basic theory for atomic weapons existed, but they were developed in secret.
Agreed; I was remarking on the danger of being exposed to a few memes on the Uber Less Wrong that we seek to become. Memes which we may have designed to be very accessible and enticing to lay readers.
With a population of seven billion, it’s hard not to commit a weak version of the typical mind fallacy and assume you can assume anything about people’s motivations.
OK; so we should regard as risky any technology or social structure which we expect to significantly advance the rate of technological progress.
Diverse as we all are, humans occupy but a tiny pin-point in the mind-design space of all possible intelligent agents. That our values are mostly aligned is one reason that there are so many opportunities for the positive-sum exchanges that have fueled the economic growth of the past several hundred years. I rarely encounter anyone that really truly desires a future that I find horrible (though plenty claim to want such futures). On the other hand, plenty of people go about achieving their goals in all sorts of awfully inefficient ways.
Improving the world’s rationality to the point that people can actually achieve their goals seems like the least of our worries.