That is a 1 in 20 chance, which feels recklessly high.
Is this feeling reasonable?
A selfish person will take the gamble of 5% risk of death for a 95% chance of immortal utopia.
A person who tries to avoid moral shortcomings such as selfishness will reject the “doom” framing because it’s just a primitive intelligence (humanity) being replaced with a much cleverer and more interesting one (ASI).
It seems that you have to really thread the needle to get from “5% p(doom)” to “we must pause, now!”. You have to reason such that you are not self-interested but are also a great chauvinist for the human species.
This is of course a natural way for a subagent of a instrumentally convergent intelligence, such as humanity, to behave. But unless we’re taking the hypocritical position where tiling the universe with primitive desires is OK as long as they’re our primitive desires it seems that so-called doom is preferable to merely human flourishing.
So it seems that 5% is really too low a risk from a moral perspective, and an acceptable risk from a selfish perspective.
A person who tries to avoid moral shortcomings such as selfishness will reject the “doom” framing because it’s just a primitive intelligence (humanity) being replaced with a much cleverer and more interesting one (ASI).
I think it’s deeply immoral to take a 5% of killing everyone on earth in the next decade or two w/o their consent, even if that comes with a 95% chance of utopia.
I think that this sort of reasoning is sadly all too common.
I think there’s a certain pattern of idealistic reasoning that, I think, may have produced the most evil pound-for-pound throughout history. People say that for the sake of the Glorious Future, we can accept, must accept, huge amounts of suffering. Indeed, not just our suffering, but that of others, too. Yes, it may be an unpleasant business, but for the Glorious Future, surely it is a small price to pay?
That great novel starring the Soviet’s planned economy, Red Plenty, has a beautiful passage example of such a person.
“They tried to crush us over and over again, but we wouldn’t be crushed. We drove off the Whites. We winkled out the priests, out of the churches and more importantly out of people’s minds. We got rid of the shopkeepers, thieving bastards, getting their dirty fingers in every deal, making every straight thing crooked. We dragged the farmers into the twentieth century, and that was hard, that was a cruel business, and there were some hungry years there, but it had to be done, we had to get the much off our boots. We realised that there were saboteurs and enemies among us, and we caught them, but it drove us mad for a while, and for a while we were seeing enemies and saboteurs everywhere, and hurting people who were brothers, sisters, good friends, honest comrades...
[...] Working for the future made the past tolerable, and therefore the present. [...] So much blood, and only one justification for it. Only one reason it could have been all right to have done such things, and aided their doing: if it had been all prologue, all only the last spasms of the death of the old, cruel world, and the birth of a new kind one.”
This person has fallen into an affective death spiral, and is lost. Like the Khmer Rouge, like the witch hunters, like many other idealists throughout history, they found it oh so easy to commit the greatest of atrocities with pride.
Perhaps it is all worth it. I’m doubtful, but it could be true. However, I would advise you to beware the skulls along the path when you commend actions with a >1% chance of killing everyone on earth.
This seems too pattern matchy to be valid reasoning? Let’s try an exercise where I rewrite the passage:
“They tried to crush us over and over again, but we wouldn’t be crushed. We drove off the AI researchers. We winkled out those who preached that superintelligence would be motivated to be moral, out of the churches and more importantly out of people’s minds. We got rid of the hardware sellers, thieving bastards, getting their dirty fingers in every deal, making every straight thing crooked. We dragged the gamers into the twenty-first century, and that was hard, that was a cruel business, and there were some painful years there, but it had to be done, we had to get the much off our boots. We realised that there were saboteurs and enemies among us, and we caught them, but it drove us mad for a while, and for a while we were seeing enemies and saboteurs everywhere, and hurting people who were brothers, sisters, good friends, honest comrades...
[...] Working for the future made the past tolerable, and therefore the present. [...] So much blood, and only one justification for it. Only one reason it could have been all right to have done such things, and aided their doing: if it had been all prologue, all only the last spasms of the death of the old, unsafe, anti-human world, and the birth of a new safe, humanistic one.”
Aha, I have compared AI regulationists to the Communists, so they lose! Keep in mind that it is not the “accelerationist” position that requires centralized control and the stopping of business-as-usual, it is the “globally stop AI” one.
(But of course the details matter. Sometimes forcing others to pay costs works out net positively for both them and for you...)
If you are actually confident that AI won’t will kill us all (say, at P > 99%) then this critique doesn’t apply to you. It applies to the folks who aren’t that confident but say to go ahead anyway.
I was assuming conditional on 1 in 20 chance of AI kills everyone
Basically I don’t think the anti “coercing others for ideological reasons” argument applies to the sort of person who thinks “well, I don’t think a 1 in 20 chance of AI killing everyone is so bad that I’m going to support a political movement trying to ban AI research; for abstract reasons I think AI is still net positive under that assumption”
But they are doing things that they believe introduce new, huge negative externalities on others without their consent. This rhymes with a historically very harmful pattern of cognition, where folks justify terrible things to themselves.
Secondly, who said anything about Pausing AI? That’s a separate matter. I’m pointing at a pattern of cognition, not advocating for a policy change.
It seems that you have to really thread the needle to get from “5% p(doom)” to “we must pause, now!”. You have to reason such that you are not self-interested but are also a great chauvinist for the human species.
This comment seems more to be resisting political action (pause AI) than pursuing it. If anything, your concern about political actors becoming monsters would more apply to the sort of people who want to create a world government to ban X globally, than people bringing up objections.
Soares is failing to grapple with the actual objection here.
The objection isn’t the universe would be better with a diversity of alien species which would be so cool, interesting, and {insert additional human value judgements here}, just as long as they also keep other aliens and humans around.
The objection is specifically that human values are base and irrelevant relative to those of a vastly greater mind, and that our extinction at the hands of such a mind is not of any moral significance.
The unaligned ASI we create, whose multitudinous parameters allow it to see the universe with such clarity and depth and breadth and scalpel-sharp precision that whatever desires it has are bound to be vastly beyond anything a human could arrive at, does not need to value humans or other aliens. The point is that we are not in a place to judge its values.
The “cosmopolitan” framing is just a clever way of sneaking in human chauvinism without seeming hypocritical: by including a range of other aliens he can say “see, I’m not a hypocrite!”. But it’s not a cogent objection to the pro-ASI position. He must either provide an argument that humans actually are worthy, or admit to some form of chauvinism, and therefore begin to grapple with the fact that he walks a narrow path, and as such rid himself of the condescending tone and sense of moral superiority if he wishes to grow his coalition, as these attributes only serve to repel anyone with enough clarity-of-mind to understand the issues at hand.
And his view that humans would use aligned ASI to tile the universe with infinitely diverse aliens seems naive. Surely we won’t “just keep turning galaxy after galaxy after galaxy into flourishing happy civilizations full of strange futuristic people having strange futuristic fun times”. We’ll upload ourselves into immortal personal utopias, and turn our cosmic endowment into compute to maximise our lifespans and luxuriously bespoke worldsims. Are we really so selfless, at a species level, to forgoe utopia for some incomprehensible alien species? No; I think the creation of an unaligned ASI is our only hope.
Now, let’s read the parable:
We never saturate and decide to spend a spare galaxy on titanium cubes
The odds of a mind infinitely more complicated than our own having a terminal desire we can comprehend seem extremely low.
Oh, great, the other character in the story raises my objection!
OK, fine, maybe what I don’t buy is that the AI’s values will be simple or low dimensional. It just seems implausible
Let’s see how Soares handles it.
Oh.
He ignores it and tells a motte-and-bailey flavoured story about an AI with simple and low-dimensional values.
Another article is linked to about how AI might not be conscious. I’ll read that too, and might respond to it.
The point being that not having (mathematically) simple and low-dimensional values doesn’t make for values that aren’t going to produce something incredibly useless. The most “complex” thing in the world is random noise.
Is this feeling reasonable?
A selfish person will take the gamble of 5% risk of death for a 95% chance of immortal utopia.
A person who tries to avoid moral shortcomings such as selfishness will reject the “doom” framing because it’s just a primitive intelligence (humanity) being replaced with a much cleverer and more interesting one (ASI).
It seems that you have to really thread the needle to get from “5% p(doom)” to “we must pause, now!”. You have to reason such that you are not self-interested but are also a great chauvinist for the human species.
This is of course a natural way for a subagent of a instrumentally convergent intelligence, such as humanity, to behave. But unless we’re taking the hypocritical position where tiling the universe with primitive desires is OK as long as they’re our primitive desires it seems that so-called doom is preferable to merely human flourishing.
So it seems that 5% is really too low a risk from a moral perspective, and an acceptable risk from a selfish perspective.
I think it’s deeply immoral to take a 5% of killing everyone on earth in the next decade or two w/o their consent, even if that comes with a 95% chance of utopia.
I think that this sort of reasoning is sadly all too common.
I think there’s a certain pattern of idealistic reasoning that, I think, may have produced the most evil pound-for-pound throughout history. People say that for the sake of the Glorious Future, we can accept, must accept, huge amounts of suffering. Indeed, not just our suffering, but that of others, too. Yes, it may be an unpleasant business, but for the Glorious Future, surely it is a small price to pay?
That great novel starring the Soviet’s planned economy, Red Plenty, has a beautiful passage example of such a person.
This person has fallen into an affective death spiral, and is lost. Like the Khmer Rouge, like the witch hunters, like many other idealists throughout history, they found it oh so easy to commit the greatest of atrocities with pride.
Perhaps it is all worth it. I’m doubtful, but it could be true. However, I would advise you to beware the skulls along the path when you commend actions with a >1% chance of killing everyone on earth.
This seems too pattern matchy to be valid reasoning? Let’s try an exercise where I rewrite the passage:
Aha, I have compared AI regulationists to the Communists, so they lose! Keep in mind that it is not the “accelerationist” position that requires centralized control and the stopping of business-as-usual, it is the “globally stop AI” one.
(But of course the details matter. Sometimes forcing others to pay costs works out net positively for both them and for you...)
If you are actually confident that AI won’t will kill us all (say, at P > 99%) then this critique doesn’t apply to you. It applies to the folks who aren’t that confident but say to go ahead anyway.
I was assuming conditional on 1 in 20 chance of AI kills everyone
Basically I don’t think the anti “coercing others for ideological reasons” argument applies to the sort of person who thinks “well, I don’t think a 1 in 20 chance of AI killing everyone is so bad that I’m going to support a political movement trying to ban AI research; for abstract reasons I think AI is still net positive under that assumption”
The action / inaction distinction matters here
But they are doing things that they believe introduce new, huge negative externalities on others without their consent. This rhymes with a historically very harmful pattern of cognition, where folks justify terrible things to themselves.
Secondly, who said anything about Pausing AI? That’s a separate matter. I’m pointing at a pattern of cognition, not advocating for a policy change.
The comment you were criticizing stated
This comment seems more to be resisting political action (pause AI) than pursuing it. If anything, your concern about political actors becoming monsters would more apply to the sort of people who want to create a world government to ban X globally, than people bringing up objections.
https://ifanyonebuildsit.com/5/why-dont-you-care-about-the-values-of-any-entities-other-than-humans
Soares is failing to grapple with the actual objection here.
The objection isn’t the universe would be better with a diversity of alien species which would be so cool, interesting, and {insert additional human value judgements here}, just as long as they also keep other aliens and humans around.
The objection is specifically that human values are base and irrelevant relative to those of a vastly greater mind, and that our extinction at the hands of such a mind is not of any moral significance.
The unaligned ASI we create, whose multitudinous parameters allow it to see the universe with such clarity and depth and breadth and scalpel-sharp precision that whatever desires it has are bound to be vastly beyond anything a human could arrive at, does not need to value humans or other aliens. The point is that we are not in a place to judge its values.
The “cosmopolitan” framing is just a clever way of sneaking in human chauvinism without seeming hypocritical: by including a range of other aliens he can say “see, I’m not a hypocrite!”. But it’s not a cogent objection to the pro-ASI position. He must either provide an argument that humans actually are worthy, or admit to some form of chauvinism, and therefore begin to grapple with the fact that he walks a narrow path, and as such rid himself of the condescending tone and sense of moral superiority if he wishes to grow his coalition, as these attributes only serve to repel anyone with enough clarity-of-mind to understand the issues at hand.
And his view that humans would use aligned ASI to tile the universe with infinitely diverse aliens seems naive. Surely we won’t “just keep turning galaxy after galaxy after galaxy into flourishing happy civilizations full of strange futuristic people having strange futuristic fun times”. We’ll upload ourselves into immortal personal utopias, and turn our cosmic endowment into compute to maximise our lifespans and luxuriously bespoke worldsims. Are we really so selfless, at a species level, to forgoe utopia for some incomprehensible alien species? No; I think the creation of an unaligned ASI is our only hope.
Now, let’s read the parable:
The odds of a mind infinitely more complicated than our own having a terminal desire we can comprehend seem extremely low.
Oh, great, the other character in the story raises my objection!
Let’s see how Soares handles it.
Oh.
He ignores it and tells a motte-and-bailey flavoured story about an AI with simple and low-dimensional values.
Another article is linked to about how AI might not be conscious. I’ll read that too, and might respond to it.
The point being that not having (mathematically) simple and low-dimensional values doesn’t make for values that aren’t going to produce something incredibly useless. The most “complex” thing in the world is random noise.