Hello friends. It’s hard for me to follow the analogies from aliens to AI. Why should we should expect harm from any aliens who may appear?
15:08
Hotz: “If aliens were to show up here, we’re dead, right?”
Yudkowsky: “It depends on the aliens. If I know nothing else about the aliens, I might give them something like a five percent chance of being nice.”
Hotz: “But they have the ability to kill us, right? I mean, they got here, right?”
Yudkowsky: “Oh they absolutely have the ability. Anything that can cross interstellar distances can run you over without noticing—well, they would notice, but they wouldn’t ca—”
[crosstalk]
Hotz: “I didn’t expect this to be a controversial point. But I agree with you that if you’re talking about intelligences that are on the scale of billions of times smarter than humanity… yeah, we’re in trouble.”
Having listened to the whole interview, my best guess is that Hotz believes that advanced civilizations are almost certain to be Prisoner’s Dilemma defectors in the extreme, i.e. they have survived by destroying all other beings they encounter. If so, this is quite disturbing in connection with 12:08, in which Hotz expresses his hope that our civilization will expand across the galaxy (in which case we potentially get to be the aliens).
Hotz seems certain aliens would destroy us, and Eliezer gives them only a five percent chance of being nice.
This is especially odd considering the rapidly growing evidence that humans actually have been frequently seeing and sometimes interacting with a much more advanced intelligence.
It’s been somewhat jarring for my belief in the reality of nonhuman spacecraft to grow by so much in so little time, but overall it has been a great relief to consider the likelihood that another intelligence in this universe has already succeeded in surviving far beyond humankind’s current level of technology. It means that we too could survive the challenges ahead. The high-tech guys might even help us, whoever they are.
But Hotz and Yudkowsky seem to agree that seeing advanced aliens would actually be terrible news. Why?
A 5% chance of nice aliens is better than a 100% chance of human extinction due to AI. Alas 5% seems too high.
The reason the chance is low is because of orthogonality hypothesis. An alien can have many different value systems while still being intelligent, alien value systems can be very diverse, and most alien value systems place no intrinsic value on bipedal humanoids.
A common science fiction intuition pump is to imagine that an evolutionary intelligence explosion happened in a different Earth species and extrapolate likely consequences. There’s also the chance that the aliens are AIs that were not aligned with their biological creators and wiped them out.
Thanks for pointing to the orthogonality thesis as a reason for believing the chance would be low that advanced aliens would be nice to humans. I followed up by reading Bostrom’s “The Superintelligent Will,” and I narrowed down my disagreement to how this point is interpreted:
In a similar vein, even if there are objective moral facts that any fully rational agent would comprehend, and even if these moral facts are somehow intrinsically motivating (such that anybody who fully comprehends them is necessarily motivated to act in accordance with them) this need not undermine the orthogonality thesis. The thesis could still be true if an agent could have impeccable instrumental rationality even whilst lacking some other faculty constitutive of rationality proper, or some faculty required for the full comprehension of the objective moral facts. (An agent could also be extremely intelligent, even superintelligent, without having full instrumental rationality in every domain.)
Just because it’s possible that an agent could have impeccable instrumental rationality while lacking in epistemic rationality to some degree, I expect the typical case that leads to very advanced intelligence would eventually involve synergy between growing both in concert, as many here at Less Wrong are working to do. In other words, a highly competent general intelligence is likely to be curious about objective facts across a very diverse range of topics.
So while aliens could be instrumentally advanced enough to make it to Earth without having ever made basic discoveries in a particular area, there’s no reason for us to expect that it is specifically the area of morality where they will be ignorant or delusional. A safer bet is that they have learned at least as many objective facts as humans have about any given topic on expectation, and that a topic where the aliens have blind spots in relation to some humans is an area where they would be curious to learn from us.
A policy of unconditional harmlessness and friendliness toward all beings is a Schelling Point that could be discovered in many ways. I grant that humans may have it relatively easy to mature on the moral axis because we are conscious, which may or may not be the typical case for general intelligence. That means we can directly experience within our own awareness facts about how happiness is preferred to suffering, how anger and violence lead to suffering, how compassion and equanimity lead to happiness, and so on. We can also see these processes operating in others. But even a superintelligence with no degree of happiness is likely to learn whatever it can from humans, and learning something like love would be a priceless treasure to discover on Earth.
If aliens show up here, I give them at least a 50% chance of being as knowledgeable as the wisest humans in matters of morality. That’s ten times more than Yudkowsky gives them and perhaps infinitely more than Hotz does!
Have humans learnt any objective moral facts? What sort t thing is an objective moral fact? Something like an abstract mathematical theorem , a perceivable object, or a game theoretic equilibrium...?
My view is that humans have learned objective moral facts, yes. For example:
If one acts with an angry or greedy mind, suffering is guaranteed to follow.
I posit that this is not limited to humans. Some people who became famous in history due to their wisdom who I expect would agree include Mother Teresa, Leo Tolstoy, Marcus Aurelius, Martin Luther King Jr., Gandhi, Jesus, and Buddha.
I don’t claim that all humans know all facts about morality. Sadly, it’s probably the case that most people are quite lost, ignorant in matters of virtuous conduct, which is why they find life to be so difficult.
The form you described is called an argument. It requires a series of facts. If you’re working with propositions such as
All beings want to be happy.
No being wants to suffer.
Suffering is caused by confusion and ignorance of morality.
...
then I suppose it could be called a “moral” argument made of “moral” facts and “moral” reasoning, but it’s really just the regular form of an argument made of facts and reasoning. The special thing about moral facts is that direct experience is how they are discovered, and it is that same experiential reality to which they exclusively pertain. I’m talking about the set of moment-by-moment first-person perspectives of sentient beings, such as the familiar one you can investigate right now in real time. Without a being experiencing a sensation come and go, there is no moral consideration to evaluate. NULL.
“Objective moral fact” is Bostrom’s term from the excerpt above, and the phrasing probably isn’t ideal for this discussion. Tabooing such words is no easy feat, but let’s do our best to unpack this. Sticking with the proposition we agree is factual:
If one acts with an angry or greedy mind, suffering is guaranteed to follow.
What kind of fact is this? It’s a fact that can be discovered and/or verified by any sentient being upon investigation of their own direct experience. It is without exception. It is highly relevant for benefiting oneself and others—not just humans. For thousands of years, many people have been revered for articulating it and many more have become consistently happy by basing their decisions on it. Most people don’t; it continues to be a rare piece of wisdom at this stage of civilization. (Horrifyingly, a person on the edge of starting a war or shooting up a school currently would receive advice from ChatGPT to increase “focused, justified anger.”)
Humankind has discovered and recorded a huge body of such knowledge, whatever we wish to call it. If the existence of well-established, verifiable, fundamental insights into the causal nature of experiential reality comes as a surprise to anyone working in fields like psychotherapy or AI alignment, I would urge them to make an earnest and direct inquiry into the matter so they can see firsthand whether such claims have merit. Given the chance, I believe many nonhuman general intelligences would also try and succeed at understanding this kind of information.
(Phew! I packed a lot of words into this comment because I’m too new here to speak more than three times per day. For more on the topic, see the chapter on morality in Dr. Daniel M. Ingram’s book that was reviewed on Slate Star Codex.)
It makes no sense to me that a species that’s evolved to compete with other species will have a higher chance of being nice than a system we can at least somewhat control the development of and take highly detailed “MRI”s of.
Disagree: values come from substrate and environmental. I would almost certainly ally myself with biological aliens versus a digital “humanity” as the biological factor will create a world of much more reasonable values to me.
We are a species that has evolved in competition with other species. Yet, I think there is at least a 5% chance that if we encountered an intelligent alien species that we wouldn’t try to wipe them out (unless they were trying to wipe us out).
Biological evolution of us and aliens would in itself be a commonality, that might produce some common values, whereas there need be no common values with an AI created by a much different process and not successfully aligned.
Biological evolution actively selects for values that we don’t want whereas in AI training we actively select for values we do want. Alien life may not also use the biosphere the same way we do. The usual argument about common values is almost everything needs to breathe air, but at the same time competing and eliminating competing species is a common value among biological life.
Yet, I think there is at least a 5% chance that if we encountered an intelligent alien species that we wouldn’t try to wipe them out (unless they were trying to wipe us out).
Can you tell me why? We have wiped out every other intelligent species more or less. Subgroups of our species are also actively wiping out other subgroups of our species they don’t like.
It think if we encountered aliens who were apparently not hostile, but presumably strange, and likely disgusting or disturbing in some ways, there would be three groups (likely overlapping) of people opposed to wiping them out:
Those who see wiping them out as morally wrong.
Those who see wiping them out as imprudent—we might fail, and then they wipe us out, or other aliens now see us as dangerous, and wipe us out.
Those who see wiping them out as not profitable—better to trade with them.
There would also be three groups in favour of wiping them out:
Those who see wiping them out as morally good—better if the universe doesn’t have such disgusting beings.
Those who see wiping them out as the prudent thing to do—wipe them out before they change their mind and do that to us.
Those who see wiping them out as profitable—then we can grab their resources.
I think it’s clear that people with all these view will exist, in non-negligible numbers. I think there’s at least a 5% chance that the “don’t wipe them out” people prevail.
Subgroups of our species are also actively wiping out other subgroups of our species they don’t like.
Yes, but that’s not how interactions between groups of humans always turn out.
We didn’t really wipe out the Neanderthals (assuming we even were a factor, rather than climate, disease, etc.), seeing as they are among our ancestors.
Hello friends. It’s hard for me to follow the analogies from aliens to AI. Why should we should expect harm from any aliens who may appear?
15:08 Hotz: “If aliens were to show up here, we’re dead, right?” Yudkowsky: “It depends on the aliens. If I know nothing else about the aliens, I might give them something like a five percent chance of being nice.” Hotz: “But they have the ability to kill us, right? I mean, they got here, right?” Yudkowsky: “Oh they absolutely have the ability. Anything that can cross interstellar distances can run you over without noticing—well, they would notice, but they wouldn’t ca—” [crosstalk] Hotz: “I didn’t expect this to be a controversial point. But I agree with you that if you’re talking about intelligences that are on the scale of billions of times smarter than humanity… yeah, we’re in trouble.”
Having listened to the whole interview, my best guess is that Hotz believes that advanced civilizations are almost certain to be Prisoner’s Dilemma defectors in the extreme, i.e. they have survived by destroying all other beings they encounter. If so, this is quite disturbing in connection with 12:08, in which Hotz expresses his hope that our civilization will expand across the galaxy (in which case we potentially get to be the aliens).
Hotz seems certain aliens would destroy us, and Eliezer gives them only a five percent chance of being nice.
This is especially odd considering the rapidly growing evidence that humans actually have been frequently seeing and sometimes interacting with a much more advanced intelligence.
It’s been somewhat jarring for my belief in the reality of nonhuman spacecraft to grow by so much in so little time, but overall it has been a great relief to consider the likelihood that another intelligence in this universe has already succeeded in surviving far beyond humankind’s current level of technology. It means that we too could survive the challenges ahead. The high-tech guys might even help us, whoever they are.
But Hotz and Yudkowsky seem to agree that seeing advanced aliens would actually be terrible news. Why?
A 5% chance of nice aliens is better than a 100% chance of human extinction due to AI. Alas 5% seems too high.
The reason the chance is low is because of orthogonality hypothesis. An alien can have many different value systems while still being intelligent, alien value systems can be very diverse, and most alien value systems place no intrinsic value on bipedal humanoids.
A common science fiction intuition pump is to imagine that an evolutionary intelligence explosion happened in a different Earth species and extrapolate likely consequences. There’s also the chance that the aliens are AIs that were not aligned with their biological creators and wiped them out.
Thanks for pointing to the orthogonality thesis as a reason for believing the chance would be low that advanced aliens would be nice to humans. I followed up by reading Bostrom’s “The Superintelligent Will,” and I narrowed down my disagreement to how this point is interpreted:
Just because it’s possible that an agent could have impeccable instrumental rationality while lacking in epistemic rationality to some degree, I expect the typical case that leads to very advanced intelligence would eventually involve synergy between growing both in concert, as many here at Less Wrong are working to do. In other words, a highly competent general intelligence is likely to be curious about objective facts across a very diverse range of topics.
So while aliens could be instrumentally advanced enough to make it to Earth without having ever made basic discoveries in a particular area, there’s no reason for us to expect that it is specifically the area of morality where they will be ignorant or delusional. A safer bet is that they have learned at least as many objective facts as humans have about any given topic on expectation, and that a topic where the aliens have blind spots in relation to some humans is an area where they would be curious to learn from us.
A policy of unconditional harmlessness and friendliness toward all beings is a Schelling Point that could be discovered in many ways. I grant that humans may have it relatively easy to mature on the moral axis because we are conscious, which may or may not be the typical case for general intelligence. That means we can directly experience within our own awareness facts about how happiness is preferred to suffering, how anger and violence lead to suffering, how compassion and equanimity lead to happiness, and so on. We can also see these processes operating in others. But even a superintelligence with no degree of happiness is likely to learn whatever it can from humans, and learning something like love would be a priceless treasure to discover on Earth.
If aliens show up here, I give them at least a 50% chance of being as knowledgeable as the wisest humans in matters of morality. That’s ten times more than Yudkowsky gives them and perhaps infinitely more than Hotz does!
Have humans learnt any objective moral facts? What sort t thing is an objective moral fact? Something like an abstract mathematical theorem , a perceivable object, or a game theoretic equilibrium...?
My view is that humans have learned objective moral facts, yes. For example:
I posit that this is not limited to humans. Some people who became famous in history due to their wisdom who I expect would agree include Mother Teresa, Leo Tolstoy, Marcus Aurelius, Martin Luther King Jr., Gandhi, Jesus, and Buddha.
I don’t claim that all humans know all facts about morality. Sadly, it’s probably the case that most people are quite lost, ignorant in matters of virtuous conduct, which is why they find life to be so difficult.
It’s not a moral fact, it’s just fact. Moral fact is something of form “and that means that acting with angry or greedy mind is wrong”.
The form you described is called an argument. It requires a series of facts. If you’re working with propositions such as
All beings want to be happy.
No being wants to suffer.
Suffering is caused by confusion and ignorance of morality.
...
then I suppose it could be called a “moral” argument made of “moral” facts and “moral” reasoning, but it’s really just the regular form of an argument made of facts and reasoning. The special thing about moral facts is that direct experience is how they are discovered, and it is that same experiential reality to which they exclusively pertain. I’m talking about the set of moment-by-moment first-person perspectives of sentient beings, such as the familiar one you can investigate right now in real time. Without a being experiencing a sensation come and go, there is no moral consideration to evaluate. NULL.
“Objective moral fact” is Bostrom’s term from the excerpt above, and the phrasing probably isn’t ideal for this discussion. Tabooing such words is no easy feat, but let’s do our best to unpack this. Sticking with the proposition we agree is factual:
What kind of fact is this? It’s a fact that can be discovered and/or verified by any sentient being upon investigation of their own direct experience. It is without exception. It is highly relevant for benefiting oneself and others—not just humans. For thousands of years, many people have been revered for articulating it and many more have become consistently happy by basing their decisions on it. Most people don’t; it continues to be a rare piece of wisdom at this stage of civilization. (Horrifyingly, a person on the edge of starting a war or shooting up a school currently would receive advice from ChatGPT to increase “focused, justified anger.”)
Humankind has discovered and recorded a huge body of such knowledge, whatever we wish to call it. If the existence of well-established, verifiable, fundamental insights into the causal nature of experiential reality comes as a surprise to anyone working in fields like psychotherapy or AI alignment, I would urge them to make an earnest and direct inquiry into the matter so they can see firsthand whether such claims have merit. Given the chance, I believe many nonhuman general intelligences would also try and succeed at understanding this kind of information.
(Phew! I packed a lot of words into this comment because I’m too new here to speak more than three times per day. For more on the topic, see the chapter on morality in Dr. Daniel M. Ingram’s book that was reviewed on Slate Star Codex.)
It makes no sense to me that a species that’s evolved to compete with other species will have a higher chance of being nice than a system we can at least somewhat control the development of and take highly detailed “MRI”s of.
Disagree: values come from substrate and environmental. I would almost certainly ally myself with biological aliens versus a digital “humanity” as the biological factor will create a world of much more reasonable values to me.
We are a species that has evolved in competition with other species. Yet, I think there is at least a 5% chance that if we encountered an intelligent alien species that we wouldn’t try to wipe them out (unless they were trying to wipe us out).
Biological evolution of us and aliens would in itself be a commonality, that might produce some common values, whereas there need be no common values with an AI created by a much different process and not successfully aligned.
Biological evolution actively selects for values that we don’t want whereas in AI training we actively select for values we do want. Alien life may not also use the biosphere the same way we do. The usual argument about common values is almost everything needs to breathe air, but at the same time competing and eliminating competing species is a common value among biological life.
Can you tell me why? We have wiped out every other intelligent species more or less. Subgroups of our species are also actively wiping out other subgroups of our species they don’t like.
Can you tell me why?
It think if we encountered aliens who were apparently not hostile, but presumably strange, and likely disgusting or disturbing in some ways, there would be three groups (likely overlapping) of people opposed to wiping them out:
Those who see wiping them out as morally wrong.
Those who see wiping them out as imprudent—we might fail, and then they wipe us out, or other aliens now see us as dangerous, and wipe us out.
Those who see wiping them out as not profitable—better to trade with them.
There would also be three groups in favour of wiping them out:
Those who see wiping them out as morally good—better if the universe doesn’t have such disgusting beings.
Those who see wiping them out as the prudent thing to do—wipe them out before they change their mind and do that to us.
Those who see wiping them out as profitable—then we can grab their resources.
I think it’s clear that people with all these view will exist, in non-negligible numbers. I think there’s at least a 5% chance that the “don’t wipe them out” people prevail.
Subgroups of our species are also actively wiping out other subgroups of our species they don’t like.
Yes, but that’s not how interactions between groups of humans always turn out.
We didn’t really wipe out the Neanderthals (assuming we even were a factor, rather than climate, disease, etc.), seeing as they are among our ancestors.