But why does it matter what we want, if we aren’t ever able to know if what we want is correct for the universe at large?
There is no sense in which what we want may be correct or incorrect for the universe at large, because the universe does not care. Caring is a thing that minds do, and the universe is not a mind.
What if our only purpose is to simply enable the next stage of intelligence, then to disappear into the past?
Our purpose is whatever we choose it to be; purposes are goals seen from another angle. There is no source of purposefulness outside the universe. My goals require that humans stick around, so our purpose with respect to my goal system does not involve disappearing into the past. I think most peoples’ goal systems are similar.
There is no sense in which what we want may be correct or incorrect for the universe at large, because the universe does not care. Caring is a thing that minds do, and the universe is not a mind.
Yes, I agree, and I realize that that isn’t what I was actually trying to say. What I meant was, there is a set of possible, superlatively rational intelligences that may make better use of the universe than humanity (or humanity + a constrained FAI). If Omega reveals to you that such an intelligence would come about if you implement AGI with no Friendly constraint, at the cost of the extinction of humanity, would you build it? This to me drives directly to the heart of whether you value rationality over existence. You don’t personally ‘win’, humanity doesn’t ‘win’, but rationality is maximized.
My goals require that humans stick around, so our purpose with respect to my goal system does not involve disappearing into the past. I think most peoples’ goal systems are similar.
I think we need to unpack that a little, because I don’t think you mean “humans stick around more or less unchanged from their current state”. This is what I was trying to drive at about the Neanderthals. In some sense we ARE Neanderthals, slightly farther along an evolutionary timescale, but you wouldn’t likely feel any moral qualms about their extinction.
So if you do expect that humanity will continue to evolve, probably into something unrecognizable to 21st century humans, in what sense does humanity actually “stick around”? Do you mean you, personally, want to maintain your own conscious self indefinitely, so that no matter what the future, “you” will in some sense be part of it? Or do you mean “whatever intelligent life exists in the future, its ancestry is strictly human”?
“Better” is defined by us. This is the point of the metaethics sequence! A universe tiled with paperclips is not better than what we have now. Rationality is not something one values, it’s someone ones uses to get what they value.
You seem to be imagining FAI as some kind of anthropomorphic intelligence with some sort of “constraint” that says “make sure biological humans continue to exist”. This is exactly the wrong way to implement FAI. The point of FAI is simply for the AI to do what is right (as opposed to what is prime, or paperclip-maximising). In EY’s plan, this involves the AI looking at human minds to discover what we mean by right first.
Now, the right thing may not involve keeping 21st century humanity around forever. Some people will want to be uploaded. Some people will just want better bodies. And yes, most of us will want to “live forever”. But the right thing is definitely not to immediately exterminate the entire population of earth.
This to me drives directly to the heart of whether you value rationality over existence. You don’t personally ‘win’, humanity doesn’t ‘win’, but rationality is maximized.
Why should I value rationality if it results in me losing everything I care about? What is the virtue, to us, of someone else’s rationality?
There is no sense in which what we want may be correct or incorrect for the universe at large, because the universe does not care. Caring is a thing that minds do, and the universe is not a mind.
Our purpose is whatever we choose it to be; purposes are goals seen from another angle. There is no source of purposefulness outside the universe. My goals require that humans stick around, so our purpose with respect to my goal system does not involve disappearing into the past. I think most peoples’ goal systems are similar.
Yes, I agree, and I realize that that isn’t what I was actually trying to say. What I meant was, there is a set of possible, superlatively rational intelligences that may make better use of the universe than humanity (or humanity + a constrained FAI). If Omega reveals to you that such an intelligence would come about if you implement AGI with no Friendly constraint, at the cost of the extinction of humanity, would you build it? This to me drives directly to the heart of whether you value rationality over existence. You don’t personally ‘win’, humanity doesn’t ‘win’, but rationality is maximized.
I think we need to unpack that a little, because I don’t think you mean “humans stick around more or less unchanged from their current state”. This is what I was trying to drive at about the Neanderthals. In some sense we ARE Neanderthals, slightly farther along an evolutionary timescale, but you wouldn’t likely feel any moral qualms about their extinction.
So if you do expect that humanity will continue to evolve, probably into something unrecognizable to 21st century humans, in what sense does humanity actually “stick around”? Do you mean you, personally, want to maintain your own conscious self indefinitely, so that no matter what the future, “you” will in some sense be part of it? Or do you mean “whatever intelligent life exists in the future, its ancestry is strictly human”?
“Better” is defined by us. This is the point of the metaethics sequence! A universe tiled with paperclips is not better than what we have now. Rationality is not something one values, it’s someone ones uses to get what they value.
You seem to be imagining FAI as some kind of anthropomorphic intelligence with some sort of “constraint” that says “make sure biological humans continue to exist”. This is exactly the wrong way to implement FAI. The point of FAI is simply for the AI to do what is right (as opposed to what is prime, or paperclip-maximising). In EY’s plan, this involves the AI looking at human minds to discover what we mean by right first.
Now, the right thing may not involve keeping 21st century humanity around forever. Some people will want to be uploaded. Some people will just want better bodies. And yes, most of us will want to “live forever”. But the right thing is definitely not to immediately exterminate the entire population of earth.
‘Better’ by what standard?
Why should I value rationality if it results in me losing everything I care about? What is the virtue, to us, of someone else’s rationality?