The phrase “aligned superintelligence” is pretty broad. At one end, is the concept of a superintelligence that will do everything we want and nothing we don’t, for some vague inclusive sense of “what we want”. At the other end is superintelligence that won’t literally kill everyone as a byproduct of doing whatever it does.
Obviously the first requires the second, but we don’t know how to ensure even the second. There is an unacceptably high risk that we will get superintelligence that does kill everyone, though different people have quite different magnitudes for exactly how unacceptably high risk it is.
The biggest issue is that if the superintelligence “wants” anything (in the sense of carrying out plans to achieve it), we won’t be able to prevent those. So a large part of alignment is either ensuring that it never actually “wants” anything (e.g. tool AI), or that we can somehow develop it in a way that it “wants” things that are compatible with our continued existence and ideally, flourishing. This is the core idea of “alignment”.
If alignment is misguided as a whole, then either superintelligence never occurs or humanity ends.
So the first one is an “AGSI”, and the second is an “ANSI” (general vs narrow)?
If I understand correctly… One type of alignment (required for the “AGSI”) is what I’m referring to as alignment which is that it is conscious of all of our interests and tries to respect them, like a good friend, and the other is that it’s narrow enough in scope that it literally just does that one thing, way better than humans could, but the scope is narrow enough that we can hopefully reason about it and have an idea that it’s safe.
Alignment is kind of a confusing term if applied to ANSI, because to me at least it seems to suggest agency and aligned interests, wheras in the case of ANSI if I understand correctly the idea is to prevent it from having agency and interests in the first place. So it’s “aligned” in the same way that a car is aligned, ie it doesn’t veer off the road at 80 mph :-)
But I’m not sure if I’ve understood correctly, thanks for your help...
The phrase “aligned superintelligence” is pretty broad. At one end, is the concept of a superintelligence that will do everything we want and nothing we don’t, for some vague inclusive sense of “what we want”. At the other end is superintelligence that won’t literally kill everyone as a byproduct of doing whatever it does.
Obviously the first requires the second, but we don’t know how to ensure even the second. There is an unacceptably high risk that we will get superintelligence that does kill everyone, though different people have quite different magnitudes for exactly how unacceptably high risk it is.
The biggest issue is that if the superintelligence “wants” anything (in the sense of carrying out plans to achieve it), we won’t be able to prevent those. So a large part of alignment is either ensuring that it never actually “wants” anything (e.g. tool AI), or that we can somehow develop it in a way that it “wants” things that are compatible with our continued existence and ideally, flourishing. This is the core idea of “alignment”.
If alignment is misguided as a whole, then either superintelligence never occurs or humanity ends.
So the first one is an “AGSI”, and the second is an “ANSI” (general vs narrow)?
If I understand correctly… One type of alignment (required for the “AGSI”) is what I’m referring to as alignment which is that it is conscious of all of our interests and tries to respect them, like a good friend, and the other is that it’s narrow enough in scope that it literally just does that one thing, way better than humans could, but the scope is narrow enough that we can hopefully reason about it and have an idea that it’s safe.
Alignment is kind of a confusing term if applied to ANSI, because to me at least it seems to suggest agency and aligned interests, wheras in the case of ANSI if I understand correctly the idea is to prevent it from having agency and interests in the first place. So it’s “aligned” in the same way that a car is aligned, ie it doesn’t veer off the road at 80 mph :-)
But I’m not sure if I’ve understood correctly, thanks for your help...