Okay, Eliezer will have worded this much better elsewhere, but I might as well give this a shot. The basic idea of friendly AI is this.
When you design an AI, part of the design that you make is what it is that the AI wants. It doesn’t have any magical defaults that you don’t code in, it is just the code, it is only what you’ve written in to it. If you’ve written it to value something other than human values, it will likely destroy humanity since we are a threat to its values. If you’ve written it to value human values, then it will keep humanity alive and protect us and devote its resources to furthering human values.
It will not change its values, since if it does that it won’t optimize its values. This is practically a tautology, but people still seem to find it surprising.
Thanks for short and clear explanation. Yes, I understand these ideas, even the last point. But with all due respect to Eliezer and others, I don’t think there is a way for us to control a superior being. Some control may work at early stages when AI is not truly intelligent yet, but the idea of fully grown AI implies, by definition, that there is no control over it. Just think about it. This also sounds as a tautology. Of course we can try to always keep AI in an underdeveloped state, so that we can control it, but practically that is not possible. Somebody, somewhere, due to yet another crisis, …, etc, will let it go. It will grow according to some natural informational laws that we don’t know yet and will develop some natural values independent not only of our wishes, but any other contingencies. That’s how I see it. Now you can vote me down.
Pretty much everyone here agrees with you that we can’t control a superintelligent system, most especially Eliezer, who has written many many words championing that position.
So if you’re under the impression that this is a point that you dispute with this community, you have misunderstood the consensus of this community.
In particular, letting a system do what it wants is generally considered the opposite of controlling it.
“So if you’re under the impression that this is a point...”
Yes, I’m under that impression. Because the whole idea about “Friendly AI” implies a subtle, indirect, but still control. The idea here is not to control AI at its final stage, rather to control what this final stage is going to be. But I don’t think such indirect control is possible. Because in my view, the final shape of AI is invariant of any contingencies, including our attempts to make it “friendly” (or “non-friendly”). However, I can admit that on early stages of AI evolution such control may be possible, and even necessary. Therefore, researching “Friendly AI” topic is NOT a waste of time after all. It helps to figure out how to make a transition to the fully grown AI in the least painful way.
Go ahead guys and vote me down. I’m not taking this personally. I understand, this is just a quick way to express your disagreement with my viewpoints. I want to see the count. It’ll give an idea, how strong you disagree with me.
the final shape of AI is invariant of any contingencies
Ah, cool. Yes, this is definitely a point of disagreement.
For my own part, I think real intelligence is necessarily contingent. That is, different minds will respond differently to the same inputs, and this is true regardless of “how intelligent” those minds are. There is no single ideal mind that every mind converges on as its “final” or “fully grown” stage.
I don’t think there is a way for us to control a superior being. Some control may work at early stages when AI is not truly intelligent yet, but the idea of fully grown AI implies, by definition, that there is no control over it.
Yes, this is why Friendly AI is difficult. Making an optimizing process that will care about what we want, in the way we want it to care, once we can no longer control it, is not something we know how to do yet.
Okay, Eliezer will have worded this much better elsewhere, but I might as well give this a shot. The basic idea of friendly AI is this.
When you design an AI, part of the design that you make is what it is that the AI wants. It doesn’t have any magical defaults that you don’t code in, it is just the code, it is only what you’ve written in to it. If you’ve written it to value something other than human values, it will likely destroy humanity since we are a threat to its values. If you’ve written it to value human values, then it will keep humanity alive and protect us and devote its resources to furthering human values.
It will not change its values, since if it does that it won’t optimize its values. This is practically a tautology, but people still seem to find it surprising.
Thanks for short and clear explanation. Yes, I understand these ideas, even the last point. But with all due respect to Eliezer and others, I don’t think there is a way for us to control a superior being. Some control may work at early stages when AI is not truly intelligent yet, but the idea of fully grown AI implies, by definition, that there is no control over it. Just think about it. This also sounds as a tautology. Of course we can try to always keep AI in an underdeveloped state, so that we can control it, but practically that is not possible. Somebody, somewhere, due to yet another crisis, …, etc, will let it go. It will grow according to some natural informational laws that we don’t know yet and will develop some natural values independent not only of our wishes, but any other contingencies. That’s how I see it. Now you can vote me down.
Pretty much everyone here agrees with you that we can’t control a superintelligent system, most especially Eliezer, who has written many many words championing that position.
So if you’re under the impression that this is a point that you dispute with this community, you have misunderstood the consensus of this community.
In particular, letting a system do what it wants is generally considered the opposite of controlling it.
“So if you’re under the impression that this is a point...”
Yes, I’m under that impression. Because the whole idea about “Friendly AI” implies a subtle, indirect, but still control. The idea here is not to control AI at its final stage, rather to control what this final stage is going to be. But I don’t think such indirect control is possible. Because in my view, the final shape of AI is invariant of any contingencies, including our attempts to make it “friendly” (or “non-friendly”). However, I can admit that on early stages of AI evolution such control may be possible, and even necessary. Therefore, researching “Friendly AI” topic is NOT a waste of time after all. It helps to figure out how to make a transition to the fully grown AI in the least painful way.
Go ahead guys and vote me down. I’m not taking this personally. I understand, this is just a quick way to express your disagreement with my viewpoints. I want to see the count. It’ll give an idea, how strong you disagree with me.
This isn’t true of human beings, what’s different about AIs?
Ah, cool. Yes, this is definitely a point of disagreement.
For my own part, I think real intelligence is necessarily contingent. That is, different minds will respond differently to the same inputs, and this is true regardless of “how intelligent” those minds are. There is no single ideal mind that every mind converges on as its “final” or “fully grown” stage.
Yes, this is why Friendly AI is difficult. Making an optimizing process that will care about what we want, in the way we want it to care, once we can no longer control it, is not something we know how to do yet.