Human values are not a fragile, tiny target, but a “natural abstraction” that intelligence tends to converge on.
My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
What we are actually hoping for with ASI is much, much harder to get than an ASI that has human values. Humans are animals seeking control and power, putting ourselves above any other beings in our environment. We don’t want ASI to be seeking control and power, putting itself above any other beings in its environment (which would be the “natural abstraction that intelligence tends to converge on”). We want ASI to selflessly care about the long-term flourishing of humanity, forever and ever. And this certainly is “a fragile, tiny target”. As opposed to power-seeking, this does not naturally follow from instrumental convergence.
I strongly agree with and support this comment. It seems rather unfortunate how frequently people fail to distinguish between “human values”, “humanity’s values”, “human friendly values”, and “humanity friendly values”, each of which seem very importantly distinct from one another, and only the last seems like a worthy goal for ASI.
My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
This claims seems contradictory because part of the IABIED view is that human values are unlikely to be internalized in an ASI’s preferences. Though it’s true if you are defining the “IABIED thesis” as simply “ASI is likely to cause human extinction”.
In my opinion, the IABIED thesis is not just that ASI is an extinction risk, but also the claim that this would occur because of the ASI ending up with alien values misaligned with human survival. In other words, it’s a specific argument about the nature of AI training and AI value construction that leads to human extinction.
These two claims are different:
This seems to be the view you are describing: AI is an extinction risk because it will imitate the worst of human values.
This is the actual IABIED view in my opinion: AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival.
I think we essentially agree. The only difference seems to be with the word “alien” in “AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival”. In my opinion, they may be alien, but more likely they may also be the familiar power-oriented values implied by instrumental convergence, which also coincidentally constitute an important subset of “human values”.
I would argue that more focus is warranted on values as being emergent, independent of whether they are in the training data in some form. The right to live for instance feels very fundamental morally, but it can also be seen as a necessary means for effective cooperation amongst individuals and thus something that will emerge in sufficiently advanced societies of a certain type. The term to put this under would be moral convergent evolution.
It’s not a given that in ASI such a value would emerge as it would be pretty dissimilar to a human. Nevertheless I think it would be very interesting to try to analyze which values seem more probable to emerge in AGI and ASI coming to life among us. Again, (mostly) independent of training data, but just based on evolutionary attractiveness.
My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
What we are actually hoping for with ASI is much, much harder to get than an ASI that has human values. Humans are animals seeking control and power, putting ourselves above any other beings in our environment. We don’t want ASI to be seeking control and power, putting itself above any other beings in its environment (which would be the “natural abstraction that intelligence tends to converge on”). We want ASI to selflessly care about the long-term flourishing of humanity, forever and ever. And this certainly is “a fragile, tiny target”. As opposed to power-seeking, this does not naturally follow from instrumental convergence.
I strongly agree with and support this comment. It seems rather unfortunate how frequently people fail to distinguish between “human values”, “humanity’s values”, “human friendly values”, and “humanity friendly values”, each of which seem very importantly distinct from one another, and only the last seems like a worthy goal for ASI.
This claims seems contradictory because part of the IABIED view is that human values are unlikely to be internalized in an ASI’s preferences. Though it’s true if you are defining the “IABIED thesis” as simply “ASI is likely to cause human extinction”.
In my opinion, the IABIED thesis is not just that ASI is an extinction risk, but also the claim that this would occur because of the ASI ending up with alien values misaligned with human survival. In other words, it’s a specific argument about the nature of AI training and AI value construction that leads to human extinction.
These two claims are different:
This seems to be the view you are describing: AI is an extinction risk because it will imitate the worst of human values.
This is the actual IABIED view in my opinion: AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival.
I think we essentially agree. The only difference seems to be with the word “alien” in “AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival”. In my opinion, they may be alien, but more likely they may also be the familiar power-oriented values implied by instrumental convergence, which also coincidentally constitute an important subset of “human values”.
I would argue that more focus is warranted on values as being emergent, independent of whether they are in the training data in some form. The right to live for instance feels very fundamental morally, but it can also be seen as a necessary means for effective cooperation amongst individuals and thus something that will emerge in sufficiently advanced societies of a certain type. The term to put this under would be moral convergent evolution.
It’s not a given that in ASI such a value would emerge as it would be pretty dissimilar to a human. Nevertheless I think it would be very interesting to try to analyze which values seem more probable to emerge in AGI and ASI coming to life among us. Again, (mostly) independent of training data, but just based on evolutionary attractiveness.