My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
This claims seems contradictory because part of the IABIED view is that human values are unlikely to be internalized in an ASI’s preferences. Though it’s true if you are defining the “IABIED thesis” as simply “ASI is likely to cause human extinction”.
In my opinion, the IABIED thesis is not just that ASI is an extinction risk, but also the claim that this would occur because of the ASI ending up with alien values misaligned with human survival. In other words, it’s a specific argument about the nature of AI training and AI value construction that leads to human extinction.
These two claims are different:
This seems to be the view you are describing: AI is an extinction risk because it will imitate the worst of human values.
This is the actual IABIED view in my opinion: AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival.
I think we essentially agree. The only difference seems to be with the word “alien” in “AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival”. In my opinion, they may be alien, but more likely they may also be the familiar power-oriented values implied by instrumental convergence, which also coincidentally constitute an important subset of “human values”.
This claims seems contradictory because part of the IABIED view is that human values are unlikely to be internalized in an ASI’s preferences. Though it’s true if you are defining the “IABIED thesis” as simply “ASI is likely to cause human extinction”.
In my opinion, the IABIED thesis is not just that ASI is an extinction risk, but also the claim that this would occur because of the ASI ending up with alien values misaligned with human survival. In other words, it’s a specific argument about the nature of AI training and AI value construction that leads to human extinction.
These two claims are different:
This seems to be the view you are describing: AI is an extinction risk because it will imitate the worst of human values.
This is the actual IABIED view in my opinion: AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival.
I think we essentially agree. The only difference seems to be with the word “alien” in “AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival”. In my opinion, they may be alien, but more likely they may also be the familiar power-oriented values implied by instrumental convergence, which also coincidentally constitute an important subset of “human values”.