This rhymes with what Paul Christiano and his various interlocutors (e.g. Buck and Ryan above) think, but I think you’ve put forward a much weaker version of it than they do.
To be clear, I’m not at all expecting ASI to “self-align”, “develop values that are benevolent towards us”, or to pursue “cooperation, appeasement and general benevolence”.
(I think you understand my view, after all, you just said “rhyme”, not agree. Regardless, clarifying here.)
What I think is:
Misaligned AIs which takeover probably won’t cause literal human extinction (though large numbers of people might die in the takeover and literal extinction is totally plausible). This takeover would still be extremely bad in expectation for currently living humans and very bad (in my views) from a longtermist perspective (as in, bad for the long run future and acausal interactions).
If the first ASI is misaligned with arbitrary ambitious aims, AI takeover is likely (at least if there aren’t reasonably competitive AIs which are pretty aligned).
To be clear, I’m not at all expecting ASI to “self-align”, “develop values that are benevolent towards us”, or to pursue “cooperation, appeasement and general benevolence”.
(I think you understand my view, after all, you just said “rhyme”, not agree. Regardless, clarifying here.)
What I think is:
Misaligned AIs which takeover probably won’t cause literal human extinction (though large numbers of people might die in the takeover and literal extinction is totally plausible). This takeover would still be extremely bad in expectation for currently living humans and very bad (in my views) from a longtermist perspective (as in, bad for the long run future and acausal interactions).
We might be able to make trades/deals with earlier misaligned AIs that are pretty helpful (and good for both us and AIs).
If the first ASI is misaligned with arbitrary ambitious aims, AI takeover is likely (at least if there aren’t reasonably competitive AIs which are pretty aligned).