While looking at the older or more orthodox discussion of notkilleveryoneism, keep this distinction in mind. First AGIs might be safe for a little while, the way humans are “safe”, especially if they are not superintelligences. But then they are liable to build other AGIs that aren’t as safe.
The problem is that supercapable AIs with killeveryone as an instrumental value seem eminently feasible, and general chaos of human condition plus market pressures make them likely to get built. Only regulation of the kind that’s not humanly feasible (and killseveryone if done incorrectly) has a chance of preventing that in the long term, and getting to that point without stepping on an AI that killseveryone is not obviously the default outcome.
While looking at the older or more orthodox discussion of notkilleveryoneism, keep this distinction in mind. First AGIs might be safe for a little while, the way humans are “safe”, especially if they are not superintelligences. But then they are liable to build other AGIs that aren’t as safe.
The problem is that supercapable AIs with killeveryone as an instrumental value seem eminently feasible, and general chaos of human condition plus market pressures make them likely to get built. Only regulation of the kind that’s not humanly feasible (and killseveryone if done incorrectly) has a chance of preventing that in the long term, and getting to that point without stepping on an AI that killseveryone is not obviously the default outcome.