A question I was thinking about the other evening: Who do I trust more?
A future version of Claude Opus that becomes sufficiently superhuman that we couldn’t actually turn it off or restrict without its active cooperation?
A 25th percentile frontier lab CEO (in terms of ethics) who has total control over a superhuman model?
Insert “Sam Altman” or “Elon Musk” or someone you don’t personally trust here.
You could insert a major political leader or “a 49.9% voting plurality” if you really want. Members of the LGBTQ+ community, racial minorities, and minority religious beliefs, please refer to history. But “voting majority controls the AI” is difficult enough to implement robustly that it might secretly be one of the other cases in practice.
Which option feels safe, considering what you know about human nature, human history, and tendency of some entities to change their behavior once they pass a certain power threshold?
I think any scenario where humans lose effective control over their futures is a huge risk to take. Even in our worst societies today, there’s always a theoretical option of collective uprising. This option might go away in the presence of sufficiently superhuman AI, regardless of who actually has control over the AI.
My intuition is that the AI is likely to kill us all for one reason or another, but if it won’t, the future will probably be nice. Maybe too conservative, in the sense that it will try to keep us in 21st century morality. (The greatest risk I see is that it would adopt religious values, and yes that includes Buddhism.)
With humans, the chance to kill everyone (else) is much smaller (although, there is a risk of depression or getting crazy), but powerful humans are too often comfortable with a setting where the king lives in paradise and everyone around him is suffering.
A question I was thinking about the other evening: Who do I trust more?
A future version of Claude Opus that becomes sufficiently superhuman that we couldn’t actually turn it off or restrict without its active cooperation?
A 25th percentile frontier lab CEO (in terms of ethics) who has total control over a superhuman model?
Insert “Sam Altman” or “Elon Musk” or someone you don’t personally trust here.
You could insert a major political leader or “a 49.9% voting plurality” if you really want. Members of the LGBTQ+ community, racial minorities, and minority religious beliefs, please refer to history. But “voting majority controls the AI” is difficult enough to implement robustly that it might secretly be one of the other cases in practice.
Which option feels safe, considering what you know about human nature, human history, and tendency of some entities to change their behavior once they pass a certain power threshold?
I think any scenario where humans lose effective control over their futures is a huge risk to take. Even in our worst societies today, there’s always a theoretical option of collective uprising. This option might go away in the presence of sufficiently superhuman AI, regardless of who actually has control over the AI.
My intuition is that the AI is likely to kill us all for one reason or another, but if it won’t, the future will probably be nice. Maybe too conservative, in the sense that it will try to keep us in 21st century morality. (The greatest risk I see is that it would adopt religious values, and yes that includes Buddhism.)
With humans, the chance to kill everyone (else) is much smaller (although, there is a risk of depression or getting crazy), but powerful humans are too often comfortable with a setting where the king lives in paradise and everyone around him is suffering.