Well, yeah, I agree with that! You might notice this item in my candidate “What’s next” list:
Prevent economic incentives from destroying all value. Markets have been remarkably aligned so far but I fear their future effects. (Intelligence Curse, Gradual Disempowerment. Remarkably European takes.)
Market forces are not the only processes which can lead to gradual disempowerment though, something I wasn’t clear about in my comment above.
It could happen that we tell the AI to optimize shareholder value, and them doing that results in the loss of all value. But it could also happen that we tell the AI to optimize shareholder value, which they do until they mostly control the world and then slowly switch to doing whatever they actually care about. This would be a very different failure case than in the first one where we did solve alignment and ‘just gave the AI bad goals’.
You should not assume such things. Humans invented scheming to take over, it might be the very reason we are intelligent.
Here too I might have been unclear: Most people have ‘dreams’ of taking over the world, being king, winning the lottery, or other proxies for gaining a lot of power. I do too. Yet, you will find no evidence of such in any of my work emails, and, if you could read my mind, in any of my thoughts during work.
The only things I seriously scheme about are actionable: Get a promotion, get the best possible deal in a negotiation, found a startup, etc. If you seriously scheme about unactionable things, I claim that you are just wasting time. The same holds for current AIs, hence we don’t see them do it.
If I’m correct, we’ll see more and more scheming/power grabbing with future AI basically any time they think they can get away with it.
We don’t know but we never really know and must act under uncertainty
Well, I consider all of your points on the “What’s next” list to be very worthwhile pursuits. Even if you were completely wrong about alignment being solved, switching to any of those full time would still do a lot of good for the world!
Market forces are not the only processes which can lead to gradual disempowerment though, something I wasn’t clear about in my comment above.
It could happen that we tell the AI to optimize shareholder value, and them doing that results in the loss of all value.
But it could also happen that we tell the AI to optimize shareholder value, which they do until they mostly control the world and then slowly switch to doing whatever they actually care about. This would be a very different failure case than in the first one where we did solve alignment and ‘just gave the AI bad goals’.
Here too I might have been unclear: Most people have ‘dreams’ of taking over the world, being king, winning the lottery, or other proxies for gaining a lot of power. I do too. Yet, you will find no evidence of such in any of my work emails, and, if you could read my mind, in any of my thoughts during work.
The only things I seriously scheme about are actionable: Get a promotion, get the best possible deal in a negotiation, found a startup, etc.
If you seriously scheme about unactionable things, I claim that you are just wasting time. The same holds for current AIs, hence we don’t see them do it.
If I’m correct, we’ll see more and more scheming/power grabbing with future AI basically any time they think they can get away with it.
Well, I consider all of your points on the “What’s next” list to be very worthwhile pursuits. Even if you were completely wrong about alignment being solved, switching to any of those full time would still do a lot of good for the world!