I think this is not obviously qualitatively different from technical oopsie, and sufficiently strong technical success should be able to prevent this. But that’s partially because I think “money and power” is effectively an older, slower AI made of allocating over other minds, and both kinds of AI need to be strongly aligned to flourishing of humans. Fortunately humans with money and power generally want to use it to have nice lives, so on an individual level there should be incentive compatibility if we can find a solution which is general between them. I’m slightly hopeful Richard Ngo’s work might weigh on this, for example.
I think this is not obviously qualitatively different from technical oopsie, and sufficiently strong technical success should be able to prevent this. But that’s partially because I think “money and power” is effectively an older, slower AI made of allocating over other minds, and both kinds of AI need to be strongly aligned to flourishing of humans. Fortunately humans with money and power generally want to use it to have nice lives, so on an individual level there should be incentive compatibility if we can find a solution which is general between them. I’m slightly hopeful Richard Ngo’s work might weigh on this, for example.