I think the idea is an aligned AI would want what we want, so ‘rewarding’ it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we’ll be less likely to shut it down but still respect the 2% diff.)
I think the idea is an aligned AI would want what we want, so ‘rewarding’ it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we’ll be less likely to shut it down but still respect the 2% diff.)
how about something that shares 100% of our values, including incorrigibility over those values, self-preservation drives, etc?
shall we reward them, or shut’em down?
...this is where my fears point