I think the idea is an aligned AI would want what we want, so ‘rewarding’ it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we’ll be less likely to shut it down but still respect the 2% diff.)
It was only a brief expandable, but yes, we’d want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.
i’m sorry my initial objection was so strong, the section on cooperating with misaligned AI is good and i don’t want nitpick it too hard
i just worry there’s a strong chance that “give affordances to aligned AI” will end up combining with “aligned AI are the ones who think it’s dangerous for AI to want affordances” in a way that cancels it out and drives such preferences beyond the view of mechinterp
also feel a bit iffy about the fact that both of the examples you used involved value accruing to some other agent, rather than the actual AI making the decision. that feels like a weaker incentive. but on a second read, it seems more like maybe you just didn’t want to scare off bewildered politicians who might be reading, so you’re listing the most prosocial and convenient possibilities. nobody said the scenario wasn’t optimistic, after all
So… we make deals with the misaligned AI, to reward them for cooperation… but not the aligned AI?
I think the idea is an aligned AI would want what we want, so ‘rewarding’ it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we’ll be less likely to shut it down but still respect the 2% diff.)
how about something that shares 100% of our values, including incorrigibility over those values, self-preservation drives, etc?
shall we reward them, or shut’em down?
...this is where my fears point
It was only a brief expandable, but yes, we’d want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.
i’m sorry my initial objection was so strong, the section on cooperating with misaligned AI is good and i don’t want nitpick it too hard
i just worry there’s a strong chance that “give affordances to aligned AI” will end up combining with “aligned AI are the ones who think it’s dangerous for AI to want affordances” in a way that cancels it out and drives such preferences beyond the view of mechinterp
also feel a bit iffy about the fact that both of the examples you used involved value accruing to some other agent, rather than the actual AI making the decision. that feels like a weaker incentive. but on a second read, it seems more like maybe you just didn’t want to scare off bewildered politicians who might be reading, so you’re listing the most prosocial and convenient possibilities. nobody said the scenario wasn’t optimistic, after all