I don’t think this goes through. If I have no preference between two things, but I do prefer to not be money-pumped, it doesn’t seem like I’m going to trade those things so as to be money-pumped.
I am commenting because I think this might be a crucial crux: do smart/rational enough agents always act like maximizers? If not, adequate alignment might be much more feasible than if we need to find exactly the right goal and how to get it into our AGI exactly right.
Human preferences are actually a lot more complex. We value food very highly when hungry and water when we’re thirsty. That can come out of power-seeking, but that’s not actually how it’s implemented. Perhaps more importantly, we might value stamp collecting really highly until we get bored with stamp collecting. I don’t think these can be modeled as a maximizer of any sort.
If humans would pursue multiple goals even if we could edit them (and were smart enough to be consistent), then a similar AGI might only need to be minimally aligned for success. That is, it might stably value human flourishing as a small part of its complex utility function.
I’m not sure whether that’s the case, but I think it’s important.
I don’t think this goes through. If I have no preference between two things, but I do prefer to not be money-pumped, it doesn’t seem like I’m going to trade those things so as to be money-pumped.
I am commenting because I think this might be a crucial crux: do smart/rational enough agents always act like maximizers? If not, adequate alignment might be much more feasible than if we need to find exactly the right goal and how to get it into our AGI exactly right.
Human preferences are actually a lot more complex. We value food very highly when hungry and water when we’re thirsty. That can come out of power-seeking, but that’s not actually how it’s implemented. Perhaps more importantly, we might value stamp collecting really highly until we get bored with stamp collecting. I don’t think these can be modeled as a maximizer of any sort.
If humans would pursue multiple goals even if we could edit them (and were smart enough to be consistent), then a similar AGI might only need to be minimally aligned for success. That is, it might stably value human flourishing as a small part of its complex utility function.
I’m not sure whether that’s the case, but I think it’s important.