Intuitively your thesis doesn’t sound right to me. My guess (1) most people do “reward hack” themselves quite a bit, and (2) to the extent that they don’t, it’s because they care about “doing the real thing.” “Being real” feels to me like something that’s meaningfully different than a lot of my other preference? Like it’s sort of the basis for all other values.
I’d like to clarify that most people do reward hack themselves all the time; I think reward hacking is the default. What I’m grappling with here is rather why people don’t reward-hack themselves literally to death.
In terms of “doing the real thing”, could you elaborate on what “the real thing” means in the frame I’m using of external vs. internal states, or in some other frame you resonate with?
What stops an agent from generating adversarial fulfilment criteria for its goals that are easier to satisfy than the “real”, external goals?
Because like, they terminally don’t want to do? I guess in your frame, what I’d say is that people terminally value having their internal (and noisy) metrics not be too far off from the external states they are supposed to represent.
Intuitively your thesis doesn’t sound right to me. My guess (1) most people do “reward hack” themselves quite a bit, and (2) to the extent that they don’t, it’s because they care about “doing the real thing.” “Being real” feels to me like something that’s meaningfully different than a lot of my other preference? Like it’s sort of the basis for all other values.
I’d like to clarify that most people do reward hack themselves all the time; I think reward hacking is the default. What I’m grappling with here is rather why people don’t reward-hack themselves literally to death.
In terms of “doing the real thing”, could you elaborate on what “the real thing” means in the frame I’m using of external vs. internal states, or in some other frame you resonate with?
Because like, they terminally don’t want to do? I guess in your frame, what I’d say is that people terminally value having their internal (and noisy) metrics not be too far off from the external states they are supposed to represent.