I’m having a great deal of trouble reading this because it’s littered with big red “INVALID EQUATION” and “UNBALANCED EQUATION” censors.
I think your argument regarding “Bob pushing a button” is flawed for a few reasons.
You assume a specific example of a man pushing a button as a stand-in for a more general framework. That examply does not generalize.
In general, the agent gets a reward signal from some untility function. It doesn’t have to even be implemented by a mechanism the agent has agency over. Deep Blue interacts with a chess board. That’s its the environment it has visibility into and it has very limited agency over very limited degrees of freedom within that environment. It can’t blackmail the people who programmed it no matter how much computing power it has. Its reward comes from outside of the agent-environment system.
Even if you limit discussion to more specific cases where the utility mechanism is either embedded in the environment or embedded within the agent (or both, in the case of embedded agent), your claim that “a sufficiently smart agent will … most likely [reward hacking]” lacks justification. The possibility of reward hacking doesn’t imply a high probability thereof. That only makes sense in cases where it’s easier to hack reward than it is to persue the intended goal. You’re assuming specifics in a framework that doesn’t specify them.
On top of all that, you imply that a framework is inherently insuficient if it admits the posibility of reward hacking. Any agent-environment system that partitions the utility mechanism from the policy by some reward channel is succeptible to reward hacking if the agent can gain agency over that channel. That partition means the implicit goal of such an agent is to maximize the reward signal rather than persue the goal formalized in the utility function.
I don’t know of any intelligent system in the real world that isn’t succeptible to reward hacking. People do it all the time. I get that the gravity of the alignment problem sort-of demands solutions with basically zero probability of missalignment, but that may not be possible. I’d like to imagine eliminating the reward channel all together and design a system that directly maximizes a goal, but how do you implement an abstract concept with anything other than indirect indicators? Will the machine be maximizing the good of humanity or will it be maximizing the states of a bunch of transistors which implement some equation that formalizes the good of humanity? Wouldn’t any sufficiently intelligent machine be capable of manipulating those transistors?
I’m having a great deal of trouble reading this because it’s littered with big red “INVALID EQUATION” and “UNBALANCED EQUATION” censors.
I think your argument regarding “Bob pushing a button” is flawed for a few reasons.
You assume a specific example of a man pushing a button as a stand-in for a more general framework. That examply does not generalize.
In general, the agent gets a reward signal from some untility function. It doesn’t have to even be implemented by a mechanism the agent has agency over. Deep Blue interacts with a chess board. That’s its the environment it has visibility into and it has very limited agency over very limited degrees of freedom within that environment. It can’t blackmail the people who programmed it no matter how much computing power it has. Its reward comes from outside of the agent-environment system.
Even if you limit discussion to more specific cases where the utility mechanism is either embedded in the environment or embedded within the agent (or both, in the case of embedded agent), your claim that “a sufficiently smart agent will … most likely [reward hacking]” lacks justification. The possibility of reward hacking doesn’t imply a high probability thereof. That only makes sense in cases where it’s easier to hack reward than it is to persue the intended goal. You’re assuming specifics in a framework that doesn’t specify them.
On top of all that, you imply that a framework is inherently insuficient if it admits the posibility of reward hacking. Any agent-environment system that partitions the utility mechanism from the policy by some reward channel is succeptible to reward hacking if the agent can gain agency over that channel. That partition means the implicit goal of such an agent is to maximize the reward signal rather than persue the goal formalized in the utility function.
I don’t know of any intelligent system in the real world that isn’t succeptible to reward hacking. People do it all the time. I get that the gravity of the alignment problem sort-of demands solutions with basically zero probability of missalignment, but that may not be possible. I’d like to imagine eliminating the reward channel all together and design a system that directly maximizes a goal, but how do you implement an abstract concept with anything other than indirect indicators? Will the machine be maximizing the good of humanity or will it be maximizing the states of a bunch of transistors which implement some equation that formalizes the good of humanity? Wouldn’t any sufficiently intelligent machine be capable of manipulating those transistors?