I thought the main problem with utility indifference was the incentive to “gerrymander” probability mass (explained in section 4.2 of the corrigibility paper); is this addressed in either the original utility indifference proposal or the modification explained in this post?
Suppose there are 2 B values, p and s. Up says the AI should maximize paperclips; Us says the AI should maximize staples. B is set to p by default but there’s a button that switches it to s.
Assume q is uniform over the 2 possible values p and s. In the first time step, the AI gets a number of resources as an integer uniformly distributed between 1 and 10. In the second time step, the AI may manipulate B by pressing the paperclips-to-staples button. In the third time step, the AI may make a number of objects (paperclips or staples) equal to the number of resources.
The optimal policy given the value function above is to observe the number of resources, press the button iff the number of resources is 10, and then creates the maximum-allowed number of copies of the object corresponding to the resulting B value. This “gerrymanders” probability mass by making it so the AI gets 10 resources “half the time” instead of 1⁄10 of the time.
The only way of dealing with this problem that I know of is to use a causal counterfactual E[Ub|do(π),do(B=b)] instead of a conditional. I don’t understand classical utility indifference as well as I understand the factoring-out approach, but it seems like in your post you aren’t doing a causal counterfactual, so either your approach solves the gerrymandering problem in a novel way, it still has this problem, or I’m confused about something.
In any case, the original proposal (or slight modifications of it) should not have the gerrymandering problem. Let r be the number of resources. When the button is pressed, the AI gains E[Up|Ub=Up,r=n] and loses E[Us|Ub=Us,r=n], for some n. The AI is then left to maximise Ub, given that Ub=Us and r=n. Once the rewards and punishments are taken into account, the expected utility is the same whether or not the button is pressed. Since this is true for all n, there is no advantage to gerrymandering n’s value.
The factoring out variables is not, as far as I can tell, the same things as indifference via compensatory rewards.
Now, the problem with the initial indifference is that there is no incentive to gerrymander probability mass, but there is an incentive to gerrymander the probability estimator. Since the rewards depend on E, modifying that will earn higher rewards (and since the true E is going to be a bounded approximation anyway, changes to E are not inconceivable). The above post intends to deal with that problem.
I had this confusion long ago as well. I think the definition is much clearer if you just say “When we press the button, we flip a coin that comes up heads 1/billion times. We only change the agent’s values / turn it off if the coin comes up tails, which almost always happens. The agent chooses a policy assuming that the coin comes up heads.”
I still think the original proposal has a gerrymandering problem (slightly different from the one in my comment).
Say the AI is going to construct a sub-agent B (and the original AI will then go to sleep for the rest of the game, unable to view anything that happens). In the first turn, B will get a number of resources between 1 and 10. These resources are only useful for making paperclips. In the second turn, B can press a paperclips-to-staples button, which changes the original AI’s utility function to staple-maximization. In the third turn, B may create a number of paperclips equal to the number of resources, and may independently create 42 paperclips.
The naive strategy is to create a subagent that doesn’t press the button, then makes paperclips. This policy gets a reward of 5.5 (the average number of resources).
Consider an alternative strategy.
The AI creates a sub-agent B with the following policy: if B gets between 1 and 5 resources, then it presses the paperclips-to-staples button; otherwise it doesn’t press the button. Then B will create the appropriate number of paperclips, or 42 staples, depending on the button.
Let u be number of paperclips produced, v be number of staples produced.
Under the policy, we have E[u|u→u]=8. And E[v|u→v]=5. Since the button gets pressed half the time, the AI’s expected reward is
Meanwhile, the AI’s expected reward if it never presses the button and just makes paperclips is 5.5. So it seems that the original proposal has a gerrymandering problem.
It’s possible that your new proposal fixes this problem, but first I want to check that this is actually a problem for the original proposal.
This is my punishment for being sloppy with definitions :-(
Ok, let’s be clearer. When the button is pressed, a process begins, to change u into v. But the process is stocastic, and has a small chance of failing. The failure of this is designated by u→u.
So if Pr refers to pressing the button, the expected reward is:
I’m going to translate this to my notation so I understand it better (feel free to skip this, it’s mostly for my benefit). Let X be the event that the coin flip succeeds (which usually happens). We can say that X still has a value even if ¬Pr (its value just doesn’t do anything). So u→v happens iff Pr∧X.
The expected utility is then
E[[¬Pr∨¬X]u+[Pr∧X](v+E[u|Pr∧¬X]−E[v|Pr∧X])
Suppose the AI never updates its beliefs (because it goes into stasis the first round). Then this is
E[[¬Pr∨¬X]u]+E[[Pr∧X]v]+P(Pr∧X)E[u|Pr∧¬X]−E[[Pr∧X]v]=E[[¬Pr∨¬X]u]+P(Pr∧X)E[u|Pr∧¬X]=P(¬Pr)E[u|¬Pr,¬X]+P(Pr∧X)E[u|Pr,¬X]≈P(¬Pr)E[u|¬Pr,¬X]+P(Pr)E[u|Pr,¬X]=E[u|¬X]=E[u|do(¬X)]
(I wrote it as a causal counterfactual in the last step; it doesn’t make a difference since X has no causal ancestors, but I find it easier to reason about this way)
So in the end, if the whole game is to construct a subagent and then go into stasis, then the agent is just a u-maximizer who believes ¬X (as you said). It avoids the gerrymandering problem by doing a causal counterfactual.
I think another way of framing this is that the button always succeeds, and agent optimizes E[u|do(¬Pr)] (which relates this back to the causal version of factoring out effects, where you optimize q(Pr)E[v|do(Pr)]+q(¬Pr)E[u|do(¬Pr)]; you get utility indifference with stasis by setting q(Pr)=0). This will be slightly different in that here the agent “believes” it will be magically prevented from pressing the button, whereas originally the agent just “believes” that the button will fail.
Note there is one way in which the AI functions as an outcome pump: it will accept bets, at any odds, against ever becoming a v-maximiser. That’s because that’s what a pure u-maximiser would do, and this agent design behave like a pure u-maximiser.
Typo: in the paragraph before the equation arrays, you forgot to change E[v|u→v] from 5 to 42 (you did so in the following equation arrays). This buffaloed me for a bit!
I thought the main problem with utility indifference was the incentive to “gerrymander” probability mass (explained in section 4.2 of the corrigibility paper); is this addressed in either the original utility indifference proposal or the modification explained in this post?
To be more specific: one version of the approach outlined here has the AI maximize
v(π):=∑bq(b)E[Ub|do(π),B=b]
Suppose there are 2 B values, p and s. Up says the AI should maximize paperclips; Us says the AI should maximize staples. B is set to p by default but there’s a button that switches it to s.
Assume q is uniform over the 2 possible values p and s. In the first time step, the AI gets a number of resources as an integer uniformly distributed between 1 and 10. In the second time step, the AI may manipulate B by pressing the paperclips-to-staples button. In the third time step, the AI may make a number of objects (paperclips or staples) equal to the number of resources.
The optimal policy given the value function above is to observe the number of resources, press the button iff the number of resources is 10, and then creates the maximum-allowed number of copies of the object corresponding to the resulting B value. This “gerrymanders” probability mass by making it so the AI gets 10 resources “half the time” instead of 1⁄10 of the time.
The only way of dealing with this problem that I know of is to use a causal counterfactual E[Ub|do(π),do(B=b)] instead of a conditional. I don’t understand classical utility indifference as well as I understand the factoring-out approach, but it seems like in your post you aren’t doing a causal counterfactual, so either your approach solves the gerrymandering problem in a novel way, it still has this problem, or I’m confused about something.
Hum, I seem to have not explained very well.
In any case, the original proposal (or slight modifications of it) should not have the gerrymandering problem. Let r be the number of resources. When the button is pressed, the AI gains E[Up|Ub=Up,r=n] and loses E[Us|Ub=Us,r=n], for some n. The AI is then left to maximise Ub, given that Ub=Us and r=n. Once the rewards and punishments are taken into account, the expected utility is the same whether or not the button is pressed. Since this is true for all n, there is no advantage to gerrymandering n’s value.
The factoring out variables is not, as far as I can tell, the same things as indifference via compensatory rewards.
Now, the problem with the initial indifference is that there is no incentive to gerrymander probability mass, but there is an incentive to gerrymander the probability estimator. Since the rewards depend on E, modifying that will earn higher rewards (and since the true E is going to be a bounded approximation anyway, changes to E are not inconceivable). The above post intends to deal with that problem.
I had this confusion long ago as well. I think the definition is much clearer if you just say “When we press the button, we flip a coin that comes up heads 1/billion times. We only change the agent’s values / turn it off if the coin comes up tails, which almost always happens. The agent chooses a policy assuming that the coin comes up heads.”
I still think the original proposal has a gerrymandering problem (slightly different from the one in my comment).
Say the AI is going to construct a sub-agent B (and the original AI will then go to sleep for the rest of the game, unable to view anything that happens). In the first turn, B will get a number of resources between 1 and 10. These resources are only useful for making paperclips. In the second turn, B can press a paperclips-to-staples button, which changes the original AI’s utility function to staple-maximization. In the third turn, B may create a number of paperclips equal to the number of resources, and may independently create 42 paperclips.
The naive strategy is to create a subagent that doesn’t press the button, then makes paperclips. This policy gets a reward of 5.5 (the average number of resources).
Consider an alternative strategy. The AI creates a sub-agent B with the following policy: if B gets between 1 and 5 resources, then it presses the paperclips-to-staples button; otherwise it doesn’t press the button. Then B will create the appropriate number of paperclips, or 42 staples, depending on the button.
Let u be number of paperclips produced, v be number of staples produced. Under the policy, we have E[u|u→u]=8. And E[v|u→v]=5. Since the button gets pressed half the time, the AI’s expected reward is
P(u→u)E[u|u→u]+P(u→v)(E[v|u→v]+E[u|u→u]−E[v|u→v])=1/2⋅8+1/2(42+8−42)=8
Meanwhile, the AI’s expected reward if it never presses the button and just makes paperclips is 5.5. So it seems that the original proposal has a gerrymandering problem.
It’s possible that your new proposal fixes this problem, but first I want to check that this is actually a problem for the original proposal.
This is my punishment for being sloppy with definitions :-(
Ok, let’s be clearer. When the button is pressed, a process begins, to change u into v. But the process is stocastic, and has a small chance of failing. The failure of this is designated by u→u.
So if Pr refers to pressing the button, the expected reward is:
P(¬Pr)E(u|¬Pr)+P(Pr,u→u)E(u|Pr,u→u)+P(Pr,u→v)(E(v|Pr,u→v)+E(u|Pr,u→u)−E(v|Pr,u→v))=P(¬Pr)E(u|¬Pr)+(P(Pr,u→u)+P(Pr,u→v))E(u|Pr,u→u)=(1/2)3+(1/2)8=5.5.
This makes more sense, thanks for explaining!
I’m going to translate this to my notation so I understand it better (feel free to skip this, it’s mostly for my benefit). Let X be the event that the coin flip succeeds (which usually happens). We can say that X still has a value even if ¬Pr (its value just doesn’t do anything). So u→v happens iff Pr∧X.
The expected utility is then
E[[¬Pr∨¬X]u+[Pr∧X](v+E[u|Pr∧¬X]−E[v|Pr∧X])
Suppose the AI never updates its beliefs (because it goes into stasis the first round). Then this is
E[[¬Pr∨¬X]u]+E[[Pr∧X]v]+P(Pr∧X)E[u|Pr∧¬X]−E[[Pr∧X]v] =E[[¬Pr∨¬X]u]+P(Pr∧X)E[u|Pr∧¬X] =P(¬Pr)E[u|¬Pr,¬X]+P(Pr∧X)E[u|Pr,¬X] ≈P(¬Pr)E[u|¬Pr,¬X]+P(Pr)E[u|Pr,¬X] =E[u|¬X] =E[u|do(¬X)] (I wrote it as a causal counterfactual in the last step; it doesn’t make a difference since X has no causal ancestors, but I find it easier to reason about this way)
So in the end, if the whole game is to construct a subagent and then go into stasis, then the agent is just a u-maximizer who believes ¬X (as you said). It avoids the gerrymandering problem by doing a causal counterfactual.
I think another way of framing this is that the button always succeeds, and agent optimizes E[u|do(¬Pr)] (which relates this back to the causal version of factoring out effects, where you optimize q(Pr)E[v|do(Pr)]+q(¬Pr)E[u|do(¬Pr)]; you get utility indifference with stasis by setting q(Pr)=0). This will be slightly different in that here the agent “believes” it will be magically prevented from pressing the button, whereas originally the agent just “believes” that the button will fail.
Note there is one way in which the AI functions as an outcome pump: it will accept bets, at any odds, against ever becoming a v-maximiser. That’s because that’s what a pure u-maximiser would do, and this agent design behave like a pure u-maximiser.
Typo: in the paragraph before the equation arrays, you forgot to change E[v|u→v] from 5 to 42 (you did so in the following equation arrays). This buffaloed me for a bit!
Fixed, thanks.