Actually, −(q−u)2 does work, but “by coincidence” and has other negative properties.
Let me explain. First of all, note that things like −(q−u)4 do not work.
To show this: Let u=+2 with probability 1/3, and −1 with probability 2/3 (I’m dropping the 0≤u≤1 for this example, for simplicity). Then E(u)=0 (so the correct q is 0) while E(u3)=2≠0. Then in the expansion of −(q−u)4, you will get 4qu3, which in expectation is not 0. Hence the q1 term in E(−(q−u)4) is non-zero, which means that q=0 cannot be a maximum of this function.
Then why does −(q−u)2 work then? Because it’s −q2+2qu (which is linear in u), minus u2 (non-linear in u, but the AI can’t affect its value, so it’s irrelevant in a boxed setup).
What other “negative properties” might −(q−u)2 have? Suppose we allow the AI to affect the value of u, somehow, by something that is independent of the value of its output q. Then an AI maximising −q2+2qu will always set q=E(u), for a total expectation of E(u)2. Therefore it will also seek to maximise E(u)2, which maximises E(u) if u≥0. So the agent will output the correct q and maximise E(u) simultaneously.
But if it instead tries to maximise −(q−u)2, then it will still pick q=E(u), and gets expected utility of E(u)2−E(u2)=−Var(u). Therefore it will pick actions that minimise the variance of u, irrelevant of expectation.
Even without being able to affect u, this messes up the rest of my setup. In particular, my “pick y and q so that you maximise vy(2qu−q2)” becomes maximising vy(−(q−u)2) and the AI will now select the y that minimises P(Y=y)∗Var(u|Y=y), instead of maximising E(u∧(Y=y)). If ever Var(u|Y=y)=0 or P(Y=y)=0, it will choose those ys.
A loss function plays the same role as a utility function—i.e., we train the learner to minimize its expected loss.
I don’t really understand your remark about linearity. Concretely, why is −(q−u)2 not an appropriate utility function?
Actually, −(q−u)2 does work, but “by coincidence” and has other negative properties.
Let me explain. First of all, note that things like −(q−u)4 do not work.
To show this: Let u=+2 with probability 1/3, and −1 with probability 2/3 (I’m dropping the 0≤u≤1 for this example, for simplicity). Then E(u)=0 (so the correct q is 0) while E(u3)=2≠0. Then in the expansion of −(q−u)4, you will get 4qu3, which in expectation is not 0. Hence the q1 term in E(−(q−u)4) is non-zero, which means that q=0 cannot be a maximum of this function.
Then why does −(q−u)2 work then? Because it’s −q2+2qu (which is linear in u), minus u2 (non-linear in u, but the AI can’t affect its value, so it’s irrelevant in a boxed setup).
What other “negative properties” might −(q−u)2 have? Suppose we allow the AI to affect the value of u, somehow, by something that is independent of the value of its output q. Then an AI maximising −q2+2qu will always set q=E(u), for a total expectation of E(u)2. Therefore it will also seek to maximise E(u)2, which maximises E(u) if u≥0. So the agent will output the correct q and maximise E(u) simultaneously.
But if it instead tries to maximise −(q−u)2, then it will still pick q=E(u), and gets expected utility of E(u)2−E(u2)=−Var(u). Therefore it will pick actions that minimise the variance of u, irrelevant of expectation.
Even without being able to affect u, this messes up the rest of my setup. In particular, my “pick y and q so that you maximise vy(2qu−q2)” becomes maximising vy(−(q−u)2) and the AI will now select the y that minimises P(Y=y)∗Var(u|Y=y), instead of maximising E(u∧(Y=y)). If ever Var(u|Y=y)=0 or P(Y=y)=0, it will choose those ys.