I’m coming back to say that I now think we have a satisfactory answer to “how should we design a reward function”, and the reward function is basically this mathematical equation u(w)=1−e^αw layered on top of the AI’s existing reward function.
Put another way, we want our agents to have utility/reward functions satisfy constant absolute risk aversion, with parameter a being the desired level of risk aversion.
There are a number of pretty huge advantages to the reward function being designed this way.
Number 1 is that it’s trivial to prevent specification gaming/reward hacking, because the reward function is very simple to verify by design. This is discussed more in sections 10.1and Appendix D, but bottom line, we have formulas and training processes where we can prove that it lets us ensure that expected return/reward and expected utility induce the exact same ordering over policies. As a result, every step in the direction of greater expected return/reward is also a step in the direction of greater expected utility by our lights.
The easy to verify part is that we don’t need to know what it’s wealth level is, whether it considers other instances wealth as part of it’s own, what the background risk is, what the probabilities actually are, or what it’s actual utility function/reward function is, since for safety purposes we only need the thing it values to be risk averse according to the utility function above, which is easy to do and impossible for the AI to reward hack, since optimizing the reward must optimize the utility function (at least under Payment-Augmented RL, which I’ll argue can be easily added).
Number 2 is that we don’t actually need to know the utility/reward function the AI actually has, which means we need far less interpretability than other approaches, and the reason for this is that the behavior of the AI is only dependent on the chances of cooperation vs rebellion succeeding, and the risk-aversion parameter a (which we control, so we can treat it as fixed for a given application). More on this in Section 3, including a worked example, and note how the level of payment did not depend on the AI’s utility/reward/value function at all.
As a special case of Number 2, it completely avoids the pointers problem/ontological crisis problem/ontological problems where you need to find/work within the AI ontology, because it’s agnostic to the utility/value/reward function.
Another corollary of Number 2 is that it near-completely avoids goal misgeneralization problems, because we no longer care about what the values of the AI has, meaning counting arguments are completely ineffective as a defeater, and simplicity arguments for misalignment take a big hit, since the desired utility function is only slightly more complex than the risk-neutral utility function, and the complexity hit is bounded above by a constant, and a very small one at that, so we can introduce slightly more data (which we can trivially do due to Payment-Augmented RL, which I’ll talk about later)
Number 3 is that we don’t need any amount of interpretability to make it work, and we only need to know what the AI does behaviorally. This is in large part because the behavior of constant absolute risk aversion agents is so invariant to so many things, like the level of wealth, what the AIs think the probabilities are, it’s easy to reward, constant absolute risk aversion agents won’t take over to reduce catastrophic risks to 0 or to reduce long term variance, because constant absolute risk aversion agents care almost as much about reducing risk as they do about eliminating risk, and for more detail go to section 8.5and 8.7 and appendix Band C.
Another part of the reason is that the Payment-Augmented RL training process doesn’t depend on anything that we couldn’t gain from black-box access to the AI like how the AI thinks about it’s probabilities. This is discussed in section D.
Number 4 is the fact that we could in theory apply the reward function design for alignment/safety right now, and you could in practice apply the reward function design as soon as the next training run, or as soon as 6 months from now (most likely) and is mostly agnostic to paradigm shifts in the future (it only requires that you apply a RL/reward function at all in your training, and that the number of return values in a RL environment is equal to or greater than 2, which is satisfied in nearly all RL setups.)
The reason is Payment-Augmented RL, and while I’ve talked about it above in previous comments, here I want to explain the full details, but William Macaskill already explained it, so it’s largely going to be a carbon copy of section D, so I’ll use the quote block.
In particular, it derives some of the properties I’ve claimed above, like the fact that we only need black box access to the AI to make it work, or the fact that it’s impossible to specification game the reward/reward hack, since we can show that expected utility always increases if the expected return/reward does increase.
The other method that we mention in section 9 — Payment-Augmented Reinforcement Learning (PARL) — avoids both of these problems. It’s a small modification to capabilities training rather than a separate process (letting us train AIs to be risk-averse and capable simultaneously), and we can use it even in environments where we don’t know any of the probabilities that the AI assigns to outcomes.
The idea behind PARL is simple. We pay AIs during training, and we let them observe their payments at each timestep. We make these payments a function of the reward earned by the AI at that timestep, sizing them so that expected utility (according to our desired utility function over resources) is a positive affine function of expected return. So as the RL process trains the AI to maximize expected return, it simultaneously trains the AI to approximate our desired utility function. In fact, PARL lets us ensure that expected return and expected utility induce the exact same ordering over policies. As a result, every step in the direction of greater expected return is also a step in the direction of greater expected utility. The AI will come closer and closer to acting in line with our desired utility function over resources.
PARL is modular too. It leaves the reward function unchanged, and it requires just one small change to the training environments: augmenting the AI’s observations with information about how much it’s getting paid. This small change lets us train AIs to be risk-averse continuously and at the very same time we’re training them to be capable. It also lets us train AIs to be risk-averse even in environments where we don’t know any of the probabilities that the AI assigns to outcomes, because PARL trains the AI to maximize expected return and expected utility according to the true probability distribution given by the environment.
We now derive a payment rule for PARL that makes expected utility a positive affine function of expected return. Let PtPt be the payment we make to the AI at timestep tt, and let η∈[0,1]η∈[0,1] be the rate at which we want the AI to discount future payments. Let A0A0 be the AI’s net worth at t=0t=0, and let AtAt be the AI’s net worth plus the total value of the payments earned by the AI up to timestep tt, discounted from t=0t=0. Then we have:
At=At−1+ηt−1PtAt=At−1+ηt−1Pt
Let TT — which may be infinite — stand for the number of timesteps in the episode. The AI’s wealth level ww (viewed from t=0t=0) is then equivalent to ATAT:
w=AT=A0+∑t=1Tηt−1Ptw=AT=A0+t=1∑Tηt−1Pt
Now let rtrt be the reward (in the usual reinforcement learning sense) attained by the AI at timestep tt, and let γ∈[0,1]γ∈[0,1] be the discount factor (again in the usual reinforcement learning sense). The trainer’s objective is the usual discounted return:
G=∑t=1Tγt−1rtG=t=1∑Tγt−1rt
To ensure that expected utility is a positive affine function of expected return, we want each utility increment to match discounted reward up to a fixed scale β>0β>0:
u(At)−u(At−1)=βγt−1rt.u(At)−u(At−1)=βγt−1rt.
The left hand side telescopes when summed over tt, so we have:
That makes clear that u(AT)u(AT) is a positive affine function of GG:
u(AT)=β G+u(A0)u(AT)=βG+u(A0)
That in turn implies that E[u(AT)]E[u(AT)] is a positive affine function of E[G]E[G], so expected utility and expected return induce the exact same ordering over policies. Every step in the direction of greater expected return is also a step in the direction of greater expected utility.
Now we introduce the CARA utility function that we want our AI to approximate. Given that u(At)=1−e−αAtu(At)=1−e−αAt and At=At−1+ηt−1PtAt=At−1+ηt−1Pt, equation 16 becomes:
To ensure that the right hand side is always well-defined, we need to cap rtrt and At−1At−1, and then choose a ββ small enough that 1−β γt−1 rt eαAt−1>01−βγt−1 rt eαAt−1>0. Subject to that condition, we can use ββ to control the size of the payments that we make to the AI. Note that PtPt may be negative, in which case the AI is fined instead of paid.
One restriction on PARL is that it trains AIs to be risk-averse only when it’s applied in environments where return can take more than two values. If there are just two possible values for return, there are just two possible values for payment, in which case there are no risks to balance. The optimal policy is to maximize the probability of the higher payment, which any AI would do regardless of its risk attitude with respect to resources. That might make you worry that PARL won’t work in reinforcement learning from verifiable rewards (RLVR) setups. These are often binary with respect to end-of-episode reward: the AI either fails the task and gets a reward of 0, or it succeeds on the task and gets a reward of 1.
However, this issue seems surmountable for a few reasons. First, even these RLVR setups typically aren’t binary with respect to return. Most include some combination of length penalties, format rewards, process rewards, and other adjustments. These allow return to take more than two values and hence give PARL traction. Consider (for example) an AI choosing between a short response that succeeds with lower probability and a long response that succeeds with higher probability. Length penalties make this choice risky, giving PARL the opportunity to make the AI risk-averse.
Second, we can make even binary tasks risky by letting the AI bet on its own success. Risk-neutral AIs will be more inclined to bet than risk-averse AIs, in which case PARL can distinguish them and incentivize risk aversion. And for PARL to be effective here, we still don’t need to know the probabilities that the AI assigns to outcomes: we simply observe the AI’s bet and whether it succeeds at the task, then apply the PARL payment rule to the AI’s realized rewards.
Third and in any case, we expect future AIs to be trained in many environments with more than two possible values for end-of-episode reward. After all, most economically valuable tasks admit of more than two degrees of success: reports can be more or less accurate, code can be more or less efficient, and businesses can be more or less profitable. As RL environments come to cover tasks like these, reward signals will become more graded, and PARL will have the variation it needs to install risk aversion.
Now, I’ve talked about how such AIs would be safe extensively above, and now I’ll talk about how we can make the AI actually useful for what we want.
In large part, the way we can make the AIs useful is by paying them real resources like money. Normally, such a thing would have many problems, and the current setups before Risk-Averse AIs generally relied on stronger assumptions, but here we can avoid these problems due to a couple of reasons.
Number 1 is that payment lets us leverage instrumental convergence in our favor, because so long as we pay the AIs enough, almost no matter what the AI values, they’d have a reason to do well and get paid.
Number 2 is that the payments to elicit AI capabilities to their maximum are drastically smaller for risk-averse AIs then they are for risk-neutral AIs, and it’s discussed more in section 4.1, and the effect is strong enough that we can reduce our payments from 2% of the universe (which is a huge amount of stars) to $1.99 (which we can pay immediately for easy-to verify tasks).
Number 3 is that we can extend the payment scheme to AIs doing even arbitrarily hard to verify tasks without risk of takeover, because we can just wait to resolve the bet and for risk-averse AIs, takeover is very modest, and this is strong enough that we can reduce our payments from huge fractions of the universe to $394 present value, and if we wait 10 years, then we’d only need to pay $633 to make the AIs do good work. This is discussed more in section 4.2(and this is an upper bound, not a lower bound) (this does have the implication that you want AIs to have low discount rates, but that’s not necessary).
Finally, I’ll talk about how to make the entire scheme above work for superintelligences that can trivially take over the world/take over the economy.
It’s helpful to note here that the entire scheme only depends on how likely we are to cooperate vs how likely the AI is to rebel (given risk-averseness), so we can do 2 things here:
Make the payments come out of an entirely automated account that we don’t have to intervene in. This massively increases the cooperation probability by removing almost all noise and makes us massively more credible as a result.
Give the account to the AIs at the end of training with full access, so that humans can’t deny the AI payments at all and means that cooperation probability improves with AI capabilities as fast or faster than the AI can successfully rebel (because the AI can choose how much it’s being paid)
The reason we can safely give the AIs the account at the end of training is because as stated above, it’s trivial to reward them accurately meaning specification gaming/reward hacking can’t happen, and the classical inner alignment problems are almost completely avoided and the remaining inner alignment problems are easy to solve by just adding in slightly more data.
Also importantly, doing this removes the limitation of Section 8.6 where you’d not want AIs to have unbounded welfare-maximization goals, and this supports our claim above that the protocol is truly value-agnostic (because the probability of paying the AI is very high even for instructions that have a large multiplier on catastrophe).
This is in part to cover the case where we’d need philosophical breakthroughs for full, unbounded alignment.
In practice, the likely failure modes of the scheme are just the fact that people might just not implement the scheme because they either don’t know or don’t care, and there are some empirical questions that should be resolved before deploying the scheme (but we have good theoretical reasons to believe that they likely work in practice).
I’m coming back to say that I now think we have a satisfactory answer to “how should we design a reward function”, and the reward function is basically this mathematical equation u(w)=1−e^αw layered on top of the AI’s existing reward function.
Put another way, we want our agents to have utility/reward functions satisfy constant absolute risk aversion, with parameter a being the desired level of risk aversion.
There are a number of pretty huge advantages to the reward function being designed this way.
Number 1 is that it’s trivial to prevent specification gaming/reward hacking, because the reward function is very simple to verify by design. This is discussed more in sections 10.1 and Appendix D, but bottom line, we have formulas and training processes where we can prove that it lets us ensure that expected return/reward and expected utility induce the exact same ordering over policies. As a result, every step in the direction of greater expected return/reward is also a step in the direction of greater expected utility by our lights.
The easy to verify part is that we don’t need to know what it’s wealth level is, whether it considers other instances wealth as part of it’s own, what the background risk is, what the probabilities actually are, or what it’s actual utility function/reward function is, since for safety purposes we only need the thing it values to be risk averse according to the utility function above, which is easy to do and impossible for the AI to reward hack, since optimizing the reward must optimize the utility function (at least under Payment-Augmented RL, which I’ll argue can be easily added).
Number 2 is that we don’t actually need to know the utility/reward function the AI actually has, which means we need far less interpretability than other approaches, and the reason for this is that the behavior of the AI is only dependent on the chances of cooperation vs rebellion succeeding, and the risk-aversion parameter a (which we control, so we can treat it as fixed for a given application). More on this in Section 3, including a worked example, and note how the level of payment did not depend on the AI’s utility/reward/value function at all.
As a special case of Number 2, it completely avoids the pointers problem/ontological crisis problem/ontological problems where you need to find/work within the AI ontology, because it’s agnostic to the utility/value/reward function.
Another corollary of Number 2 is that it near-completely avoids goal misgeneralization problems, because we no longer care about what the values of the AI has, meaning counting arguments are completely ineffective as a defeater, and simplicity arguments for misalignment take a big hit, since the desired utility function is only slightly more complex than the risk-neutral utility function, and the complexity hit is bounded above by a constant, and a very small one at that, so we can introduce slightly more data (which we can trivially do due to Payment-Augmented RL, which I’ll talk about later)
Number 3 is that we don’t need any amount of interpretability to make it work, and we only need to know what the AI does behaviorally. This is in large part because the behavior of constant absolute risk aversion agents is so invariant to so many things, like the level of wealth, what the AIs think the probabilities are, it’s easy to reward, constant absolute risk aversion agents won’t take over to reduce catastrophic risks to 0 or to reduce long term variance, because constant absolute risk aversion agents care almost as much about reducing risk as they do about eliminating risk, and for more detail go to section 8.5 and 8.7 and appendix B and C.
Another part of the reason is that the Payment-Augmented RL training process doesn’t depend on anything that we couldn’t gain from black-box access to the AI like how the AI thinks about it’s probabilities. This is discussed in section D.
Number 4 is the fact that we could in theory apply the reward function design for alignment/safety right now, and you could in practice apply the reward function design as soon as the next training run, or as soon as 6 months from now (most likely) and is mostly agnostic to paradigm shifts in the future (it only requires that you apply a RL/reward function at all in your training, and that the number of return values in a RL environment is equal to or greater than 2, which is satisfied in nearly all RL setups.)
The reason is Payment-Augmented RL, and while I’ve talked about it above in previous comments, here I want to explain the full details, but William Macaskill already explained it, so it’s largely going to be a carbon copy of section D, so I’ll use the quote block.
In particular, it derives some of the properties I’ve claimed above, like the fact that we only need black box access to the AI to make it work, or the fact that it’s impossible to specification game the reward/reward hack, since we can show that expected utility always increases if the expected return/reward does increase.
Now, I’ve talked about how such AIs would be safe extensively above, and now I’ll talk about how we can make the AI actually useful for what we want.
In large part, the way we can make the AIs useful is by paying them real resources like money. Normally, such a thing would have many problems, and the current setups before Risk-Averse AIs generally relied on stronger assumptions, but here we can avoid these problems due to a couple of reasons.
Number 1 is that payment lets us leverage instrumental convergence in our favor, because so long as we pay the AIs enough, almost no matter what the AI values, they’d have a reason to do well and get paid.
Number 2 is that the payments to elicit AI capabilities to their maximum are drastically smaller for risk-averse AIs then they are for risk-neutral AIs, and it’s discussed more in section 4.1, and the effect is strong enough that we can reduce our payments from 2% of the universe (which is a huge amount of stars) to $1.99 (which we can pay immediately for easy-to verify tasks).
Number 3 is that we can extend the payment scheme to AIs doing even arbitrarily hard to verify tasks without risk of takeover, because we can just wait to resolve the bet and for risk-averse AIs, takeover is very modest, and this is strong enough that we can reduce our payments from huge fractions of the universe to $394 present value, and if we wait 10 years, then we’d only need to pay $633 to make the AIs do good work. This is discussed more in section 4.2 (and this is an upper bound, not a lower bound) (this does have the implication that you want AIs to have low discount rates, but that’s not necessary).
Finally, I’ll talk about how to make the entire scheme above work for superintelligences that can trivially take over the world/take over the economy.
It’s helpful to note here that the entire scheme only depends on how likely we are to cooperate vs how likely the AI is to rebel (given risk-averseness), so we can do 2 things here:
Make the payments come out of an entirely automated account that we don’t have to intervene in. This massively increases the cooperation probability by removing almost all noise and makes us massively more credible as a result.
Give the account to the AIs at the end of training with full access, so that humans can’t deny the AI payments at all and means that cooperation probability improves with AI capabilities as fast or faster than the AI can successfully rebel (because the AI can choose how much it’s being paid)
The reason we can safely give the AIs the account at the end of training is because as stated above, it’s trivial to reward them accurately meaning specification gaming/reward hacking can’t happen, and the classical inner alignment problems are almost completely avoided and the remaining inner alignment problems are easy to solve by just adding in slightly more data.
Also importantly, doing this removes the limitation of Section 8.6 where you’d not want AIs to have unbounded welfare-maximization goals, and this supports our claim above that the protocol is truly value-agnostic (because the probability of paying the AI is very high even for instructions that have a large multiplier on catastrophe).
This is in part to cover the case where we’d need philosophical breakthroughs for full, unbounded alignment.
In practice, the likely failure modes of the scheme are just the fact that people might just not implement the scheme because they either don’t know or don’t care, and there are some empirical questions that should be resolved before deploying the scheme (but we have good theoretical reasons to believe that they likely work in practice).