Why Not Subagents?
Alternative title for economists: Complete Markets Have Complete Preferences
The justification for modeling real-world systems as “agents”—i.e. choosing actions to maximize some utility function—usually rests on various coherence theorems. They say things like “either the system’s behavior maximizes some utility function, or it is throwing away resources” or “either the system’s behavior maximizes some utility function, or it can be exploited” or things like that. [...]
Now imagine an agent which prefers anchovy over mushroom pizza when it has anchovy, but mushroom over anchovy when it has mushroom; it’s simply never willing to trade in either direction. There’s nothing inherently “wrong” with this; the agent is not necessarily executing a dominated strategy, cannot necessarily be exploited, or any of the other bad things we associate with inconsistent preferences. But the preferences can’t be described by a utility function over pizza toppings.
… that’s what I (John) wrote four years ago, as the opening of Why Subagents?. Nate convinced me that it’s wrong: incomplete preferences do imply a dominated strategy. A system with incomplete preferences, which can’t be described by a utility function, can contract/precommit/self-modify to a more-complete set of preferences which perform strictly better even according to the original preferences.
This post will document that argument.
Epistemic Status: This post is intended to present the core idea in an intuitive and simple way. It is not intended to present a fully rigorous or maximally general argument; our hope is that someone else will come along and more properly rigorize and generalize things. In particular, we’re unsure of the best setting to use for the problem setup; we’ll emphasize some degrees of freedom in the High-Level Potential Problems (And Potential Solutions) section.
Context
The question which originally motivated my interest was: what’s the utility function of the world’s financial markets? Just based on a loose qualitative understanding of coherence arguments, one might think that the inexploitability (i.e. efficiency) of markets implies that they maximize a utility function. In which case, figuring out what that utility function is seems pretty central to understanding world markets.
In Why Subagents?, I argued that a market of utility-maximizing traders is inexploitable, but not itself a utility maximizer. The relevant loophole is that the market has incomplete implied preferences (due to path dependence). Then, I argued that any inexploitable system with incomplete preferences could be viewed as a market/committee of utility-maximizing subagents, making utility-maximizing subagents a natural alternative to outright utility maximizers for modeling agentic systems.
More recently, Nate counterargued that a market of utility maximizers will become a utility maximizer. (My interpretation of) his argument is that the subagents will contract with each other in a way which completes the market’s implied preferences. The model I was previously using got it wrong because it didn’t account for contracts, just direct trade of goods.
More generally, Nate’s counterargument implies that agents with incomplete preferences will tend to precommit/self-modify in ways which complete their preferences.
… but that discussion was over a year ago. Neither of us developed it further, because it just didn’t seem like a core priority. Then the AI Alignment Awards contest came along, and an excellent entry by Elliot Thornley proposed that incomplete preferences could be used as a loophole to make the shutdown problem tractable. Suddenly I was a lot more interested in fleshing out Nate’s argument.
To that end, this post will argue that systems with incomplete preferences will tend to contract/precommit in ways which complete their preferences.
The Pizza Example
As a concrete example of a system of two subagents with incomplete preference, suppose that John and David have different preferences for pizza toppings, and need to choose one together. We both agree that cheese (C) is the least-preferred default option, and sausage (S) is the best. But in between, John prefers pepperoni > mushroom > anchovy (because John lacks taste), while David prefers anchovy > pepperoni > mushroom (because David is a heathen). Specifically, our utilities are:
| David’s Utility | John’s Utility | |
| Cheese | 0 | 0 |
| Mushroom | 1 | 2 |
| Pepperoni | 2 | 3 |
| Anchovy | 3 | 1 |
| Sausage | 4 | 4 |
Or, visually,
Mechanically, for this weird toy model, we’ll imagine that John and David will be offered some number of opportunities (let’s say 3) to trade their current pizza for another, randomly chosen, pizza. If the offered topping is preferred by both of them, then they take the trade. Otherwise, one of them vetos, so they don’t take the trade. Why these particular mechanics? Well, with those mechanics, the preferences can be interpreted in a pretty straightforward way which plays well with other coherence-style arguments—in particular, it’s easy to argue against circular preferences.
(Note that we’re not saying trades have the form e.g. “(mushroom → pepperoni)?” as we would probably usually imagine trades; they instead have the form “(whatever you have now → pepperoni)”. The section Value vs Utility talks about moving our core claim/argument to a more standard notion of trade.)
No Utility Function
One reasonable-seeming way to handle incomplete preferences using a utility function is to just say that two options with “no preference” between them have the same utility—i.e. “no preference” = “indifference”. What goes wrong with that?
Well, in the pizza example, there’s no preference between mushroom and anchovy, so they would have to have the same utility. And there’s no preference between anchovy and pepperoni, so they would have to have the same utility. But that means mushroom and pepperoni have the same utility, which conflicts with the preference for pepperoni over mushroom. So, we can’t represent these preferences via a utility function.
Generalizing: whenever we have state B preferred over state A, and some third state C which has no preference relative to either A or B, we cannot represent the preferences using a utility function. Later, we’ll call that condition “strong incompleteness”, and show that non-strongly incomplete preferences can be represented using a utility function.
Next, let’s see what kind of tricks we can use when preferences are strongly incomplete.
A Contract
Now for the core idea. With these preferences, John and David will turn down trades from mushroom to anchovy (because John vetos it), and turn down trades from anchovy to pepperoni (because David vetos it), even though both prefer pepperoni over mushroom. In principle, both might do better in expectation if John could give up some anchovy for pepperoni, and David could give up some mushroom for anchovy, so that the net shift is from mushroom to pepperoni (a shift which they both prefer).
Before any trade offers come along, John and David sign a contract. John agrees to not veto mushroom → anchovy trades. In exchange, David agrees to not veto anchovy → pepperoni trades. Now the two together have completed their preferences: sausage > pepperoni > anchovy > mushroom > cheese.
… but that won’t always work; it depends on the numbers. For instance, what if there were a lot more opportunities to trade anchovy → pepperoni than mushroom → anchovy? Then an agreement to not veto anchovy → pepperoni would be pretty bad for David, and wouldn’t be fully balanced out by the extra mushroom → anchovy trades. We need some way to make the anchovy → pepperoni trade happen less often (in expectation), to balance things out. If the two trades happen the same amount (in expectation), then there is no expected change in anchovy, just a shift of probability mass from mushroom to pepperoni. Then David and John both do better.
So how do we make the two trades happen the same amount (in expectation)?
One More Trick: Randomization
Solution: randomize the contract. As soon as the contract is signed, some random numbers will be generated. With some probability, John will agree to never veto mushroom → anchovy trades. With some other probability, David will agree to never veto anchovy → pepperoni trades. Then, we choose the two probabilities so that the net expected anchovy states is unchanged by the contract: increasing John’s probability continuously increases expected anchovy, increasing David’s probability continuously decreases expected anchovy, so with the right choice of the two probabilities we can achieve zero expected change in anchovy. Then, the net effect of the contract is to shift some expected mushroom into expected pepperoni; it’s basically a pure win.
That’s the general trick: randomize the preference-completion in such a way that expected anchovy stays the same, while expected mushroom is turned into expected pepperoni.
Claim
Suppose a system has incomplete preferences over a set of states. (For simplicity, we’ll assume the set of states is finite.) Mechanically, this means that there is a “current state” at any given time, and over time the system is offered opportunities to “trade”, i.e. transition to another state; the system accepts a trade A → B if-and-only-if it prefers B over A.
Claim: the system’s preferences can be made complete (via a potentially-randomized procedure) in such a way that the new distribution of states can be viewed as the old distribution with some probability mass shifted from less-preferred to more-preferred states.
Stronger Subclaim
For the argument, we’ll want to split out the case where a strict improvement can be achieved by completing the preferences.
Suppose there exists three states A, B, C such that:
The system prefers B over A
The system has no preference between either A and C or B and C
We’ll call this case “strongly incomplete preferences”. The pizza example involves strongly incomplete preferences: take A to be mushroom, B to be pepperoni, and C to be anchovy.
Claims:
Strongly incomplete preferences can be randomly completed in such a way that the new distribution of states can be viewed as the old distribution with some strictly positive probability mass shifted from less-preferred to more-preferred states.
Non-strongly incomplete preferences (either complete or “weakly incomplete”) encode a utility maximizer.
In other words: strongly incomplete preferences imply that a strict improvement can be achieved by (possibly randomly) completing the preferences, while non-strongly incomplete preferences imply that the system is a utility maximizer.
In the case of weakly incomplete preferences (i.e. incomplete but not strongly incomplete), we also claim that the preferences can be randomly completed in such a way that the system is indifferent between its original preferences and the (expected) randomly-completed preferences, via a similar trick to the rest of the argument. But that’s not particularly practically relevant, so we won’t talk about it further.
Argument
First Step: Strong Incompleteness
In the case of strong incompleteness, we can directly re-use our argument from the pizza example. We have three states A, B, C such that B is preferred over A, and there is no preference between either A and C or B and C. Then, we randomly add a preference for C over A with probability , and we randomly add a preference for B over C with probability .
Frequency of state C increases continuously with , decreases continuously with , and is equal to its original value when both probabilities are 0. So:
Check whether frequency of state C is higher or lower than original when both probabilities are set to 1.
If higher, then set = 1. With = 0 the frequency of state C must then be lower than original (since frequency of C is decreasing in ), with = 1 it’s higher than original by assumption, so by the intermediate value theorem there must be some value such that the frequency of C stays the same. Pick that value.
If lower, swap with and “higher” with “lower” in the previous bullet.
The resulting randomized transformation of the preferences keeps the frequencies of each state the same, except it shifts some probability mass from a less-preferred state (A) to a more-preferred state (B).
(Potential issue with the argument: shifting probability mass from A to B may also shift around probability mass among states downstream of those two states. However, it should generally only shift things in net “good” ways, once we account for the terminal vs instrumental value issue discussed under Value vs Utility. In other words, if we’re using the instrumental value functions, then shifting probability mass from an option valued less by all subagents to one valued more by all subagents should be an expected improvement for all subagents, after accounting for downstream shifts.)
Third Step: Equilibrium Conditions
The second step will argue that non-strongly incomplete preferences encode a utility maximizer. But it’s useful to see how that result will be used before spelling it out, so we’ll do the third step first. To that end, assume that non-strongly incomplete preferences encode a utility maximizer.
Then we have:
If the preferences are strongly incomplete, then there exists some contract/precommitment which “strictly improves” expected outcome states (under the original preferences)
If the preferences are not strongly incomplete, then the system is a utility maximizer.
The last step is to invoke stable equilibrium: strongly incomplete preferences are “unstable” in the sense that the system is incentivized to update from them to more complete preferences, via contract or precommitment. The only preferences which are stable under contracts/precommitments are non-strongly-incomplete preferences, which encode utility maximizers.
Now, we haven’t established which distribution of preferences the system will end up sampling from when randomly completing its preferences, in more complex cases where preferences are strongly incomplete in many places at once. But so long as it ends up at some non-dominated choice, it must end up with non-strongly-incomplete preferences with probability 1 (otherwise it could modify the contract for a strict improvement in cases where it ends up with non-strongly-incomplete preferences). And, so long as the space of possibilities is compact and arbitrary contracts are allowed, all we have left is a bargaining problem. The only way the system would end up with dominated preference-distribution is if there’s some kind of bargaining breakdown.
Point is: non-dominated strategy implies utility maximization.
Second Step: No Strong Incompleteness
Assume the preferences have no strong incompleteness. We’re going to construct a utility function for them. The strategy will be:
Construct “indifference sets”—i.e. sets of states between which the utility function will be indifferent
Show that there is a complete ordering between the “indifference sets”, so we can order them and assign each a utility based on the ordering
Indifference set construction: put each state in its own set. Then, pick two sets such that there is no preference between any states in either set, and merge the two. Iterate to convergence. At this point, the states are partitioned into sets such that:
there are no preferences between any two states in the same set, and
there is at least one preference between at least one pair of states in any two different sets.
Those will turn out to be our indifference sets.
In order to order the indifference sets, we need to show that:
for any pair of states in two different sets, there is a preference between them
the ordering is consistent—i.e. if one state in set S is preferred to one state in set T, then any state in S is preferred to any state in T.
(Also we need acyclicity, but that follows trivially from acyclicity of the preferences once we have these two properties.)
To show those, first recall that there is at least one preference between at least one pair of states in any two different sets. Visually:
In order to have no strong incompleteness, all these preferences must also be present (though we don’t yet know their direction):
Those preferences must be present because, otherwise, we could establish strong incompleteness like this:
We can also establish the direction of each of the preferences by noting that, by assumption, there is no preference between any two states in the same set:
So:
Finally, notice that we’ve now established some more preferences between states in the two sets, so we can repeat the argument with another edge to show that even more preferences are present:
… and once we’ve iterated the argument to convergence, we’ll have the key result: if one state in one set is preferred to another state in another set, then any state in the first set is preferred to any state in the second.
And now we can assign a utility function: order the sets, enumerate them in order, then the number of each set is the utility assigned to each state in that set. A state with a higher utility is always preferred over a state with a lower utility, and there is indifference/no preference between two states with the same utility.
(Note that this is a little different from standard definitions—for mathematical convenience, people typically define utility maximizers to take trades in both directions when indifferent, whereas here our utility maximizer might take trades in both directions between an indifferent pair, or take trades in neither direction between an indifferent pair. For practical purposes, the distinction does not matter; just assume that the agent maintains some small “bid/ask spread”, so a nonzero incentive is needed to induce trade, and the two models become equivalent.)
High-Level Potential Problems With This Argument (And Potential Solutions)
Value vs Utility
Suppose that, in the pizza example, instead of offers to trade a new pizza for whatever pizza David and John currently have, there are offers to trade a specific type of pizza for another specific type—e.g. a mushroom <-> anchovy trade, rather than a mushroom <-> (whatever we have) trade.
In that setup, we might sometimes want to trade “down” to a less-preferred option, in hopes of trading it for a more-preferred option in the future. For instance, if there are lots of vegetarians around offering to trade their sausage pizza for mushroom, then David and John would have high instrumental value for mushroom pizza (because we can probably trade it for sausage), even though neither of us terminally values mushroom. Instrumental and terminal value diverge.
Then, the right way to make the argument would be to calculate the (instrumental) value functions of each subagent (in the dynamic programming sense of the phrase), and use that in place of the (terminal) utilities of each subagent. The argument should then mostly carry over, but there will be one major change: the value function is potentially time-dependent. It’s not “mushroom pizza” which has a value assigned to it, but rather “mushroom pizza at time t”. That, in turn, gets into issues of updating and myopia.
Inconsistent Myopia
A myopic agent in this context would be one which just always trades to more (terminally) preferred options, and never to less (terminally) preferred options, without e.g. strategically trading for a less-preferred mushroom pizza in hopes of later trading the mushroom for more-preferred sausage.
As currently written, our setup implicitly assumed that kind of myopia, which means the subagents are implicitly not thinking about the future when making their choices. … which makes it really weird that the subagents would make contracts/precommitments/self-modications entirely for the sake of future performance. They’re implicitly inconsistently myopic: myopic during trading, but nonmyopic beforehand when choosing to contract/precommit/self-modify.
That said, that kind of inconsistent myopia does make sense for plenty of realistic situations. For instance, maybe the preferences will be myopic during trading, but a designer optimizes those preferences beforehand. Or instead of a designer, maybe evolution/SGD optimizes the preferences.
Alternatively, if the argument is modified to use instrumental values rather than terminal utilities (as the previous section suggested), then the inconsistent myopia issue would be resolved; subagents would simply be non-myopic.
Updating
Once we use instrumental values rather than terminal utilities on states, it’s possible that those values will change over time. They could change purely due to time—for instance, if David and John are hoping to trade a mushroom pizza for sausage, then as the time left to trade winds down, we’ll become increasingly desperate to get rid of that mushroom pizza; its instrumental value falls.
Instrumental value could also change due to information. For instance, if David and John learn that there aren’t as many vegetarians as we expected looking to trade away sausage for mushroom, then that also updates our instrumental value for mushroom pizza.
In order for the argument to work in such situations, the contract/precommitment/self-modification will probably also need to allow for updating over time—e.g. commit to a policy rather than a fixed set of preferences.
Different Beliefs
The argument implicitly assumes that David and John have the same beliefs about what distribution of trade offers we’ll see. If we have different beliefs, then there might not be completion-probabilities which we both find attractive.
On the other hand, if our beliefs differ, then that opens up a whole different set of possible contract-types—e.g. bets and insurance. So there may be some way to use bets/insurance to make the argument work again.
Implications For…
AI Alignment: So much for that idea…
Either we can’t leverage incomplete preferences for safety properties (e.g. shutdownability), or we need to somehow circumvent the above argument.
Economists: If there’s no representative agent, then why ain’t you rich yet?
In economic jargon, completion of the preferences means there exists a representative agent—i.e. the system’s preferences can be summarized by a single utility maximizer. These days most economists assert that there is no representative agent in most real-world markets, so: if there’s no representative agent, then why aren’t you rich yet? And if there is, then what’s its utility function?
Insofar as we view the original incomplete preferences in this model as stemming from multiple subagents with veto power (as in the pizza example), there’s an expected positive sum gain from the contract which completes the preferences. Which means that some third party could, in principle, get paid a share of those gains in exchange for arranging the contract. In practice, most of the work would probably be in designing the contracts in such a way that the benefits are obvious to laypeople, and marketing them. Classic financial engineering business.
So this is the very best sort of economic theorem, where either a useful model holds in the real world, or there’s money to be made.
Conclusions
Main claim, stated two ways:
A group of utility-maximizing subagents have an incentive to form contracts under which they converge to a single utility maximizer
A system with incomplete preferences has an incentive to precommit/self-modify in such a way that the preferences are completed
In general, they do this using randomization over preference-completions. The only expected change each contract/precommitment/self-modification induces is a shift of probability mass from some states to same-or-more-preferred states for all of the subagents; thus each contract is positive-sum.
There is lots more work to be done here, as outlined in the potential problems section. The argument should probably be reframed in terms of value functions (over time) rather than static utility functions in order to more clearly handle instrumentally, though not terminally, valuable actions. The commitments that the subagents make may be better cast as policies rather than fixed preferences. Also, the subagents may have different beliefs about the future, which the argument in this post did not handle.
If the argument holds then this is bad news for alignment hopes that leverage robust incomplete preferences / non E[Utility] maximizers, and also raises some questions about the empirical consensus that modern real-world markets are not expected utility maximizers.
- Apologizing is a Core Rationalist Skill by (2 Jan 2024 17:47 UTC; 152 points)
- Invulnerable Incomplete Preferences: A Formal Statement by (30 Aug 2023 21:59 UTC; 124 points)
- What’s Hard About The Shutdown Problem by (20 Oct 2023 21:13 UTC; 98 points)
- The Shutdown Problem: Incomplete Preferences as a Solution by (23 Feb 2024 16:01 UTC; 50 points)
- Crystal Healing — or the Origins of Expected Utility Maximizers by (25 Jun 2023 3:18 UTC; 49 points)
- The Shutdown Problem: Incomplete Preferences as a Solution by (EA Forum; 23 Feb 2024 16:01 UTC; 26 points)
- The Case for Convexity by (9 Aug 2023 14:09 UTC; 19 points)
- Agents which are EU-maximizing as a group are not EU-maximizing individually by (4 Dec 2023 18:49 UTC; 3 points)
- 's comment on Invulnerable Incomplete Preferences: A Formal Statement by (9 Sep 2023 10:29 UTC; 2 points)
Can we crank this in reverse: given a utility function, design a market that whose representative agent has this utility function?
It seems like trivially, we could just have the market where a singleton agent has the desired utility function. But I wonder if there’s some procedure where we can “peel off” sub-agents within a market and end up at a market composed of the simplest possible sub-agents for some metric of complexity.
Either there is some irreducible complexity there, or maybe there is a Universality theorem proving that we can express any utility function using a market of agents who all have some extremely simple finite state, similar to how we can show any form of computation can be expressed using Turing Machines.
Some nits we know about but didn’t include in the problems section:
P[mushroom->anchovy] = 0. The current argument does not handle the case where subagents believe that there is a probability of 0 on one of the possible states. It wouldn’t be possible to complete the preferences exactly as written, then.
Indifference. If anchovy were placed directly above mushroom in the preference graph above (so that John is truly indifferent between them), then that might require some special handling. But also it might just work if the “Value vs Utility” issue is worked out. If the subagents are not myopic / handle instrumental values, then whether anchovy is less, identically, or more desirable than mushroom doesn’t really matter so much on its own as opposed to what opportunities are possible afterward from the anchovy state relative to the mushroom state.
Also, I think I buy the following part but I really wish it were more constructive.
Great post! Lots of cool ideas. Much to think about.
But I still think both these claims are wrong.
And that’s because you only consider one rule for decision-making with incomplete preferences: a myopic veto rule, according to which the agent turns down a trade if the offered option is ranked lower than its current option according to one or more of the agent’s utility functions.
The myopic veto rule does indeed lead agents to pursue dominated strategies in single-sweetening money-pumps like the one that you set out in the post. I made this point in my coherence theorems post:
But the myopic veto rule isn’t the only possible rule for decision-making with incomplete preferences. Here’s another. I can’t think of a better label right now, so call it ‘Caprice’ since it’s analogous to Brian Weatherson’s rule of the same name for decision-making with multiple probability functions:
Don’t make a sequence of trades (with result X) if there’s another available sequence (with result Y) such that Y is ranked at least as high as X on each of your utility functions and ranked higher than X on at least one of your utility functions. Choose arbitrarily/stochastically among the sequences of trades that remain.
The Caprice Rule implies the policy that I suggested in my coherence theorems post:
If I previously turned down some option Y, I will not settle on any option that I strictly disprefer to Y.
And that makes the agent immune to single-souring money-pumps (in which the agent first has the opportunity to trade in A for B and then (conditional on making that trade) has the opportunity to trade in B for A-).
The Caprice Rule also implies the following policy:
If in future I will be able to settle on some option Y, I will not instead settle on any option that I strictly disprefer to Y.
And that makes the agent immune to single-sweetening money-pumps like the one that you discuss. If the agent recognises that – conditional on trading in mushroom (analogue in my post: A) for anchovy (B) – they will be able to trade in anchovy (B) for pepperoni (A+), then they will make at least the first trade, and thereby avoid pursuing a dominated strategy. As a result, an agent abiding by the Caprice Rule can’t shift probability mass from mushroom (A) to pepperoni (A+) by probabilistically precommitting to take certain trades in a way that makes their preferences complete. The Caprice Rule already does the shift.
And an agent abiding by the Caprice Rule can’t be represented as maximising utility, because its preferences are incomplete. In cases where the available trades aren’t arranged in some way that constitutes a money-pump, the agent can prefer (/reliably choose) A+ over A, and yet lack any preference between (/stochastically choose between) A+ and B, and lack any preference between (/stochastically choose between) A and B. Those patterns of preference/behaviour are allowed by the Caprice Rule.
For a Caprice-Rule-abiding agent to avoid pursuing dominated strategies in single-sweetening money-pumps, that agent must be non-myopic: specifically, it must recognise that trading in A for B and then B for A+ is an available sequence of trades. And you might think that this is where my proposal falls down: actual agents will sometimes be myopic, so actual agents can’t always use the Caprice Rule to avoid pursuing dominated strategies, so actual agents are incentivised to avoid pursuing dominated strategies by instead probabilistically precommitting to take certain trades in ways that make their preferences complete (as you suggest).
But there’s a problem with this response. Suppose an agent is myopic. It finds itself with a choice between A and B, and it chooses A. As a matter of fact, if it had chosen B, it would have later been offered A+. Then the agent leaves with A when it could have had A+. But since the agent is myopic, it won’t be aware of this fact, and so note two things. First, it’s unclear whether the agent’s behaviour deserves the name ‘dominated strategy’. The agent pursues a dominated strategy only in the same sense that I pursue a dominated strategy when I fail to buy a lottery ticket that (unbeknownst to me) would have won. Second and more importantly, the agent’s failure to get A+ won’t lead the agent to change its preferences, since it’s myopic and so unaware that A+ was available.
And so we seem to have a dilemma for money-pumps for completeness. In money-pumps where the agent is non-myopic about the available sequences of trades, the agent can avoid pursuit of dominated strategies by acting in accordance with the Caprice Rule. In money-pumps where the agent is myopic, failing to get A+ exerts no pressure on the agent to change its preferences, since the agent is not aware that it could have had A+.
You recognise this in the post and so set things up as follows: a non-myopic optimiser decides the preferences of a myopic agent. But this means your argument doesn’t vindicate coherence arguments as traditionally conceived. Per my understanding, the conclusion of coherence arguments was supposed to be: you can’t rely on advanced agents not to act like expected-utility-maximisers, because even if these agents start off not acting like EUMs, they’ll recognise that acting like an EUM is the only way to avoid pursuing dominated strategies. I think that’s false, for the reasons that I give in my coherence theorems post and in the paragraph above. But in any case, your argument doesn’t give us that conclusion. Instead, it gives us something like: a non-myopic optimiser of a myopic agent can shift probability mass from less-preferred to more-preferred outcomes by probabilistically precommitting the agent to take certain trades in a way that makes its preferences complete. That’s a cool result in its own right, and maybe your post isn’t trying to vindicate coherence arguments as traditionally conceived, but it seems worth saying that it doesn’t.
You’re right that a non-myopic designer might set things up so that their myopic agent’s preferences are complete. And maybe SGD makes this hard to avoid. But if I’m right about the shutdown problem, we as non-myopic designers should try to set things up so that our agent’s preferences are incomplete. That’s our best shot at getting a corrigible agent. Training by SGD might present an obstacle to this (I’m still trying to figure this out), but coherence arguments don’t.
That’s how I think the argument in your post can be circumvented, and why I still think we can use incomplete preferences for shutdownability/corrigibility:
That’s the main point I want to make. Here’s a more minor point: I think that even in the case where you have a non-myopic optimiser deciding the preferences of a myopic agent, non-domination by itself doesn’t imply utility maximisation. You also need the assumption that the non-myopic optimiser takes some kinds of money-pumps to be more likely than others. Here’s an example to illustrate why I think that. Suppose that our non-myopic optimiser predicts that each of the following money-pumps are equally likely to occur, with probability 0.5. Call the first ‘the A+ money-pump’ and the second ‘the B+ money-pump’:
A+ money-pump
B+ money-pump
The non-myopic optimiser knows that the agent will be myopic in deployment. Currently, the agent’s preferences are incomplete: it lacks a preference between A and B. Either it abides by the veto rule and sticks with whatever it already has, or it chooses stochastically between A and B. That difference won’t matter here: we can just say that the agent chooses A with probability p and chooses B with probability 1-p. The non-myopic optimiser is considering precommitting the agent to choose either A or B with probability 1, with the consequence that the agent’s preferences would then be complete. Does precommitting dominate not precommitting?
No. The agent pursues a dominated strategy if and only if the A+ money-pump occurs and the agent chooses A or the B+ money-pump occurs and the agent chooses B. As it stands, those probabilities are 0.5, p, 0.5, and 1-p respectively, so that the agent’s probability of pursuing a dominated strategy is 0.5p+0.5(1-p)=0.5. And the non-myopic optimiser can’t change this probability by precommitting the agent to choose A or B. Doing so changes only the value of p, and 0.5p+0.5(1-p)=0.5 no matter what the value of p.
That’s why I think you also need the assumption that the non-myopic optimiser believes that the myopic agent is more likely to encounter some kinds of money-pumps than others in deployment. The non-myopic optimiser has to think, e.g., that the A+ money-pump is more likely than the B+ money-pump. Then making the agent’s preferences complete can decrease the probability that the agent pursues a dominated strategy. But note a few things:
(1) If the probabilities of the A+ money-pump and the B+ money-pump are each non-zero, then precommitting the agent to choose one of A and B doesn’t just shift probability mass from a less-preferred outcome to a more-preferred outcome. It also shifts probability mass between A and B, and between A+ and B+. For example, precommitting to always choose A sends the probability of B and of A+ down to zero. And it’s not so clear that the new probability distribution is superior to the old one. This new probability distribution does give a smaller probability of the agent pursuing a dominated strategy, but minimising the probability of pursuing a dominated strategy isn’t always best. Consider an example with complete preferences:
First A- money-pump
Second A- money-pump
Suppose the probability of the First A- money-pump is 0.6 and the probability of the Second A- money-pump is 0.4. Then precommitting to always choose A- minimises the probability of pursuing a dominated strategy. But if the difference in value between A- and A is much greater than the difference in value between A and A+, then it would be better to precommit to choosing A.
(2) As the point above suggests, given your set-up of a non-myopic optimiser deciding the preferences of a myopic agent, and the assumption that some kinds of decision-trees are more likely than others, it can also be that the non-myopic optimiser can decrease the probability that an agent with complete preferences pursues a dominated strategy by precommitting the agent to take certain trades. You make something like this point in the ‘Value vs Utility’ section: if there are lots of vegetarians around, you might want to trade down to mushroom pizza. And you can see it by considering the First A- money-pump above: if that’s especially likely, the non-myopic optimiser might want to precommit the agent to trade in A for A-. This makes me think that the lesson of the post is more about the instrumental value of commitments in your non-myopic-then-myopic setting than it is about incomplete preferences.
(3) Return to the A+ money-pump and the B+ money-pump from above, and suppose that their probabilities are 0.6 and 0.4 respectively. Then the non-myopic optimiser can decrease the probability of the myopic agent pursuing a dominated strategy by precommitting the agent to always choose B, but doing so will only send that probability down to 0.4. If the non-myopic optimiser wants the probability of a dominated strategy lower than that, it has to make the agent non-myopic. And in cases where an agent with incomplete preferences is non-myopic, it can avoid pursuing dominated strategies by acting in accordance with the Caprice Rule.
Wait… doesn’t the caprice rule just directly modify its preferences toward completion over time? Like, every time a decision comes up where it lacks a preference, a new preference (and any implied by it) will be added to its preferences.
Intuitively: of course the caprice rule would be indifferent to completing its preferences up-front via contract/commitment, because it expects to complete its preferences over time anyway; it’s just lazy about the process (in the “lazy data structure” sense).
Yeah ‘indifference to completing preferences’ remains an issue and I’m still trying to figure out if there’s a way to overcome it. I don’t think ‘expects to complete its preferences over time’ plays a role, though. I think the indifference to completing preferences is just a consequence of the fact that turning preferential gaps into strict preferences won’t lead the agent to behave in ways that it disprefers from its current perspective. I go into a bit more detail on this in my contest entry:
On directly modifying preferences towards completion over time, that’s right but the agent’s preferences will only become complete once it’s had the opportunity to choose a sufficiently wide array of options. Depending on the details, that might never happen or only happen after a very long time. I’m still trying to figure out the details.
Can you explain more how this might work?
I might be totally wrong about this, but if you have a myopic agent with preferences A>B, B>C and C>A, it’s not totally clear to me why they would change those preferences to act like an EUM. Sure, if you keep offering them a trade where they can pay small amounts to move in these directions, they’ll go round and round the cycle and only lose money, but do they care? At each timestep, their preferences are being satisfied. To me, the reason you can expect a suitably advanced agent to not behave like this is that they’ve been subjected to a selection pressure / non-myopic optimiser that is penalising their losses.
This seems right to me. It feels weird to talk about an agent that has been sufficiently optimized for not pursuing dominated strategies but not for non-myopia. Doesn’t non-myopia dominate myopia in many reasonable setups?
That’s almost the counterargument that I’d give, but importantly not quite. The problem with the Caprice Rule is not that the agent needs to be non-myopic, but that the agent needs to know in advance which trades will be available. The agent can be non-myopic—i.e. have a model of future trades and optimize for future state—but still not know which trades it will actually have an opportunity to make. E.g. in the pizza example, when David and I are offered to trade mushroom for anchovy, we don’t yet know whether we’ll have an opportunity to trade anchovy for pepperoni later on.
More general point: I think relying on decision trees as our main model of the agents’ “environment” does not match the real world well, especially when using relatively small/simple trees. It seems to me that things like the Caprice rule are mostly exploiting ways in which decision trees are a poor model of realistic environments.
The assumption that we know in advance which trades will be available is one aspect of the problem, which could in-principle be handled by adding random choice nodes to the trees.
Another place where I suspect this is relevant (though I haven’t pinned it down yet): the argument in the post has a corner case when the probability of being offered some trade is zero. In that case, the agent will be indifferent between the completion and its original preferences, because the completion will just add a preference which will never actually be traded upon. I suspect that most of your examples are doing a similar thing—it’s telling that, in all your counterexamples, the agent is indifferent between original preferences and the completion; it doesn’t actively prefer the incomplete preferences. (Unless I’m missing something, in which case please correct me!) That makes me think that the small decision trees implicitly contain a lot of assumptions that various trades have zero probability of happening, which is load-bearing for your counterexamples. In a larger world, with a lot more opportunities to trade between various things, I’d expect that sort of issue to be much less relevant.
It’s easy to extend the Caprice Rule to this kind of case. Suppose we have an agent that’s uncertain whether – conditional on trading mushroom (A) for anchovy (B) – it will later have the chance to trade in anchovy (B) for pepperoni (A+). Suppose in its model the probabilities are 50-50.
Then our agent with a model of future trades can consider what it would choose conditional on finding itself in node 2: it can decide with what probability p it would choose A+, with the remaining probability 1-p going to B. Then, since choosing B at node 1 has a 0.5 probability of taking the agent to node 2 and a 0.5 probability of taking the agent to node 3, the agent can regard the choice of B at node 1 as the lottery 0.5p(A+)+(1-0.5p)(B) (since, conditional on choosing B at node 1, the agent will end up with A+ with probability 0.5p and end up with B otherwise).
So for an agent with a model of future trades, the choice at node 1 is a choice between A and 0.5p(A+)+(1-0.5p)(B). What we’ve specified about the agent’s preferences over the outcomes A, B, and A+ doesn’t pin down what its preferences will be between A and 0.5p(A+)+(1-0.5p)(B) but either way the Caprice-Rule-abiding agent will not pursue a dominated strategy. If it strictly prefers one of A and 0.5p(A+)+(1-0.5p)(B) to the other, it will reliably choose its preferred option. If it has no preference, neither choice will constitute a dominated strategy.
And this point generalises to arbitrarily complex/realistic decision trees, with more choice-nodes, more chance-nodes, and more options. Agents with a model of future trades can use their model to predict what they’d do conditional on reaching each possible choice-node, and then use those predictions to determine the nature of the options available to them at earlier choice-nodes. The agent’s model might be defective in various ways (e.g. by getting some probabilities wrong, or by failing to predict that some sequences of trades will be available) but that won’t spur the agent to change its preferences, because the dilemma from my previous comment recurs: if the agent is aware that some lottery is available, it won’t choose any dispreferred lottery; if the agent is unaware that some lottery is available and chooses a dispreferred lottery, the agent’s lack of awareness means it won’t be spurred by this fact to change its preferences. To get over this dilemma, you still need the ‘non-myopic optimiser deciding the preferences of a myopic agent’ setting, and my previous points apply: results from that setting don’t vindicate coherence arguments, and we humans as non-myopic optimisers could decide to create artificial agents with incomplete preferences.
I commented on the original post last year regarding the economics angle:
Based on this lit review and the Wikipedia page and ChatGPT [1], I’m 90% sure that “representative agent” in economics means the idea that the market’s aggregate preferences are similar to the typical individuals’ preferences, and the general question of whether a market has any complete preference ordering does not fall within the scope of the term.
[1] GPT4 says “The representative agent is assumed to behave in a way that represents the average or typical behavior of the group in the aggregate. In macroeconomics, for instance, a representative agent might be used to describe the behavior of all households in an economy.
This modeling approach is used to reduce the complexity of economic models, making them more tractable, but it has also received criticism.”
Interesting argument! I think it goes through—but only under certain ecological / environmental assumptions:
That decisions / trades between goods are reversible.
That there are multiple opportunities to make such trades / decisions in the environment.
But this isn’t always the case! Consider:
Both John and David prefer living over dying.
Hence, John would not trade (John Alive, David Dead) for (John Dead, David Alive), and vice versa for David.
This is already a case of weakly incomplete preferences which, while technically reducible to a complete order over “indifference sets”, doesn’t seem well described by a utility function! In particular, it seems really important to represent the fact that neither person would trade their life for the other’s life, even though both (John Alive, David Dead) and (John Dead, David Alive) lie in the same “indifference / incommensurability set”.
(I think it’s better to call it an “incommensurability set”—just because two elements in a lattice share a least upper bound, it doesn’t mean they are themselves comparable).
Now let’s try and make the preferences strongly incomplete:
John prefers living freely over imprisonment, and imprisonment to dying.
Even if David was dead, he would prefer that John be alive over John being imprisoned.
Apart from the fact that you can’t reverse death (at least with current technology), this is similar to the pizza scenario: The system as a whole prefers:
(John Free, David Alive) > (John Free, David Dead) > (John Imprisoned, David Dead) > Both Dead
(John Free, David Alive) > (John Imprisoned, David Alive) > (John Dead, David Alive) > Both Dead
No preferences between options of the form (X, David Dead) and (John Dead, Y).
If John and David could contract to go from (John Imprisoned, David Dead) to (John Dead, David Alive) and then to (John Alive, David Dead) when those trades are offered, that would result in an improvement in achieving preferred outcomes on average. But of course, they can’t because death is irreversible!
Rather than talking about reversibility, can this situation be described just by saying that the probability of certain opportunities is zero? For example, if John and David somehow know in advance that no one will ever offer them pepperoni in exchange for anchovies, then the maximum amount of probability mass that can be shifted from mushrooms to pepperoni by completing their preferences happens to be zero. This doesn’t need to be a physical law of anchovies; it could just be a characteristic of their trade partners.
But in this hypothetical, their preferences are effectively no longer strongly incomplete—or at least, their trade policy is no longer strongly incomplete. Since we’ve assumed away the edge between pepperoni and anchovies, we can (vacuously) claim that John and David will collectively accept 100% of the (non-existent) trades from anchovies to pepperoni, and it becomes possible to describe their trade policy as being a utility maximizer. (Specifically, we can say anchovies = mushrooms because they won’t trade between them, and say pepperoni > mushrooms because they will trade mushrooms for pepperoni. The original problem was that this implies that pepperoni > anchovies, which is false in their preferences, but it is now (vacuously) true in their trade policy if such opportunities have probability zero.)
It seems to me that it’s not right to assume the probability of opportunities to trade are zero?
Suppose both John and David are alive on a desert island right now (but slowly dying), and there’s a chance that a rescue boat will arrive that will save only one of them, leaving the other to die. What would they contract to? Assuming no altruistic preferences, presumably neither would agree to only the other person being rescued.
It seems more likely here that bargaining will break down, and one of them will kill off the other, resulting in an arbitrary resolution of who ends up on the rescue boat, not a “rational” resolution.
Doesn’t irreversibility imply that there is zero probability of a trade opportunity to reverse the thing? I’m not proposing a new trait that your original scenario didn’t have; I’m proposing that I identified which aspect of your scenario was load-bearing.
I don’t think I understand how your new hypothetical is meant to be related to anything discussed so far. As described, the group doesn’t have strongly incomplete preferences, just 2 mutually-exclusive objectives.
Zero probability of trade is indeed the feature which would make the argument in the OP potentially not go through, when irreversibility is present. (Though we would still get a weakened form of the argument from the OP, in which we complete the preferences by adding a preference for a trade which has zero probability, and the original system is indifferent between that completion and its original preferences.)
Well, it can be overcame by future contracts, no? We replace “Jonh dead” with “John dies tomorrow” and perform trades today.
While I’ve focused on death here, I think this is actually much more general—there are a lot of irreversible decisions that people make (and that artificial agents might make) between potentially incommensurable choices. Here’s a nice example from Elizabeth Anderson’s “Value in Ethics & Economics” (Ch. 3, P57 re: the question of how one should live one’s life, to which I think irreversibility applies
Similar incommensurability applies, I think, to what kind of society we collectively we want to live in, given that path dependency makes many choices irreversible.
As I understand this, the rough sketch of this approach is basically to realize that incomplete preferences are compatible with a family of utility functions rather than a single one (since they don’t specify how to trade-off between incomparable outcomes), and that you can use randomization to select within this family (implemented via contracts), thereby narrowing in on completed preferences / a utility function. Is that description on track?
If so, is it a problem that the subagents/committee/market may have preferences that are a function of this dealmaking process, like preferences about avoiding the coordination/transaction costs involved, or preferences about how to do randomization? Like, couldn’t you end up with a situation where “completing the preferences” is dispreferred, such that the individual subagents do not choose to aggregate into a single utility maximizer?
I think the idea of contracts is interesting. I’m probably less optimistic (or pessimistic?) than the authors that the sub-agents can always contract to complete their preferences. For one thing, contracts might be expensive to make. Second, even free contracts might not be incentivized, for the usual reasons rational agents cannot always avoid inefficiencies in trading (cf the Myerson-Satterthwaite theorem).
On might object that Myerson–Satterthwaite doesn’t allow for smart contracts that can conditionally disclose private information. But then I’d argue that these types of smart contracts are probably expensive to make, and thus not always incentivized.
How do agents with preferential gaps fit into this? I think preferential gaps are a kind of weak incompleteness, and thus handled by your second step?
Context: I’m pretty interested in the claims in this post, and their implications. A while ago, I went back and forth with EJT a bit on his coherence theorems post. The thread ended here with a claim by EJT:
I didn’t have a counterpoint at the time, but I am pretty skeptical that this claim is true, intuitively.
An agent with even infinitely many preferential gaps seems very close in mind-space to an agent with complete preferences: all it is missing is a relatively simple-to-describe function which “breaks the tie” on things it is already very close to indifferent about. And different choices of tiebreaker function seem unlikely to lead to importantly different behavior: for any choice of tiebreaker function, you are back to an EU maximizer.
The only remaining hope is to avoid having the agent ever pick or be imbued with a tiebreaker function at all. That requires at least two things:
The agent’s creators must not initialize it with such a tiebreaker function (seems unlikely to happen by default, but maybe if the creators are alignment researchers who know what they are doing, it’s possible)
The agent itself must be stable enough that it never chooses to self-modify or drift into completeness on its own. And I think your claim, if I’m understanding it correctly, is that such stability is unlikely, because completing the preferences can lead to a strict improvement in outcomes under the preferences of the original agent.
Am I understanding your claims correctly, and do you agree with my reasoning that EJT’s claim is thus unlikely to be true?
This is probably a dumb beginner question indicative of not understanding the definition of key terms, but to reveal my ignorance anyway—isn’t any company that consistently makes a profit successfully exploiting the market? And if it is, why do we say that markets are inexploitable, if they’re built on the existence of countless actors exploiting them?
Two answers here.
First, the standard economics answer: economic profit ≠ accounting profit. Economic profit is how much better a company does than their opportunity cost; accounting profit is revenue minus expenses. A trading firm packed with top-notch physicists, mathematicians, and programmers can make enormous accounting profit and yet still make zero economic profit, because the opportunity costs for such people are quite high. “Efficient markets” means zero economic profits, not zero accounting profits.
Second answer: as Zvi is fond of pointing out, the efficient market hypothesis is false (even after accounting for the distinction between economic and accounting profit). For instance, Renaissance—a real trading firm packed with top-notch physicists, mathematicians, and programmers—in fact makes far more money than the opportunity cost of its employees and capital. That said, market efficiency is still a very good approximation for a lot of purposes, and I’d be very curious to know whether selection pressures have already induced the trades which would make markets approximately aggregate into a utility maximizer.
Where is this contest entry? All my usual search methods are failing me...
I don’t think it’s been posted publicly yet. Elliot said I was welcome to cite it publicly, but didn’t explicitly say whether I should link it. @EJT ?
Hi Wei, happy to send it your way! I plan to post it publicly once I’ve had a chance to go back over it and improve the structure/writing/exposition.
John and David, great post! I’m going to write a reply this week.
Hi EJT, I’m starting research on incomplete preferences / subagents and would love to see this entry too if possible!
Yep, sent!
Not sure if this is the same as the awards contest entry, but EJT also made this earlier post (“There are no coherence theorems”) arguing that certain Dutch Book / money pump arguments against incompleteness fail!
So, what this might look like in practice (to check my understanding):
Entrepreneur approaches deadlocked bay area nimbys who are all vetoing each other’s stuff. Entrepreneur proposes a deal where each party i is supposed to agree to refrain from vetoing [initiative] with probability Xi, for various initiatives. The probabilities are cleverly chosen such that they all think it’s a good deal… supposing they all have similar credences, this should be possible, according to this theorem?
Epistemic Status: Really unsure about a lot of this.
It’s not clear to me that the randomization method here is sufficient for the condition of not missing out on sure gains with probability 1.
Scenario: B is preferred to A, but preference gap between A & C and B & C, as in the post.
Suppose both your subagents agree that the only trades that will ever be offered are A->C and C->B. These trades occur with a Poisson distribution, with λ = 1 for the first trade and λ = 3 for the second. Any trade that is offered must be immediately declined or accepted. If I understand your logic correctly, this would mean randomizing the preferences such that
pC→B = 1⁄3,
pA→C = 1
In the world where one of each trade is offered, the agent always accepts A->C but will only accept C->B 1⁄3 of the time, thus the whole move from A->B only happens with probability 1⁄3. So the agent misses out on sure gains with probability 2⁄3.
In other words, I think you’ve sufficiently shown that this kind of contract can taken a strongly-incomplete agent and make them not-strongly-incomplete with probability >0 but this is not the same as making them not-strongly-incomplete with probability 1, which seems to me to be necessary for expected utility maximization.
Yeah, the argument is not intended to be “here’s the optimal contract/modification to complete the preferences”. The argument is roughly:
If preferences are strongly incomplete, then there exists at least one contract/modification which is a pareto-improvement.
Therefore, non-dominated strategies must not be strongly incomplete.
So, the argument in the post allows the possibility that the preferences will be completed in some other way which does even better.
Really enjoyed this post, my question is how does this intersect with issues stemming from other VNM axioms e.g. Independence as referenced by Scott Garrabrant?
https://www.lesswrong.com/s/4hmf7rdfuXDJkxhfg/p/Xht9swezkGZLAxBrd
It seems to me that you don’t get expected utility maximizers solely from not-strong-Incompleteness, as there are other conditions that are necessary to support that conclusion.
I haven’t dug into independence violations much, and don’t know what their analogues would be in the context of non-VNM coherence arguments, so I don’t know.
Something I have a vague inkling about based on what you and Scott have written is that the same method by which we can rescue the Completeness axiom i.e. via contracts/commitments may also doom the Independence axiom. As in, you can have one of them (under certain premises) but not both?
This may follow rather trivially from the post I linked above so it may just come back to whether that post is ‘correct’, but it might also be a question of trying to marry/reconcile these two frameworks by some means. I’m hoping to do some research on this area in the next few weeks, let me know if you think it’s a dead end I guess!
Huh, sounds cool. Definitely worth investigating.
I don’t agree that focusing on extrinsic value is less myopic than focusing on intrinsic value. This world is full of false promises, self-delusion, rationalization of reckless commitment, complexity of value, bad incentives/cybernetics, and the fallaciousness of planning. My impression is that the conscientious sort of people who think so much about utility have overconfidence in the world’s structural friendliness and are way more screwed than the so-called “myopic” value-focused individuals.