Game Theory without Argmax [Part 2]

[Cleo Nardo]

Written during the SERI MATS program under the joint mentorship of John Wentworth, Nicholas Kees, and Janus.

Motivation

I am surrounded, incessantly, by other people, possessing a range of different capabilities, striving for goals both common and opposing — and, what is more, they are incessantly surrounded by me. Fangs pierce the cartesian boundary. Outsideness leaks in. Insideness leaks out. It is the source of all joy and misery in my life.

Anyway. It’s the interaction of $n\ge 2$ agents that characterises game theory.^[1] Classical game theory says that when two utility-maximisers interact in a shared environment, they will choose options in nash equilibrium, where a profile of options is in nash equilibrium if no player can increase their utility function by unilaterally changing their option.

What does higher-order game theory (which dispenses with utility functions and maximisation) say about the interaction between agents? Would a satisficer cooperate with a quantiliser in prisoner’s dilemma?

Let’s find out.

---

Preliminaries

All you need to know is Definition 3 from [Part 1], and some basic classical game theory.

Definition 3: Let $X$ be any set of options and $R$ be any set of payoffs. An optimiser is any functional $\psi :(X\to R)\to P\left(X\right)$. A $\psi$-task is any function $u:X\to R$. An option $x\in X$ is $\psi$-optimal for a task $u:X\to R$ if and only if $x\in \psi \left(u\right)$.

Let’s review the textbook definition of classical simultaneous games.

Definition 5.

1. Let $\{X_i{\}}_{i\in I}$ be a family of sets indexed by a set $I$ of players.
   
   A classical simultaneous game over $\{X_i{\}}_{i\in I}$ is a function $g:{\prod }_{i\in I}{X}_i\to R^I$.
   

2. The option-profiles of $g$ are the elements of the product $X=\prod _{i\in I}{X}_i$.
   

3. The $i$-th deviation map $U_i:X\to (X_i\to X)$ is given by $U_i\left(x\right)(\alpha {)}_j=\{\begin{array}{cc}\alpha & \text{if}i=j\\ x_j& \text{otherwise}\end{array}$ for all $x\in X$, $i\in I$, $\alpha \in X_i$, and $j\in I$.
   
   In the finite case, where $I=\{1,\dots ,n\}$, then $U_i(x_1,\dots ,x_n):X_i\to X,\alpha \mapsto (x_1,\dots ,x_{i-1},\alpha ,x_i,\dots ,x_n)$.
   
   The deviation map $U_i:X\to (X_i\to X)$ describes how the $i$th player can change an option-profile $x$ by unilaterally changing their own option. Note that the deviation maps only depend on the option spaces, not the game itself.
   

4. The best-response function of $g$ is the the function $B:X\to P\left(X\right)$ defined by$B:x\mapsto \prod _{i\in I}{\text{argmax}}_{X_i}({\pi }_i\circ g\circ U_i(x\left)\right)$ where $U_i:X\to (X_i\to X)$ is the $i$th deviation map.
   

5. Finally, the nash equilibria of $g$ is the set of option-profiles $x\in X$ such that $x\in B\left(x\right)$.

Note that the fact that the option-profiles are in nash equilibrium follows logically from the assumption each player’s choice is utility-maximising for the task they face, plus the assumption that, if the option-profile is $x\in X$, then the $i$th player faces the task ${\pi }_i\circ g\circ U_i\left(x\right):X_i\to R$.

It doesn’t require the assumption that the player’s have “common knowledge” of each other’s rationality, or that the players have any kinds of beliefs whatsoever. That being said, an economist might explain the optimality of the choices by assuming that each player is rational and knows all the relevant facts (i.e. the rules of the game, the rationality of their peers, and the beliefs of their peers). However, an evolutionary biologist might appeal to natural selection to explain optimality. An ML engineer might appeal to SGD, etc.

---

Higher-Order Nash Equilibrium

By inspection, we can see that at no point does the definition of the classical nash equilibrium appeal to any special properties of $R$ or ${\text{argmax}}_X$. This suggests that we can effortless generalise the definition of nash equilibrium to any family of optimisers, rather than only utility-maximisers.

Definition 6.

1. Let $\{X_i{\}}_{i\in I}$ be a family of sets indexed by a set $I$ of players. Let $R$ be a set of payoffs, and let $\Psi =\{{\psi }_i{\}}_{i\in I}$ be a family of optimisers with ${\psi }_i\in J^P(X_i,R)$.^[2]
   
   A higher-order simultaneous $\Psi$-game is a function $g:{\prod }_{i\in I}{X}_i\to R$.
   

2. Like before, the option-profiles of $g$ are the elements of the product $X=\prod _{i\in I}{X}_i$.
   

3. Like before, the $i$-th deviation map $U_i:X\to (X_i\to X)$ is given by $U_i\left(x\right)(\alpha {)}_j=\{\begin{array}{cc}\alpha & \text{if}i=j\\ x_j& \text{otherwise}\end{array}$ for all $x\in X$, $i\in I$, $\alpha \in X_i$, and $j\in I$.
   

4. The $\Psi$-best-response function of $g$ is the the function $B:X\to P\left(X\right)$ defined by $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$ where $U_i:X\to (X_i\to X)$ is the $i$th deviation map
   

5. Like before, the $\Psi$-nash equilibria of $g$ is the set of option-profiles $x\in X$ such that $x\in B\left(x\right)$.

When clear from context, I’ll just say game, best-response, and nash equilibrium.

The intuition is this —

• Suppose that $x=(x_1,\dots ,x_n)\in X_1\times \cdots \times X_n$ is the option-profile that’s collectively chosen.

• Then player $i\in I$ can achieve a payoff $[g\circ U_i(x\left)\right]\left(\alpha \right)\in R$ by unilaterally changing their option to $\alpha \in X_i$ because $[g\circ U_i(x\left)\right]\left(\alpha \right)=g(x_1,\dots ,x_{i-1},\alpha ,x_{i+1},\dots ,x_n)$.

• So player $i\in I$ is faced with the task $g\circ U_i\left(x\right):X_i\to R$.

• Their choice $x_i\in X_i$ must be ${\psi }_i$-optimal for this task, so $x_i\in {\varphi }_i(g\circ U_i(x\left)\right)\subseteq X_i$.

• If we assume ${\psi }_i$-optimality for every player conjunctively, then $x\in B\left(x\right)$.

• If we know that $x\in B\left(x\right)$, and we know nothing else, then the possible option profiles are $\{x\in X:x\in B(x\left)\right\}\in P\left(X\right)$.

This definition applies even to unusual cases —

• Zero-player games? When $|I|=0$, then game $g$ is just an element of the payoff space $R$. There’s only one option-profile, the empty sequence, and it’s vacuously nash.

• One-player games? When $|I|=1$, then a game $g:X\to R$ is just a task for the solitary player. An option-profile for the game is specified by an option for the player, and this option-profile is nash whenever the option is optimal for the player.

• When $I$ is countably infinite, or indeed uncountably infinite, then the definition still makes sense and gives the right answer.^[3]

We obtain the classical simultaneous game when $R=R^n$ and ${\psi }_i={\text{argmax}}_X({\pi }_i\circ -)\in J^P(X,R^n)$, and in this case the higher-order nash equilibrium coincides with the classical nash equilibrium.

The higher-order nash equilibrium allows us to answer somewhat esoteric questions —

An argmaxer, a satisficer, and a quantiliser walk into a bar. They can order from menus $A$, $S$, and $Q$ respectively, and the barman will mix their three orders $(a,s,q)\in A\times S\times Q$ into a single cocktail $f(a,s,q)\in C$. The argmaxer’s preferences over the cocktails are given by $v_a:C\to R$, and likewise $v_s:C\to R$ for the satisfier and $v_q:C\to R$ for the quantiliser.

What cocktail might they be served?

Well, the possible cocktails are $f(a,s,q)\in C$ such that $a\in {\text{argmax}}_A({\lambda }^{a^{\prime }\in A}.v_a\circ f(a^{\prime },s,q\left)\right)$, $s\in {\text{satisfice}}_{s_0}({\lambda }^{s^{\prime }\in S}.v_s\circ f(a,s^{\prime },q\left)\right)$, and $q\in {\text{quant}}_{ϵ}({\lambda }^{q^{\prime }\in Q}.v_q\circ f(a,s,q^{\prime }\left)\right)$.

Many well-studied variations of the classical nash equilibrium can be seen as the higher-order nash equilibrium between non-classical optimisers. For example, epsilon-equilibrium is precisely the higher-order nash equilibrium between ${\text{argmax-}ϵ\text{-slack}}_X({\pi }_i\circ -)$.^[4]

---

Optimisation is nash-sum closed

There’s a nice upshot of higher-order game theory — the nash equilibria between optimisers is itself an optimiser!

Suppose that two agents are modelled by ${\psi }_A\in J^P(A,R)$ and ${\psi }_B\in J^P(A,R)$. We can combine them into a single function which assigns, to each game $g:A\times B\to R$, the set of $\{{\psi }_A,{\psi }_B\}$-nash equilibria of the game $g$. This function has type-signature $(A\times B\to R)\to P(A\times B)$, so it corresponds to a single optimiser with option space $(A\times B)$ with payoff space $R$.

This gives us a binary operator on optimisers,$\oplus :J^P(A,R)\times J^P(B,R)\to J^P(A\times B,R)$, given by $(a,b)\in ({\psi }_A\oplus {\psi }_B)\left(g\right)⟺a\in {\psi }_A(g\circ U_1(a,b\left)\right),b\in {\psi }_B(g\circ U_2(a,b\left)\right)$.

This operator is both associative and commutative, justifying the name “nash sum”.

For example, the nash sum $({\text{argmax}}_A\oplus {\text{argmax}}_B):(A\times B\to R\times R)\to P(A\times B)$ will calculate the classical nash equilibria of a generic payoff matrix $g:A\times B\to R^n$.

The operator can be extended to any family of optimisers, i.e.$\underset{i\in I}{⨁}:\prod _{i\in I}{J}^P(X_i,R)\to J^P(\prod _{i\in I}{X}_i,R)$ where $x\in \left(\underset{i\in I}{⨁}{\psi }_i\right)\left(g\right)⟺x\in \prod _{i=1}^n{\psi }_i(g\circ U_i(x\left)\right)$.

Terminology:

• ${⨁}_{i\in I}{\psi }_i$ is a single optimiser, while $\{{\psi }_i{\}}_{i\in I}$ is a family of optimisers.

• ${⨁}_{i\in I}{\psi }_i$ is the nash-sum of $\{{\psi }_i{\}}_{i\in I}$, and $\{{\psi }_i{\}}_{i\in I}$ are subagents of ${⨁}_{i\in I}{\psi }_i$.

• An option for ${⨁}_{i\in I}{\psi }_i$ is an option-profile for $\{{\psi }_i{\}}_{i\in I}$.

• A task for the optimiser ${⨁}_{i\in I}{\psi }_i$ is a game for the family of optimisers $\{{\psi }_i{\}}_{i\in I}$.

• An option $x\in {\prod }_{i\in I}{X}_i$ is optimal for ${⨁}_{i\in I}{\psi }_i$ in the task $g:{\prod }_{i\in I}{X}_i\to R$ if and only if the option-profile $x\in {\prod }_{i\in I}{X}_i$ is a nash equilibrium for $\{{\psi }_i{\}}_{i\in I}$ in the game $g:{\prod }_{i\in I}{X}_i\to R$.

---

Totality is not nash-sum closed

Just because two optimisers ${\psi }_a\in J^P(A,R)$ and ${\psi }_b\in J^P(B,R)$ are total does not imply that their nash sum $({\psi }_a\oplus {\psi }_b)\in J^P(A\times B,R)$ is total.^[5] In other words, we may have a model of an agent which works perfectly in all solitary situations, but breaks down when many such agents interact in a shared environment.

The textbook example is Matching Pennies. Each player must choose either heads or tails — player 1 wins if the coins match and player 2 wins if they don’t. If both players are utility-maximisers then there’s no nash equilibria, even though utility-maximisers are total optimisers.

$u$	H	T
H	$(1,0)$	$(0,1)$
T	$(0,1)$	$(1,0)$

Formally speaking, $A=\{H,T\}$, $B=\{H,T\}$, and $u:A\times B\to R\times R,(x,y)\mapsto ({\delta }_{xy},1-{\delta }_{xy})$.^[6] Then $\left({\text{argmax}}_A\right({\pi }_1\circ -)\oplus {\text{argmax}}_B({\pi }_2\circ -\left)\right)\left(u\right)=\varnothing$, although ${\text{argmax}}_A({\pi }_1\circ -)\left(w\right)\ne \varnothing$ and ${\text{argmax}}_B({\pi }_1\circ -)\left(w\right)\ne \varnothing$ for all $w:\{H,T\}\to R\times R$.

This shows why we couldn’t have defined optimisers as functions with type-signature $(X\to R)\to P^{\ast }\left(X\right)$ where $P^{\ast }\left(X\right)$ denotes the set of non-empty subsets of $X$. In order to ensure nash-sum closure of optimisers, we needed to include non-total optimisers.

---

Utility-maximisation is not nash-sum closed.

It is well-known that the nash sum of $n$ utility-maximisers isn’t a utility-maximiser for any $n\ge 2$. For many games, none of the nash equilibrium are even pareto-optimal!

The textbook case is the prisoner’s dilemma. Let $B=\{C,D\}$ and consider two players${\text{argmax}}_B({\pi }_1\circ -)$ and ${\text{argmax}}_B({\pi }_2\circ -)$ which respectively value the first and second coordinate of the payoff. Their nash sum is the optimiser in $J^P(B^2,R^2)$ which calculates the classical nash equilibria of 2-by-2 payoff matrices $u:B^2\to R^2$. In particular, when $u:B^2\to R^2$ is the payoff matrix defined below, then $\left({\text{argmax}}_B\right({\pi }_1\circ -)\oplus {\text{argmax}}_B({\pi }_2\circ -\left)\right)\left(u\right)=\left\{\right(D,D\left)\right\}$. However, both players prefer the payoff $u(C,C)=(2,2)$ to $u(D,D)=(1,1)$.

$u$	$C$	$D$
$C$	$(2,2)$	$(0,3)$
$D$	$(3,0)$	$(1,1)$

---

Consequentialism is not nash-sum closed.

In fact, the nash sum of two utility-maximisers isn’t even consequentialist!^[7]

Suppose Angelica is babysitting her cousin Tommy, and each child has been given a cookie by their grandmother. If Angelica is awake while Tommy sleeps, then she can steal both cookies for herself. If Angelica sleeps while Tommy is awake then he’ll cause mayhem getting both children in trouble. If Angelica and Tommy both stay sleep, or if they both wake up, then they’ll each keep one cookie.

The payoffs $u:\{W_A,S_A\}\times \{W_T,S_T\}\to R\times R$ are summarised in the matrix below.

$u$	$W_T$	$S_T$
$W_A$	$(1,1)$	$(2,0)$
$S_A$	$(-10,-10)$	$(1,1)$

When Angelica and Tommy are both awake, this is nash — she doesn’t want to sleep because then he’ll cause mayhem, and he doesn’t want to sleep because then she’ll steal his cookie. In contrast, when they’re both sleeping, this isn’t nash — she can wake up and steal his cookie. But the same outcome will result whether Angelica and Tommy are both sleeping or both awake. So the nash sum of Angelica and Tommy isn’t a consequentialist optimiser!^[8]

This shows why we couldn’t have defined optimisers as functions with type-signature $(X\to R)\to P\left(R\right)$, which bakes-in the assumption that all optimisers are consequentialist. In order to ensure the nash-sum closure of optimisers, we needed to include nonconsequentialist optimisers.^[9]

---

Beyond CDT?

Warning: This section is a speculative extension of the existing literature, it’s merely a proof-of-concept of how higher-order game theory would extend to non-CDT agents.

Let $u:\{C,D\}\times \{C,D\}\to R\times R$ be the game defined by the payoff matrix below. We may readily check that $({\text{argmax}}_{\{C,D\}}\oplus {\text{argmax}}_{\{C,D\}})\left(u\right)=\left\{\right(1,1\left)\right\}$, which corresponds to the well-known result that two utility-maximisers will defect in the prisoners’ dilemma.

$u$	$C$	$D$
$C$	$(2,2)$	$(0,3)$
$D$	$(3,0)$	$(1,1)$

Now, some decision theorists think that it’s not generally true that two rational utility-maximisers will defect in the prisoners’ dilemma — e.g. if they’re both perfect replicas of one another then they’ll both cooperate! The reasoning is this — cooperation from the first prisoner would count as evidence (to the first player) of high utility, and defection from the first prisoner would count as evidence of low utility. This corresponds to the observation that ${\pi }_1\circ u(C,C)>{\pi }_1\circ u(D,D)$. So the first player should cooperate. Ditto the second player.

So what went wrong?

It all comes down to an ambiguity with the task $u:X\to R$. What relationship is the task $u$ supposed to capture exactly? The formalism itself is agnostic.

• According to causal decision theory, the task $u:X\to R$ is supposed to track which payoff $u\left(x\right)\in R$ is caused by the agent choosing $x\in X$.

• According to evidential decision theory, the task $u:X\to R$ is supposed to track which payoff $u\left(x\right)\in R$ is evidenced by the agent choosing $x\in X$.

So let’s explain our answer $({\text{argmax}}_{\{C,D\}}\oplus {\text{argmax}}_{\{C,D\}})\left(u\right)=\left\{\right(1,1\left)\right\}$, which agrees with CDT. We can trace the problem to the deviation maps $U_i:X\to (X_i\to X)$ in the definition of $\oplus$.

The $i$-th deviation map $U_i:X\to (X_i\to X)$ is given by $U_i\left(x\right)(\alpha {)}_j=\{\begin{array}{cc}\alpha & \text{if}i=j\\ x_j& \text{otherwise}\end{array}$ for all $x\in X$, $i\in I$, $\alpha \in X_i$, and $j\in I$.

The map $U_i\left(x\right)$ is the causal dependence of the overall option-profile on the $i$th player’s choice. Hence, when the game $u:X\to R$ is the causal dependence of the overall payoff on the option-profile, it follows that the task $u\circ U_i\left(x\right):X_i\to R$ is the causal dependence of the payoff on the $i$th player’s choice. This is why we got the CDT answer — causation composed with causation is causation.

According to EDT, however, the task is supposed to capture the evidential dependency of the player’s choice on the payoff. And when the game $u:X\to R$ is the evidential dependence of the overall payoff on the option-profile, it isn’t true that that the task $u\circ U_i\left(x\right):X_i\to R$ is the evidential dependence of the payoff on the $i$-th player’s choice. Evidence composed with causation isn’t evidence.

To achieve the EDT answer, we need is a function $U_i^E\left(x\right):X_i\to X$ which captures the evidential dependence of the option-profile on the $i$th player’s choice. In the case that all the players are replicas, $U_i^E\left(x\right):X_i\to X,\alpha \mapsto (\alpha ,\dots ,\alpha )$. Replacing $U_i:X\to (X_i\to X)$ in the definition of $⨁$ with $U_i^E:X\to (X_i\to X)$ then we’d get a different binary operator ${⨁}^E$ which yields the EDT prediction, namely that $\left({\text{argmax}}_A{\oplus }^E{\text{argmax}}_B\right)\left(u\right)=\left\{\right(C,C\left)\right\}$.

It seems that we have another free parameter in our model! We can the replace the family of deviation maps $U_i:X\to (X_i\to X)$ with an arbitrary family of non-casual deviation maps $D_i:X\to (X_i\to X)$, and thereby encode information about each player’s idiosyncratic decision theory. If the $i$th player is CDT then $D_i=U_i$ and if the $i$th player is EDT then $D_i=U_i^E$.

Suppose we have a payoff space $R$, a family of sets $\{X_i{\}}_{i\in I}$ with $X={\prod }_{i\in I}{X}_i$, a family of optimisers $\{{\psi }_i{\}}_{i\in I}$ where ${\psi }_i\in J^P(X_i,R)$, and a family of decision theories $\{D_i{\}}_{i\in I}$ where $D_i:X\to (X_i\to X)$, and a game is a map $g:X\to R$. Then we can define the non-CDT best-response function of $g$ to be the function $B:X\to P\left(X\right),x\mapsto \prod _{i=1}^n{\psi }_i(u\circ D_i(x\left)\right)$, and the non-CDT nash equilibria of $g$ to be the set $\{x\in X:x\in B(x\left)\right\}$. Or something like that.

With this machinery in place, we can answer even sillier questions —

An EDT utility-satisficer and a CDT utility-maximiser are playing the game $u:X\times X\to R\times R$. They are causally separated, but they are negligibly likely to choose different options. What options might they choose?^[10]

Well, a pair of options $(a,b)\in X\times X$ is in (non-CDT) nash equilibrium if and only if ${\pi }_1\circ u(s,s)\le {\pi }_1\circ u(a,a)$ and $\forall b^{\prime }\in B.{\pi }_2\circ u(a,b^{\prime })\le {\pi }_2\circ u(a,b)$, where $s\in X$ is the satisficer’s anchor point. We can also filter out the option profiles such that $a\ne b$. The resulting list of option-profiles is easily computable from $u:X\times X\to R\times R$ and $s\in S$.

Suppose that $u:\{C,D\}\times \{C,D\}\to R\times R$ is the prisoner’s dilemma, and $s=C$ is the satisificer’s anchor point. Then the unique nash equilibrium is $(C,D)$.

$u$	$C$	$D$
$C$	$(2,2)$	$(0,3)$
$D$	$(3,0)$	$(1,1)$

Exercise 7: If Angelica is an EDT utility-maximiser and Tommy is a CDT utility-maximiser, then which profiles are nash?

---

Recap

• Classical game theory is the study agents which choose options causing maximum utility. When many such agents (CDT-utility-maximisers) make simultaneous choices they’ll collectively choose an option-profile in nash equilibrium, i.e.$x\in B\left(x\right)$ where $B:X\to P\left(X\right),x\mapsto \prod _{i=1}^n{\text{argmax}}_{X_i}({\pi }_i\circ u\circ U_i(x\left)\right)$ is the best-response function.

• Many AI safety researchers are worried about agents which choose options causing maximum utility, either because the word “cause” or “maximum” or “utility”, so they study agents which relax one (or many) of these three assumptions.

• Today we introduced high-order game theory which generalises classical game theory by considering agents which don’t maximise utility. We generalised the classical nash equilibrium to arbitrarily many agents with arbitrary optimisers.

• Finally, I gestured at how we might generalise the nash equilibrium even further to the simultaneous choices of non-CDT agents.

Pretty cool!

---

Next time...

By tomorrow, I will hopefully have posted about mesa-optimisation, sequential choice, and non-possibilistic models of agents.

---

1. ^

   They study of $0$-player games is called automata theory (in the discrete case) and dynamics (in the continuous case), while the study of $1$-player games is called optimisation.

2. ^

   Recall that $J^P(X,R)$ denotes the set of functions of type $(X\to R)\to P\left(X\right)$.

   That is, if $u:X\to R$ and $\psi \in J^P(X,R)$, then $\psi \left(u\right)\subseteq X$.

3. ^

   See Aumann’s 1964 paper Markets with a Continuum of Trader for a game with uncountable players.

4. ^

   Recall that $\text{argmax-}ϵ\text{-slack}\in J^P(X,R)$ is the optimiser which maximises the function $u$ up to some fixed slack $ϵ>0$.

   That is, $x\in \text{argmax-}ϵ\text{-slack}\left(u\right)⟺\forall x^{\prime }\in X.u\left(x^{\prime }\right)\le u\left(x\right)+ϵ$.

5. ^

   Recall that an optimiser $\psi \in J^P(X,R)$ is total if $\psi \left(u\right)\ne \varnothing$ for all $u:X\to R$.

6. ^

   ${\delta }_{xy}=\{\begin{array}{cc}1& \text{if}x=y\\ 0& \text{otherwise}\end{array}$ is the Kronecker delta function.

7. ^

   Recall that an optimiser $\psi \in J^P(X,R)$ is consequentialist if $\psi ={\lambda }^{u:X\to R}.\{x\in X|u(x)\in q(u\left)\right\}$ for some $q:(X\to R)\to P\left(R\right)$.

   In other words, for any task $u:X\to R$, if $u\left(x\right)=u\left(x^{\prime }\right)$ then $x\in \psi \left(u\right)⟺x^{\prime }\in \psi \left(u\right)$.

   This condition says that, once we know the agent’s task, then the optimality of a particular choice is determined by its payoff.

8. ^

   Maybe there’s something deep here about group-level deontology arising from individual-level consequentialism.

9. ^

   Warning: There is a remark that one regularly encounters in metaethics: moral consequentialism is tautological because we can consider the choice of the decision-maker as a consequence of the decision itself, ensuring that any decision-rule is consequentialist.

   This metaethical remark corresponds to the following fact in higher-order decision theory: if a nonconsequentialist optimiser $\psi \in J^P(X,R)$ faces the task $u:X\to R$ then they will choose the same options as the consequentialist optimiser $\psi ({\pi }_R\circ -)\in J^P(X,X\times R)$ facing the task $u^{\prime }:X\to X\times R,x\mapsto (x,u(x\left)\right)$. It follows that whether an agent is a consequentialist heavily depends on which physical consequences of the decision we are tracking.

10. ^

    Recall that a satisficer might choose any option $x\in X$ which dominates some fixed option $s\in X$, i.e. $x\in {\text{satisfice}}_s\left(u\right)⟺u\left(s\right)\le u\left(x\right)$. The option $s$ is called the anchor point.

[Daniel Kokotajlo]

e upshot of higher-order game theory — the nash equilibria between optimisers is itself an optimiser!

Isn’t this pretty trivial though? I guess it’s still probably convenient for the math.

[Cleo Nardo]

The observation is trivial mathematically, but it motivates the characterisation of an optimiser as something with the type-signature$(X\to R)\to P\left(X\right)$.

You might instead be motivated to characterise optimisers by...

• A utility function $u:X\to R$

• A quantifier $(X\to R)\to P\left(X\right)$

• A preorder $(R,\le )$ over the outcomes

• Etc.

However, were you to characterise optimisers in any of the ways above, then the nash equilibrium between optimisers would not itself be an optimiser, and therefore we lose compositionality. The compositionality is conceptually helpfully because it means that your $n\ge 2$ definitions/​theorems reduce to the $n=1$ case.

[rotatingpaguro]

• A quantifier $(X\to R)\to P\left(X\right)$

Here you mean $(X\to R)\to P\left(X\right)$, right?

[Cleo Nardo]

Wait I mean a quantifier in $(X\to R)\to P\left(R\right)$.

If we characterise an agent with a quantifier $(X\to R)\to P\left(R\right)$, then we’re saying which payoffs the agent might achieve given each task. Namely, $r\in q\left(u\right)$ if and only if it’s possible that the agent achieves payoff $r\in R$ when faced with a task $u:X\to R$.

But this definition doesn’t play well with a nash equilibria.

[momom2]

Thank you, this is incredibly interesting! Did you ever write up more on the subject? I’m excited to see how it relates to mesa-optimisation in particular.

In the finite case, where $I=\{1,\dots ,n\}$, then $U_i(x_1,\dots ,x_n):X_i\to X,\alpha \mapsto (x_1,\dots ,x_{i-1},\alpha ,x_i,\dots ,x_n)$

Typo: I think you mean $\alpha \mapsto (x_1,\dots ,x_{i-1},\alpha ,x_{i+1},\dots ,x_n)$?

[rotatingpaguro]

I would have liked a footnote saying ${\pi }_i\left(x\right)=x_i$.

---

I’m confused about the last example, EDT-satisficer vs. CDT-maximizer. It says that we assume the agents to choose the same options. Accordingly, it says to “filter out the option profiles such that $a\ne b$”. But then, in the concrete case of the Prisoner’s dilemma, it says the solution is $(C,D)$ instead of $\varnothing$.

---

Exercise 7:

In the Prisoner’s dilemma, the answer doesn’t change compared to the one in the worked-out example, because in that example the satisficer was already anchoring to the best EDT solution $(C,C)$. So the Nash equilibrium is $(C,D)$. More in general, the first agent condition is replaced by $\forall a^{\prime }\in X.u_1(a^{\prime },a^{\prime })\le u_1(a,a)$.

---

4. The $\Psi$-best-response function of $g$ is the the function $B:X\to P\left(X\right)$ defined by $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$ where $U_i:X\to (X_i\to X)$ is the $i$th deviation map

Here $\prod _{i\in I}{A}_i=\left\{\right(a_i{)}_{i\in I}\mid \forall i.a_i\in A_i\}$, right?

[Cleo Nardo]

Yes, $\prod _{i\in I}{A}_i=\left\{\right(a_i{)}_{i\in I}.\forall i\in I.a_i\in A_i\}$, i.e. the cartesian product of a family of sets. Sorry if this wasn’t clear, it’s standard maths notation. I don’t know what the other commenter is saying.

[StrivingForLegibility]

Got it, I misunderstood the semantics of what $B\left(x\right)$ was supposed to capture. I thought the elements needed to be mutual best-responses. Thank you for the clarification, I’ve updated my implementation accordingly!

[rotatingpaguro]

I interpreted it the standard way too initially, but then I had a hunch there was… I dunno, something fishy, and then indeed it turned out @StrivingForLegibility understood it in a completely different way, so somehow it wasn’t clear! Magic.

[StrivingForLegibility]

Edit: Cleo Nardo has confirmed that they intended $\prod _{i\in I}$ to mean the cartesian product of sets, the ordinary thing for that symbol to mean in that context. I misunderstood the semantics of what $B\left(x\right)$ was intended to represent. I’ve updated my implementation to use the intended cartesian product when calculating the best response function, the rest of this comment is my initial (wrong) interpretation of $B\left(x\right)$.

I needed to go back to one of the papers cited in Part 1 to understand what that $\prod _{i\in I}$ was doing in that expression. I found the answer in A Generalization of Nash’s Theorem with Higher-Order Functionals. I’m going to do my best to paraphrase Hedges’ notation into Cleo’s notation, to avoid confusion.

TLDR: $B\left(x\right)$ is picking out the set of option-profiles $P\left(X\right)$ that are simultaneously best-responses by all players to that option-profile $x$. It does this by considering all of the option-profiles that can result by each player best-responding, then takes the intersection of those sets.

On page 6, Hedges defines the best response correspondence $B\in X\to P\left(X\right)$

$B\left(x\right)=\bigcap _{i\in I}{B}_i\left(x\right)$

Where

$B_i\in X\to P\left(X\right)$

Hedges builds up the idea of Nash Equilibria using quantifiers rather than optimizers, (like $\text{max}$ rather than $\text{argmax}$), but I believe the approaches are equivalent. Unpacking $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$ from the inside out:

$U_i\left(x\right)\in X_i\to X$

$g\circ U_i\left(x\right)\in X_i\to R$

That makes $g\circ U_i(x$) a ${\psi }_i$-task. Since ${\psi }_i\in (X_i\to R)\to P\left(X_i\right)$, we know that ${\psi }_i(g\circ U_i(x\left)\right)\in P\left(X_i\right)$.

This is where I had to go looking through papers. What sort of product takes a set of best-responses from each player, relative to a given option-profile, and returns a set of option-profiles that are simultaneously regarded by each player as a best-response? I thought about just taking the Cartesian product of the sets, but that wouldn’t get us only the mutual best-responses.

Let’s call the way that each player maps option-profiles to best-responses $b_i\in X\to P\left(X_i\right)$. This is exactly the sets we want to take the product of:

$b_i\left(x\right)={\psi }_i(g\circ U_i(x\left)\right)$

Hedges introduces notation on page 3 to handle the operation of taking an option-profile, varying one player’s option, and leaving the rest the same. Paraphrasing, Hedges defines $x(i\mapsto \alpha )\in \prod _{j\in I}{X}_j$ by

$x(i\mapsto \alpha {)}_j=\{\begin{array}{cc}\alpha & \text{if}i=j\\ x_j& otherwise\end{array}$

You can read $x(i\mapsto \alpha )$ as “give me a new copy of $x$, where the $i$th entry has been set to the value $\alpha$.” Hedges uses this to define the deviation maps equivalently to the way Cleo did. $U_i:X\to (X_i\to X)$

$U_i\left(x\right)\left(\alpha \right)=x(i\mapsto \alpha )$

The correspondences $B_i\in X\to P\left(X\right)$ take as input an option profile, and returns the set of option-profiles which are player $i$’s optimal unilateral deviations from that option profile. To construct $B_i$ from $b_i$, we want to map $b_i\left(x\right)\in P\left(X_i\right)$ to the option-profiles which deviate from $x$ in those exact ways.

$B_i\left(x\right)=\left\{x\right(i\mapsto \alpha ):\alpha \in b_i(x\left)\right\}$

We can then use Hedges’ $B\left(x\right)=\bigcap _{i\in I}{B}_i\left(x\right)$ to get the best-response correspondence! We can unpack this to get a definition of $B$ using objects that Cleo defined, using that deviation notation from Hedges:

$B\left(x\right)=\bigcap _{i\in I}\left\{x\right(i\mapsto \alpha ):\alpha \in {\psi }_i(g\circ U_i(x\left)\right)\}$

Thank you Cleo for writing this article! This was my first introduction to Higher-Order Game Theory, and I wrote up an implementation in TypeScript to help me understand how all of the pieces fit together!

[rotatingpaguro]

I’m weirded out by this. To look at everything together, I write the original expression, and your expression rewritten using the OP’s notation:

Original: $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$

Yours: $\begin{array}{cc}B\left(x\right)& =\bigcap _{i\in I}\left\{x\right(i\mapsto \alpha ):\alpha \in {\psi }_i(g\circ U_i(x\left)\right)\}\\ & =\bigcap _{i\in I}{U}_i\left(x\right)\left({\psi }_i\right(g\circ U_i\left(x\right)\left)\right)\end{array}$

(I’m using the notation that a function applied to a set is the image of that set.)

So the big pi symbol stands for

${\prod }_{i\in I}{A}_i={\bigcap }_{i\in I}{U}_i\left(x\right)\left(A_i\right)$

So it’s not a standalone operator: it’s context-dependent because it pops out an implicit $x$. The OP otherwise gives the impression of a more functional mindset, so I suspect the OP may mean something different from your guess.

Other problem with your interpretation: it yields the empty set unless all agents consider doing nothing an option. The only possible non-empty output is $\left\{x\right\}$. Reason: each set you are intersecting contains tuples with all elements equal to the ones in $x$, but for one. So the intersection will necessarily only contain tuples with all elements equal to those in $x$.

[StrivingForLegibility]

Edit: Cleo Nardo has confirmed that they intended $\prod _{i\in I}$ to mean the cartesian product of sets, the ordinary thing for that symbol to mean in that context. I misunderstood the semantics of what $B\left(x\right)$ was intended to represent. I’ve updated my implementation to use the intended cartesian product when calculating the best response function, the rest of this comment is based on my initial (wrong) interpretation of $B\left(x\right)$.

I write the original expression, and your expression rewritten using the OP’s notation:

Original: $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$

Yours: $\begin{array}{cc}B\left(x\right)& =\bigcap _{i\in I}\left\{x\right(i\mapsto \alpha ):\alpha \in {\psi }_i(g\circ U_i(x\left)\right)\}\\ & =\bigcap _{i\in I}{U}_i\left(x\right)\left({\psi }_i\right(g\circ U_i\left(x\right)\left)\right)\end{array}$

(I’m using the notation that a function applied to a set is the image of that set.)

This is a totally clear and valid rewriting using that notation! My background is in programming and I spent a couple minutes trying to figure out how mathematicians write “apply this function to this set.”

I believe the way that $B\left(x\right)$ is being used is to find Nash equilibria, using Cleo’s definition 6.5:

Like before, the $\Psi$-nash equilibria of $g$ is the set of option-profiles $x\in X$ such that $x\in B\left(x\right)$.

These are going to be option-profiles where “not deviating” is considered optimal by every player simultaneously. I agree with your conclusion that this leads $B\left(x\right)$ to take on values that are either $\left\{\right\}$ or $\left\{x\right\}$. When $B\left(x\right)=\left\{\right\}$, this indicates that $x$ is not a Nash equilibrium. When $B\left(x\right)=\left\{x\right\}$, we know that $x$ is a Nash equilibrium.

[rotatingpaguro]

Oh I see now, $B$ just needs to work to pinpoint Nash equilibria, I did not make that connection.

But anyway, the reason I’m suspicious of your interpretation is not that your math is not correct, but that it makes the OP notation so unnatural. The unnatural things are:

1. $\prod$ being context-dependent.

2. $\prod$ not having its standard meaning.

3. $U_i$ used implicitly instead of explicitly, when later it takes on a more important role to change decision theory.

4. Using $x\in B\left(x\right)$ as condition without mentioning that already $B\left(x\right)\ne \varnothing ⟺x\text{is Nash}$ if $|I|\ge 2$.

So I guess I will stay in doubt until the OP confirms “yep I meant that”.

[Cleo Nardo]

$B\left(x\right)\ne \varnothing$ isn’t equivalent to $x$ being Nash.

Suppose Alice and Bob are playing prisoner’s dilemma. Then the best-response function of every option-profile is nonempty. But only one option-profile is nash.

$x\in B\left(x\right)$ is equivalent to $x$ being Nash.