Rubi J. Hudson comments on Game Theory without Argmax [Part 1]

[Rubi J. Hudson]

I think this part uses an unfair comparison:

Supposes that $X=\{x_1,\dots ,x_n\}$ and $W^{+}=\{w_1,\dots ,w_m\}$ are small finite sets. A task $f:X\to W^{+}$ can be implemented as dictionary whose keys lie in $X$ and whose values lie in $W^{+}$, which uses $n\log m$ bits. The functional $\psi :J^P(X,W^{+})$ can be implemented as a program which receives input of type $Dict[X,W]$ and returns output of type $List\left[X\right]$. Easy!

In the subjective account, by contrast, the task $f:X\to R$ requires infinite bits to specify, and the functional $\psi :J^P(X,R)$ must somehow accept a representation of an arbitrary function $f:X\to R$. Oh no! This is especially troubling for embedded agency, where the agent’s decision theory must run on a physical substrate.

If X and W+ are small finite sets, then any behavior can be described with a utility function requiring only a finite number of bits to specify. You only need to use R as the domain when W+ is infinite, such as when outcomes are continuous, in which case the dictionaries require infinite bits to specify too.

I think this is representative of an unease I have with the framing of this sequence. It seems to be saying that the more general formulation allows for agents that behave in ways that utility maximizers cannot, but most of these behaviors exist for maximizers of certain utility functions. I’m still waiting for the punchline of what AI safety relevant aspect requires higher order game theory rather than just maximizing agents, particularly if you allow for informational constraints.