A Selection Theorem tells us something about what agent type signatures will be selected for in some broad class of environments. Two important points:
The theorem need not directly talk about selection—e.g. it could state some general property of optima, of “broad” optima, of “most” optima, or of optima under a particular kind of selection pressure (like natural selection or financial profitability).
Any given theorem need not address every question about agent type signatures; it just needs to tell us something about agent type signatures.
For instance, the subagents argument says that, when our “agents” have internal state in a coherence-theorem-like setup, the “goals” will be pareto optimality over multiple utilities, rather than optimality of a single utility function. This says very little about embeddedness or world models or internal architecture; it addresses only one narrow aspect of agent type signatures. And, like the coherence theorems, it doesn’t directly talk about selection; it just says that any strategy which doesn’t fit the pareto-optimal form is strictly dominated by some other strategy (and therefore we’d expect that other strategy to be selected, all else equal).
From: Selection Theorems: A Program For Understanding Agents
I am quite enamoured with John Wentworth’s selection theorems, but find myself somewhat dissatisfied. As Wentworth framed it, I think they are a bit off.
I think selection theorems should be conceived of as theorems about artifacts (the products of constructive optimisation processes) and their constructors (the optimisation processes that created such artifacts).
That is, I am quite unconvinced that “agent” is the “true name” of such artifacts. There are powerful artifacts that do not match the agent archetype as traditionally conceived. I do not know that the artifacts that ultimately matter would necessarily conform to the agent archetype.
Agent selection theorems are IMO ultimately too restrictive, and the selection theorem agenda should be about optimisation processes and the kind of constructs they select for.