(This originated as a comment on the post “Embedded World-Models,” but it makes a broadly applicable point and is substantial enough to be a post, so I thought I’d make it a post as well.)
This post feels quite similar to things I have written in the past to justify my lack of enthusiasm about idealizations like AIXI and logically-omniscient Bayes. But I would go further: I think that grappling with embeddedness properly will inevitably make theories of this general type irrelevant or useless, so that “a theory like this, except for embedded agents” is not a thing that we can reasonably want. To specify what I mean, I’ll use this paragraph as a jumping-off point:
Embedded agents don’t have the luxury of stepping outside of the universe to think about how to think. What we would like would be a theory of rational belief for situated agents which provides foundations that are similarly as strong as the foundations Bayesianism provides for dualistic agents.
Most “theories of rational belief” I have encountered—including Bayesianism in the sense I think is meant here—are framed at the level of an evaluator outside the universe, and have essentially no content when we try to transfer them to individual embedded agents. This is because these theories tend to be derived in the following way:
We want a theory of the best possible behavior for agents.
We have some class of “practically achievable” strategies S, which can actually be implemented by agents. We note that an agent’s observations provide some information about the quality of different strategies s∈S. So if it were possible to follow a rule like R≡ “find the best s∈S given your observations, and then follow that s,” this rule would spit out very good agent behavior.
Usually we soften this to a performance-weighted average rather than a hard argmax, but the principle is the same: if we could search over all of S, the rule R that says “do the search and then follow what it says” can be competitive with the very best s∈S. (Trivially so, since it has access to the best strategies, along with all the others.)
But usually R∉S. That is, the strategy “search over all practical strategies and follow the best ones” is not a practical strategy. But we argue that this is fine, since we are constructing a theory of ideal behavior. It doesn’t have to be practically implementable.
For example, in Solomonoff, S is defined by computability while R is allowed to be uncomputable. In the LIA construction, S is defined by polytime complexity while R is allowed to run slower than polytime. In logically-omniscient Bayes, finite sets of hypotheses can be manipulated in a finite universe but the full Boolean algebra over hypotheses generally cannot (N.B. I don’t think this last case fits my schema quite as well as the other two).
I hope the framework I’ve just introduced helps clarify what I find unpromising about these theories. By construction, any agent you can actually design and run is a single element of S (a “practical strategy”), so every fact about rationality that can be incorporated into agent design gets “hidden inside” the individual s∈S, and the only things you can learn from the “ideal theory” R are things which can’t fit into a practical strategy.
For example, suppose (reasonably) that model averaging and complexity penalties are broadly good ideas that lead to good results. But all of the model averaging and complexity penalization that can be done computably happens inside some Turing machine or other, at the level “below” Solomonoff. Thus Solomonoff only tells you about the extra advantage you can get by doing these things uncomputably. Any kind of nice Bayesian average over Turing machines that can happen computably is (of course) just another Turing machine.
This also explains why I find it misleading to say that good practical strategies constitute “approximations to” an ideal theory of this type. Of course, since R just says to follow the best strategies in S, if you are following a very good strategy in S your behavior will tend to be close to that of R. But this cannot be attributed to any of the searching over S that R does, since you are not doing a search over S; you are executing a single member of S and ignoring the others. Any searching that can be done practically collapses down to a single practical strategy, and any that doesn’t is not practical.
Concretely, this talk of approximations is like saying that a very successful chess player “approximates” the rule “consult all possible chess players, then weight their moves by past performance.” Yes, the skilled player will play similarly to this rule, but they are not following it, not even approximately! They are only themselves, not any other player.
Any theory of ideal rationality that wants to be a guide for embedded agents will have to be constrained in the same ways the agents are. But theories of ideal rationality usually get all of their content by going to a level above the agents they judge. So this new theory would have to be a very different sort of thing.
To state all this more pithily: if we design the search space to contains everything feasible, then rationality-as-search has no feasible implications. If rationality-as-search is to have feasible implications, then the search space must be weak enough for there to be something feasible that is not a point in the search space.
When does rationality-as-search have nontrivial implications?
(This originated as a comment on the post “Embedded World-Models,” but it makes a broadly applicable point and is substantial enough to be a post, so I thought I’d make it a post as well.)
This post feels quite similar to things I have written in the past to justify my lack of enthusiasm about idealizations like AIXI and logically-omniscient Bayes. But I would go further: I think that grappling with embeddedness properly will inevitably make theories of this general type irrelevant or useless, so that “a theory like this, except for embedded agents” is not a thing that we can reasonably want. To specify what I mean, I’ll use this paragraph as a jumping-off point:
Most “theories of rational belief” I have encountered—including Bayesianism in the sense I think is meant here—are framed at the level of an evaluator outside the universe, and have essentially no content when we try to transfer them to individual embedded agents. This is because these theories tend to be derived in the following way:
We want a theory of the best possible behavior for agents.
We have some class of “practically achievable” strategies S, which can actually be implemented by agents. We note that an agent’s observations provide some information about the quality of different strategies s∈S. So if it were possible to follow a rule like R≡ “find the best s∈S given your observations, and then follow that s,” this rule would spit out very good agent behavior.
Usually we soften this to a performance-weighted average rather than a hard argmax, but the principle is the same: if we could search over all of S, the rule R that says “do the search and then follow what it says” can be competitive with the very best s∈S. (Trivially so, since it has access to the best strategies, along with all the others.)
But usually R∉S. That is, the strategy “search over all practical strategies and follow the best ones” is not a practical strategy. But we argue that this is fine, since we are constructing a theory of ideal behavior. It doesn’t have to be practically implementable.
For example, in Solomonoff, S is defined by computability while R is allowed to be uncomputable. In the LIA construction, S is defined by polytime complexity while R is allowed to run slower than polytime. In logically-omniscient Bayes, finite sets of hypotheses can be manipulated in a finite universe but the full Boolean algebra over hypotheses generally cannot (N.B. I don’t think this last case fits my schema quite as well as the other two).
I hope the framework I’ve just introduced helps clarify what I find unpromising about these theories. By construction, any agent you can actually design and run is a single element of S (a “practical strategy”), so every fact about rationality that can be incorporated into agent design gets “hidden inside” the individual s∈S, and the only things you can learn from the “ideal theory” R are things which can’t fit into a practical strategy.
For example, suppose (reasonably) that model averaging and complexity penalties are broadly good ideas that lead to good results. But all of the model averaging and complexity penalization that can be done computably happens inside some Turing machine or other, at the level “below” Solomonoff. Thus Solomonoff only tells you about the extra advantage you can get by doing these things uncomputably. Any kind of nice Bayesian average over Turing machines that can happen computably is (of course) just another Turing machine.
This also explains why I find it misleading to say that good practical strategies constitute “approximations to” an ideal theory of this type. Of course, since R just says to follow the best strategies in S, if you are following a very good strategy in S your behavior will tend to be close to that of R. But this cannot be attributed to any of the searching over S that R does, since you are not doing a search over S; you are executing a single member of S and ignoring the others. Any searching that can be done practically collapses down to a single practical strategy, and any that doesn’t is not practical.
Concretely, this talk of approximations is like saying that a very successful chess player “approximates” the rule “consult all possible chess players, then weight their moves by past performance.” Yes, the skilled player will play similarly to this rule, but they are not following it, not even approximately! They are only themselves, not any other player.
Any theory of ideal rationality that wants to be a guide for embedded agents will have to be constrained in the same ways the agents are. But theories of ideal rationality usually get all of their content by going to a level above the agents they judge. So this new theory would have to be a very different sort of thing.
To state all this more pithily: if we design the search space to contains everything feasible, then rationality-as-search has no feasible implications. If rationality-as-search is to have feasible implications, then the search space must be weak enough for there to be something feasible that is not a point in the search space.