I think this article is too vague, because for almost almost claims in it I am not sure if I understand the author correctly. Below I am posting my notes. If you want to help me and others clarify understanding of this article, consider answering questions in bold, or, if you see a mistake in my notes, correcting it. Also I hope my notes help the author as a piece of feedback. I’ve only finished 2⁄3 of the article so far, but posting notes because I might become less interested in this later.
Also it’s unfortunate that unlike in https://intelligence.org/2018/11/02/embedded-models/ version of this article we don’t have hyperlinks to explanations of various concepts here. Perhaps you could add them under the corresponding images? Or have images themselves be hyperlinks or reference links (like in academic articles) to the bottom of the document where all relevant links would be stored grouped by image number.
The post says an embedded agent can’t hold an exact model of the
environment in its head, can’t think through the consequences of every
potential course of action, can’t hold in its head every possible way the environment
could be. I think this may not be necessarily true and I am not sure what assumptions the author used here.
It seems the whole article assumes countable probability spaces (even before the AIXI part). I wonder why and I wonder how realizability is defined for uncountable probability space.
--
Regarding relative bounded loss and what this bound is for, my best guess is as follows. Here I use non-conditional probability notation p(x) instead of π(x). Let e be the elementary event that is actually true. Let “expert” h be a (not necessarily elementary) event, such that p(e,h)>0. Then loss of the expert is Lh=−logp(e∣h). Loss of the prior is L=−logp(e). For their difference it holds that L−Lh=lnp(e,h)p(e)p(h)≤−lnp(h).
Initially, π(h) is your initial trust in expert h, and in each case
where it is even a little bit more correct than you, you increase your
trust accordingly; the way you do this ensures you assign an expert
probability 1 and hence copy it precisely before you lose more than logπ(h) compared to it.
Remember, p(h∣e)=p(e∣h)p(h)p(e). It follows
that probability of h increases given evidence e if and only if p(e∣h)>p(e), i.e. h “is even a little bit more
correct than you”. But I don’t understand the bit about copying the expert
h precisely before losing more than logπ(h). If the expert is an event, how can you “copy” it?
Combining this with the previous idea about viewing Bayesian learning as a
way of allocating “trust” to “experts” which meets a bounded loss
condition, we can see the Solomonoff prior as a kind of ideal machine
learning algorithm which can learn to act like any algorithm you might come
up with, no matter how clever.
It is assuming all possible algorithms are computable, not that the world is.
I don’t understand this. Our probability space is the cartesian product of
the set of all possible UTM programs and the set of all possible UTM
working tape initial configurations. Or, equivalently, the set of outputs
of UTM under these conditions. Hence our whole hypothesis space only
includes computable worlds. What does “can learn to act like any algorithm”
mean here? “It’s getting bounded loss on its predictive accuracy as
compared with any computable predictor.” Huh? Does predictor here mean
expert h? If yes, what does it mean that h is computable and why? All in
all, is the author claiming it’s impossible to have a better computable
predictor than AIXI with Solomonoff prior, even if it has non-computable
worlds in the probability space?
probabilities may not be calibrated
identification of causal structure may not work
What do these mean? I only know informally what calibration means
related to forecasting.
So, does AIXI perform well without a realizability assumption?
How is AIXI even defined without realizability, i.e. when the actual world isn’t in the probability space, or it has zero prior probability?
This is fine if the world “holds still” for us; but because the map is in
the world, it may implement some function.
Is this about the world changing because of the agent just thinking? Or
something else?
It should be noted, though, that there are additional barriers to getting
this property in a game-theoretic setting; so in their common usage cases,
“grain of truth” is technically demanding while “realizability” is a
technical convenience.
...
In game theory, on the other hand, the assumption itself may be
inconsistent. This is because games commonly yield paradoxes of
self-reference.
From the former paragraph I don’t understand anything except that (the
author claims) game
theory has more problems with grain of truth / realizability, than AIXI.
After the latter paragraph, my best guess is: for any game, if there is no
pure strategy equilibrium in it, then we
say it has no grain of truth, because for every possible outcome rational
agents wouldn’t choose it.
If we put weight in both places until a proof rules one out, the beliefs
just oscillate forever rather than doing anything useful.
Weights represent possible worlds, therefore they are on the scales right
from the beginning (the prior), we never put new weights on the scales. My
probably incorrect guess of what the author is saying is
some agent which acts like AIXI but
instead of updating on pieces of evidence as soon as he receives it, he
stockpiles it, and at some points he (boundedly) searches for proofs that
these pieces of evidence are in favor of some hypothesis and performs
update only when he finds them. But still, why oscillation?
Any computable beliefs about logic must have left out something, since the
tree will grow larger than any container.
I interpret it as there are infinitely many theorems, hence an agent with
finite amount of space or finite amount of computation steps can’t process
all of them.
Another consequence of the fact that the world is bigger than you is that
you need to be able to use high-level world models: models which involve
things like tables and chairs.
This is related to the classical symbol grounding problem; but since we
want a formal analysis which increases our trust in some system, the kind
of model which interests us is somewhat different. This also relates to
transparency and informed oversight: world-models should be made out of
understandable parts.
No idea what the second quoted paragraph means.
All in all, I doubt that high level world models are
necessary. And it’s very not clear what is meant by “high level” or
“things” here. Perhaps embedded agents can (boundedly) reason about the
world in other ways, e.g. by modeling only part of the world.
https://intelligence.org/files/OntologicalCrises.pdf explains the
ontological crisis idea better. Suppose our AIXI-like agent thinks the world is an
elementary outcome of some parameterized probability distribution with the
parameter θ. θ is either 1 or 2. We call the set of elementary outcomes
with θ=1 the first ontology (e.g. possible worlds running on classical
mechanics), and the set of elementary outcomes with θ=2
the second ontology (e.g. possible worlds running on superstrings
theory). The programmer has only programmed the agent’s utility functiom
for θ=1 part, i.e. a u function from ontology 1 to real numbers.
The agent keeps count of which value of θ is more probable and
chooses actions by considering only current ontology. If at some point he
decides that the second ontology is more useful, he switches to it.
The agent should extrapolate the utility function to θ=2 part. How can he
do it?
I think this article is too vague, because for almost almost claims in it I am not sure if I understand the author correctly. Below I am posting my notes. If you want to help me and others clarify understanding of this article, consider answering questions in bold, or, if you see a mistake in my notes, correcting it. Also I hope my notes help the author as a piece of feedback. I’ve only finished 2⁄3 of the article so far, but posting notes because I might become less interested in this later.
Also it’s unfortunate that unlike in https://intelligence.org/2018/11/02/embedded-models/ version of this article we don’t have hyperlinks to explanations of various concepts here. Perhaps you could add them under the corresponding images? Or have images themselves be hyperlinks or reference links (like in academic articles) to the bottom of the document where all relevant links would be stored grouped by image number.
The post says an embedded agent can’t hold an exact model of the environment in its head, can’t think through the consequences of every potential course of action, can’t hold in its head every possible way the environment could be. I think this may not be necessarily true and I am not sure what assumptions the author used here.
It seems the whole article assumes countable probability spaces (even before the AIXI part). I wonder why and I wonder how realizability is defined for uncountable probability space.
--
Regarding relative bounded loss and what this bound is for, my best guess is as follows. Here I use non-conditional probability notation p(x) instead of π(x). Let e be the elementary event that is actually true. Let “expert” h be a (not necessarily elementary) event, such that p(e,h)>0. Then loss of the expert is Lh=−logp(e∣h). Loss of the prior is L=−logp(e). For their difference it holds that L−Lh=lnp(e,h)p(e)p(h)≤−lnp(h).
Remember, p(h∣e)=p(e∣h)p(h)p(e). It follows that probability of h increases given evidence e if and only if p(e∣h)>p(e), i.e. h “is even a little bit more correct than you”. But I don’t understand the bit about copying the expert h precisely before losing more than logπ(h). If the expert is an event, how can you “copy” it?
I don’t understand this. Our probability space is the cartesian product of the set of all possible UTM programs and the set of all possible UTM working tape initial configurations. Or, equivalently, the set of outputs of UTM under these conditions. Hence our whole hypothesis space only includes computable worlds. What does “can learn to act like any algorithm” mean here? “It’s getting bounded loss on its predictive accuracy as compared with any computable predictor.” Huh? Does predictor here mean expert h? If yes, what does it mean that h is computable and why? All in all, is the author claiming it’s impossible to have a better computable predictor than AIXI with Solomonoff prior, even if it has non-computable worlds in the probability space?
What do these mean? I only know informally what calibration means related to forecasting.
How is AIXI even defined without realizability, i.e. when the actual world isn’t in the probability space, or it has zero prior probability?
Is this about the world changing because of the agent just thinking? Or something else?
From the former paragraph I don’t understand anything except that (the author claims) game theory has more problems with grain of truth / realizability, than AIXI. After the latter paragraph, my best guess is: for any game, if there is no pure strategy equilibrium in it, then we say it has no grain of truth, because for every possible outcome rational agents wouldn’t choose it.
Weights represent possible worlds, therefore they are on the scales right from the beginning (the prior), we never put new weights on the scales. My probably incorrect guess of what the author is saying is some agent which acts like AIXI but instead of updating on pieces of evidence as soon as he receives it, he stockpiles it, and at some points he (boundedly) searches for proofs that these pieces of evidence are in favor of some hypothesis and performs update only when he finds them. But still, why oscillation?
I interpret it as there are infinitely many theorems, hence an agent with finite amount of space or finite amount of computation steps can’t process all of them.
No idea what the second quoted paragraph means.
All in all, I doubt that high level world models are necessary. And it’s very not clear what is meant by “high level” or “things” here. Perhaps embedded agents can (boundedly) reason about the world in other ways, e.g. by modeling only part of the world.
https://intelligence.org/files/OntologicalCrises.pdf explains the ontological crisis idea better. Suppose our AIXI-like agent thinks the world is an elementary outcome of some parameterized probability distribution with the parameter θ. θ is either 1 or 2. We call the set of elementary outcomes with θ=1 the first ontology (e.g. possible worlds running on classical mechanics), and the set of elementary outcomes with θ=2 the second ontology (e.g. possible worlds running on superstrings theory). The programmer has only programmed the agent’s utility functiom for θ=1 part, i.e. a u function from ontology 1 to real numbers. The agent keeps count of which value of θ is more probable and chooses actions by considering only current ontology. If at some point he decides that the second ontology is more useful, he switches to it. The agent should extrapolate the utility function to θ=2 part. How can he do it?