(ignorance prior) Equally weight all hypotheses that explain the data.
This prior is impossible to implement for almost all situations, because there is no way to assign uniform probabilities over an infinite number of things—you just end up with 0 everywhere, which isn’t normalized, and hence is not a probability distribution.
(utilitarian prior) The likelihood of a hypothesis is proportional to the expected net utility of the agent believing it.
So problem one is that this is circular if someone values the truth.
Then if this is ignored, there’s still a problem that it’s unstable—someone who things a stock market crash is likely will think that it is high-utility to believe this, and someone who thinks a stock market crash is unlikely will think that their thought is the high-utility one, and so you have a feedback loop, naturally driving people to extremes but not favoring one extreme over another, which is bad since usually either A or ~A is true, but not both.
The third problem is sentences like “it is high utility to believe that this sentence is true and that Santa Claus exists,” (or “you will go to heaven if you believe this sentence and God exists”) though this can be solved by including a full zeroth-order theory of human utility inside what used to look like a one-sentence maxim.
However, it does have an important thing going for it—it’s a first step towards thinking about what a self-modifying agent with a utility function would choose to use as a prior.
My aesthetics prefers the O’Neill way over the Carter way: the Goldpan over the Razor.
Is this because the show repeatedly portrays one as working and the other as not working? This sounds like the common “Hollywood rationality” trope.
A proposition is true if the use of its content would be beneficial in all relevant utility functions.
There is no such proposition, if we don’t allow the Santa Claus sentences.
(social prior) The likelihood of a hypothesis is proportional to how recently and how often it has been proposed and to the social status of its proponents.
Yeah that’s just a bad plan. “It is the highest-utility option for you to grovel before me, peons.” And it has no reason to exist, since what are the people making the propositions using to guess at what’s true?
~~
The others didn’t have any really killer (e.g. “the things this outputs aren’t probabilities”) flaws I could see—they just end you up with different sets of beliefs. That can still be bad, if the belief “that’s a tiger about to leap at me” gets rated at low probability when it should have been high, but oh well.
This prior is impossible to implement for almost all situations, because there is no way to assign uniform probabilities over an infinite number of things—you just end up with 0 everywhere, which isn’t normalized, and hence is not a probability distribution.
Is there some reason we can’t bypass this problem by using, say, the surreal numbers instead of the real numbers? This is a genuine question, not a rhetorical one.
So problem one is that this is circular if someone values the truth.
The same criticism applies to Occam’s Razor and all a priori hypothesis weighting heuristics. A critic of Solomonoff induction would say “I value truth, not simplicity per se”.
For your stock market example, note that any theory which lets you predict the stock market would be high-utility. Some theories are complete opposites, like “the stock market will crash” and “the stock market will boom”. By symmetry I suspect that their weighted contributions to the sum of models will cancel out. The same goes for Santa sentences, “it is high utility to believe that this sentence is true and that Santa Claus exists” and “it is high utility to believe that this sentence is true and that Santa Claus does not exist” simply cancel out. Weighting hypotheses by utility doesn’t pull them up by their own bootstraps. It’s only a prior probability—actually making decisions will depend on posterior probabilities.
Is this because the show repeatedly portrays one as working and the other as not working? This sounds like the common “Hollywood rationality” trope.
Of course. I assumed that readers of this site would know to take a fictional example as an illustration of a principle but not as evidence.
Yeah that’s just a bad plan. “It is the highest-utility option for you to grovel before me, peons.”
Haha, obviously so. But you and I probably both know people whom it describes, so it was amusing to include.
Is there some reason we can’t bypass this problem by using, say, the surreal numbers instead of the real numbers? This is a genuine question, not a rhetorical one.
I dunno, but I don’t see any reason why that should be the case—especially on continuous domains.
So problem one is that this is circular if someone values the truth.
The same criticism applies to Occam’s Razor
Not quite—Occam’s razor is arbitrary, but it isn’t circular. The circularity comes in from thinking both “it is good to believe true things” and “things that are good to believe are more likely to be true.”
For your stock market example, note that any theory which lets you predict the stock market would be high-utility. Some theories are complete opposites, like “the stock market will crash” and “the stock market will boom”. By symmetry I suspect that their weighted contributions to the sum of models will cancel out.
“By symmetry” seems to be misapplied in this particular case. Think of a single person with a single belief. What does that person do when faced with a choice of how to update their belief?
The same goes for Santa sentences, “it is high utility to believe that this sentence is true and that Santa Claus exists” and “it is high utility to believe that this sentence is true and that Santa Claus does not exist” simply cancel out.
There’s still a problematic contradiction between “it is high utility to believe that this sentence is true and that Santa Claus exists” and “Santa Claus exists.”
I dunno, but I don’t see any reason why that should be the case—especially on continuous domains.
Now that I consider it a bit more, since the number of deterministic programs for modeling an environment is countably infinite, it should only require a hyperreal infinitesimal weight to maintain conservation of probability. The surreals are completely overkill. And furthermore, that’s only in the ideal case—a practical implementation would only examine a finite subset of programs, in which case the theoretical difficulty doesn’t even arise.
Not quite—Occam’s razor is arbitrary, but it isn’t circular. The circularity comes in from thinking both “it is good to believe true things” and “things that are good to believe are more likely to be true.”
It doesn’t look circular to me. It just looks like a restatement. If you consider a specific model of an environment, the way you evaluate its posterior probability (i.e. whether it’s true) is from its predictions, and the way you get utility from the model is also by acting on its predictions. The truth and the goodness of a belief end up being perfectly dependent factors when your posterior probability is dominated by evidence, so it doesn’t seem problematic to me to also have the truth and goodness of a belief unified for evaluation of the prior probability.
“By symmetry” seems to be misapplied in this particular case. Think of a single person with a single belief.
Hm, I think you and I are viewing these differently. I had in mind an analogy to the AIXI model of AI: it’s a single entity, but it doesn’t have just a single belief. AIXI keeps all the beliefs that fit and weights them according to the Solomonoff prior, then it acts based on the weighted combination of all those beliefs. Now obviously I haven’t done the math so I could be way off here, but I suspect that the appeal to symmetry works in the case of equal-and-opposite high-utility beliefs like the ones mentioned for the stock market and Santa clause precisely because the analogous AI model with the Goldpan would keeps all beliefs (in a weighted combination) instead of choosing just one.
It should only require a hyperreal infinitesimal weight to maintain conservation of probability.
Doing arithmetic on infinities is not the same as doing infinite sequences of arithmetic. You can talk about a hyperreal-valued uniform prior, but can you actually do anything with it that you couldn’t do with an ordinary limit?
P(o%7CH_i)n%5E{-1})
The reasons that limit doesn’t suffice to specify a uniform prior are: (1) The result of the limit depends on the order of the list of hypotheses, which doesn’t sound very uniform to me (I don’t know if it’s worse than the choice of universal Turing machine in Solomonoff, but at least the pre-theoretic notion of simplicity comes with intuitions about which UTMs are simple). (2) For even more perverse orders, the limit doesn’t have to converge at all. (Even if utility is bounded, partial sums of EU can bounce around the bounded range forever.)
Hyperreal-valued expected utility doesn’t change (1). It does eliminate (2), but I think you have to sacrifice computability to do even that much: Construction of the hyperreals involves the axiom of choice, which prevents you from actually determining which real number is infinitesimally close to the hyperreal encoded by a given divergent sequence.
You wrote the above in regards to hyperreal infinities, “the hyperreal encoded by a given divergent sequence”. I’m under the impression that hyperreal infinitesimals are encoded by convergent sequences: specifically, sequences that converge to zero. The hyperreal [1, 1⁄2, 1⁄3, 1⁄4, …] is the one that corresponds to the limit you gave. Does that adequately dispel the computability issue you raised?
In any case, non-computability isn’t a major defect of the utilitarian prior vis-a-vis the also non-computable Solomonoff prior. It is an important caution, however.
Your first objection seems much more damaging to the idea of a utilitarian prior. Indeed, there seems little reason to expect max(U(o|Hi)) to vary in a systematic way with a useful enumeration of the hypotheses.
A non-constant sequence that converges to zero encodes an infinitesimal, and I think any infinitesimal has an encoding of that form. But a sequence that’s bounded in absolute value but doesn’t converge, e.g.
), also encodes some real plus some infinitesimal. It’s this latter kind that involves the axiom of choice, to put it in an equivalence class with some convergent sequence.
[1, 1⁄2, 1⁄3, 1⁄4, …] is the infinitesimal in the proposed definition of a uniform prior, but the hyperreal outcome of the expected utility calculation is
,%20{1\over%202}\sum_{i=1}%5E2U(o%7CH_i),%20{1\over%203}\sum_{i=1}%5E3U(o%7CH_i),%20...]) which might very well be the divergent kind.
Agreed that my first objection was more important.
This prior is impossible to implement for almost all situations, because there is no way to assign uniform probabilities over an infinite number of things—you just end up with 0 everywhere, which isn’t normalized, and hence is not a probability distribution.
If you have enough data, you can get a proper posterior from an improper prior.
That’s true. I guess it could be your “prior” even if it’s not a probability distribution. You don’t just need “enough” data, then—you need data that gives you quickly-decreasing likelihoods as a function of some parameter that numbers finite bunches of hypotheses. Which I don’t think is all that common.
This prior is impossible to implement for almost all situations, because there is no way to assign uniform probabilities over an infinite number of things—you just end up with 0 everywhere, which isn’t normalized, and hence is not a probability distribution.
So problem one is that this is circular if someone values the truth.
Then if this is ignored, there’s still a problem that it’s unstable—someone who things a stock market crash is likely will think that it is high-utility to believe this, and someone who thinks a stock market crash is unlikely will think that their thought is the high-utility one, and so you have a feedback loop, naturally driving people to extremes but not favoring one extreme over another, which is bad since usually either A or ~A is true, but not both.
The third problem is sentences like “it is high utility to believe that this sentence is true and that Santa Claus exists,” (or “you will go to heaven if you believe this sentence and God exists”) though this can be solved by including a full zeroth-order theory of human utility inside what used to look like a one-sentence maxim.
However, it does have an important thing going for it—it’s a first step towards thinking about what a self-modifying agent with a utility function would choose to use as a prior.
Is this because the show repeatedly portrays one as working and the other as not working? This sounds like the common “Hollywood rationality” trope.
There is no such proposition, if we don’t allow the Santa Claus sentences.
Yeah that’s just a bad plan. “It is the highest-utility option for you to grovel before me, peons.” And it has no reason to exist, since what are the people making the propositions using to guess at what’s true?
~~
The others didn’t have any really killer (e.g. “the things this outputs aren’t probabilities”) flaws I could see—they just end you up with different sets of beliefs. That can still be bad, if the belief “that’s a tiger about to leap at me” gets rated at low probability when it should have been high, but oh well.
Is there some reason we can’t bypass this problem by using, say, the surreal numbers instead of the real numbers? This is a genuine question, not a rhetorical one.
The same criticism applies to Occam’s Razor and all a priori hypothesis weighting heuristics. A critic of Solomonoff induction would say “I value truth, not simplicity per se”.
For your stock market example, note that any theory which lets you predict the stock market would be high-utility. Some theories are complete opposites, like “the stock market will crash” and “the stock market will boom”. By symmetry I suspect that their weighted contributions to the sum of models will cancel out. The same goes for Santa sentences, “it is high utility to believe that this sentence is true and that Santa Claus exists” and “it is high utility to believe that this sentence is true and that Santa Claus does not exist” simply cancel out. Weighting hypotheses by utility doesn’t pull them up by their own bootstraps. It’s only a prior probability—actually making decisions will depend on posterior probabilities.
Of course. I assumed that readers of this site would know to take a fictional example as an illustration of a principle but not as evidence.
Haha, obviously so. But you and I probably both know people whom it describes, so it was amusing to include.
I dunno, but I don’t see any reason why that should be the case—especially on continuous domains.
Not quite—Occam’s razor is arbitrary, but it isn’t circular. The circularity comes in from thinking both “it is good to believe true things” and “things that are good to believe are more likely to be true.”
“By symmetry” seems to be misapplied in this particular case. Think of a single person with a single belief. What does that person do when faced with a choice of how to update their belief?
There’s still a problematic contradiction between “it is high utility to believe that this sentence is true and that Santa Claus exists” and “Santa Claus exists.”
Now that I consider it a bit more, since the number of deterministic programs for modeling an environment is countably infinite, it should only require a hyperreal infinitesimal weight to maintain conservation of probability. The surreals are completely overkill. And furthermore, that’s only in the ideal case—a practical implementation would only examine a finite subset of programs, in which case the theoretical difficulty doesn’t even arise.
It doesn’t look circular to me. It just looks like a restatement. If you consider a specific model of an environment, the way you evaluate its posterior probability (i.e. whether it’s true) is from its predictions, and the way you get utility from the model is also by acting on its predictions. The truth and the goodness of a belief end up being perfectly dependent factors when your posterior probability is dominated by evidence, so it doesn’t seem problematic to me to also have the truth and goodness of a belief unified for evaluation of the prior probability.
Hm, I think you and I are viewing these differently. I had in mind an analogy to the AIXI model of AI: it’s a single entity, but it doesn’t have just a single belief. AIXI keeps all the beliefs that fit and weights them according to the Solomonoff prior, then it acts based on the weighted combination of all those beliefs. Now obviously I haven’t done the math so I could be way off here, but I suspect that the appeal to symmetry works in the case of equal-and-opposite high-utility beliefs like the ones mentioned for the stock market and Santa clause precisely because the analogous AI model with the Goldpan would keeps all beliefs (in a weighted combination) instead of choosing just one.
Doing arithmetic on infinities is not the same as doing infinite sequences of arithmetic. You can talk about a hyperreal-valued uniform prior, but can you actually do anything with it that you couldn’t do with an ordinary limit?
The reasons that limit doesn’t suffice to specify a uniform prior are: (1) The result of the limit depends on the order of the list of hypotheses, which doesn’t sound very uniform to me (I don’t know if it’s worse than the choice of universal Turing machine in Solomonoff, but at least the pre-theoretic notion of simplicity comes with intuitions about which UTMs are simple). (2) For even more perverse orders, the limit doesn’t have to converge at all. (Even if utility is bounded, partial sums of EU can bounce around the bounded range forever.)
Hyperreal-valued expected utility doesn’t change (1). It does eliminate (2), but I think you have to sacrifice computability to do even that much: Construction of the hyperreals involves the axiom of choice, which prevents you from actually determining which real number is infinitesimally close to the hyperreal encoded by a given divergent sequence.
Thanks!
You wrote the above in regards to hyperreal infinities, “the hyperreal encoded by a given divergent sequence”. I’m under the impression that hyperreal infinitesimals are encoded by convergent sequences: specifically, sequences that converge to zero. The hyperreal [1, 1⁄2, 1⁄3, 1⁄4, …] is the one that corresponds to the limit you gave. Does that adequately dispel the computability issue you raised?
In any case, non-computability isn’t a major defect of the utilitarian prior vis-a-vis the also non-computable Solomonoff prior. It is an important caution, however.
Your first objection seems much more damaging to the idea of a utilitarian prior. Indeed, there seems little reason to expect max(U(o|Hi)) to vary in a systematic way with a useful enumeration of the hypotheses.
A non-constant sequence that converges to zero encodes an infinitesimal, and I think any infinitesimal has an encoding of that form. But a sequence that’s bounded in absolute value but doesn’t converge, e.g.
[1, 1⁄2, 1⁄3, 1⁄4, …] is the infinitesimal in the proposed definition of a uniform prior, but the hyperreal outcome of the expected utility calculation is
Agreed that my first objection was more important.
If you have enough data, you can get a proper posterior from an improper prior.
That’s true. I guess it could be your “prior” even if it’s not a probability distribution. You don’t just need “enough” data, then—you need data that gives you quickly-decreasing likelihoods as a function of some parameter that numbers finite bunches of hypotheses. Which I don’t think is all that common.