royf

Karma: 289

royf 24 Jan 2013 1:22 UTC
17 points
on: Right for the Wrong Reasons
Predictions are justified not by becoming a reality, but by the likelihood of their becoming a reality [1]. When this likelihood is hard to estimate, we can take their becoming a reality as weak evidence that the likelihood is high. But in the end, after counting all the evidence, it’s really only the likelihood itself that matters.

If I predict [...] that I will win [...] and I in fact lose fourteen touches in a row, only to win by forfeit

If I place a bet on you to win and this happens, I’ll happily collect my prize, but still feel that I put my money on the wrong athlete. My prior and the signal are rich enough for me to deduce that your victory, although factual, was unlikely. If I believed that you’re likely to win, then my belief wasn’t “true for the wrong reasons”, it was simply false. If I believed that “you will win” (no probability qualifier), then in the many universes where you didn’t I’m in Bayes Hell.

Conversely in the other example, your winning itself is again not the best evidence for its own likelihood. Your scoring 14 touches is. My belief that you’re likely to win is true and justified for the right reasons: you’re clearly the better athlete.

[1] Where likelihood is measured either given what I know, or what I could know, or what anybody could know—depending on why we’re asking the question in the first place.

royf 2 Jun 2012 7:13 UTC
10 points
in reply to: Eliezer Yudkowsky’s comment on: Fake Explanations
You are overstating the case by a large margin.

[Saying “I don’t know”] is still the rational thing to say when, in fact, you don’t know.

Saying “I don’t know” may be, to a large degree, the true state of your belief when you use probability theory. But in this case it’s not the rational thing to say when you use decision theory. “I don’t know” is true, but it is a non-answer to the question, and doesn’t get you points. It’s a different matter whether this point system is effective or moral, but as long as it’s there, that’s what you play by.

You can easily do worse than maximum entropy if you guess at random.

If you base your guess correctly on an incomplete model of reality, which you’ve constructed correctly from past observations, you can never do worse, on average, than maximum entropy. More evidence can never lead to less information (as per the Data Processing Inequality).

Furthermore, “getting it right” [...] does not necessarily mean that you possess any anticipation-controllers.

On the contrary, it mean exactly that. Being rewarded for predictive powers improves your model of the world, whereas “I don’t know” is an excuse for not knowing.

In fact, the mechanism employed by the teacher, for all its flaws, achieves 3 important goals:
- It motivates students to pay attention, raises their level of alertness, activates their brains.
- It rewards students to engage their past observations to generate the most accurate belief they can about the right answer. In the process, they build a better model of the world, and they make their unknown unknowns a little more known.
- By forcing students to generate a belief and commit to it before the correct answer is revealed, their hindsight bias is reduced.

royf 31 May 2012 17:12 UTC
10 points
0
in reply to: thomblake’s comment on: Focus Your Uncertainty
This solution is the Kelly strategy. It isn’t generally optimal, but this makes it optimal in this case:

if some particular outcome occurs, then your utility function is logarithmic in time spent preparing the excuse

royf 20 Sep 2012 17:03 UTC
7 points
in reply to: Unnamed’s comment on: Less Wrong Polls in Comments
To anyone thinking this is not random, with 42 votes in:
- The p-value is 0.895 (this is the probability of seeing at least this much non-randomness, assuming a uniform distribution)
- The entropy is 2.302bits instead of log(5) = 2.322bits, for 0.02bits KL-distance (this is the number of bits you lose for encoding one of these votes as if it was random)
If you think you see a pattern here, you should either see a doctor or a statistician.

royf 28 May 2012 5:03 UTC
7 points
on: What is Evidence?

If your beliefs are entangled with reality, they should be contagious among honest folk.
[...]
If your model of reality suggests that the outputs of your thought processes should not be contagious to others, then your model says that your beliefs are not themselves evidence, meaning they are not entangled with reality.

No, correct beliefs should only be contagious among honest folk who believe each other to be rational and honest. If I make the claim that The FSM is dictating these words to me, you would probably think me lunatic or liar. But if I truly can correctly recognize when I have been Touched by His Noodly Appendage, then my beliefs are entangled with reality but, understandably, not contagious. Furthermore, it would be perfectly rational for me to believe this revelation and at the same time not to consider it evidence for others. The point is that some beliefs, certainly the more extraordinary of them, should not be contagious, except through evidence as raw and unprocessed as possible.

Also, entanglement is necessary but not sufficient for correct beliefs. The fact that my beliefs contain information about the world is not enough for them to be correct. For example, if I misread the photon pattern, I could think that my shoelaces are tied when they are not, and untied when they are tied. This still has the same amount of entanglement, the same amount of information, yet the beliefs are incorrect.

royf 18 Sep 2012 18:09 UTC
6 points
in reply to: Richard_Kennaway’s comment on: The Bayesian Agent
Everything you say is essentially true.

As the designer of the agent, will you be explicitly providing it with that information in some future instalment?

Technically, we don’t need to provide the agent with p and sigma explicitly. We use these parameters when we build the agent’s memory update scheme, but the agent is not necessarily “aware” of the values of the parameters from inside the algorithm.

Let’s take for example an autonomous rover on Mars. The gravity on Mars is known at the time of design, so the rover’s software, and even hardware, is built to operate under these dynamics. The wind velocity at the time and place of landing, on the other hand, is unknown. The rover may need to take measurements to determine this parameter, and encode it in its memory, before it can take it into account in choosing further actions.

But if we are thoroughly Bayesian, then something is known about the wind prior to experience. Is it likely to change every 5 minutes or can the rover wait longer before measuring again? What should be the operational range of the instruments? And so on. In this case we would include this prior in p, while the actual wind velocity is instead hidden in the world state (only to be observed occasionally and partially).

Ultimately, we could include all of physics in our belief—there’s always some Einstein to tell us that Newtonian physics is wrong. The problem is that a large belief space makes learning harder. This is why most humans struggle with intuitive understanding of relativity or quantum mechanics—our brains are not made to represent this part of the belief space.

This is also why reinforcement learning gives special treatment to the case where there are unknown but unchanging parameters of the world dynamics: the “unknown” part makes the belief space large enough to make special algorithms necessary, while the “unchanging” part makes these algorithms possible.

For LaTeX instructions, click “Show help” and then “More Help” (or go here).

royf 7 Jan 2013 21:13 UTC
5 points
in reply to: simplicio’s comment on: How to Be Oversurprised
Philosiphically, yes.

Practically, it may be useful to distinguish between a coin and a toss. The coin has persisting features which make it either fair or loaded for a long time, with correlation between past and future. The toss is transient, and essentially all information about it is lost when I put the coin away—except through the memory of agents.

So yes, the toss is a feature of the present state of the world. But it has the very special property, that given the bias of the coin, the toss is independent of the past and the future. It’s sometimes more useful to treat a feature like that as an observation external to the world, but of course it “really” isn’t.

royf 9 Aug 2012 15:06 UTC
5 points
in reply to: Richard_Kennaway’s comment on: Reinforcement, Preference and Utility
Clearly you have some password I’m supposed to guess.

This post is not preliminary. It’s supposed to be interesting in itself. If it’s not, then I’m doing something wrong, and would appreciate constructive criticism.

royf 15 Jun 2012 18:44 UTC
5 points
in reply to: royf’s comment on: My Wild and Reckless Youth
The Harsanyi paper is very enlightening, but he’s not really arguing that people have shared priors. Rather, he’s making the following points (section 14):
- It is worthwhile for an agent to analyze the game as if all agents have the same prior, because it simplifies the analysis. In particular, the game (from that agent’s point of view) then becomes equivalent to a Bayesian complete-information game with private observations.
- The same-prior assumption is less restrictive than it may seem, because agents can still have private observations.
- A wide family of hypothetical scenarios can be analyzed as if all agents have the same prior. Other scenarios can be easily approximated by a member of this family (though the quality of the approximation is not studied).
All of this is mathematically very pleasing, but it doesn’t change my point. That’s mainly because in the context of the Harsanyi paper “prior” means before any observation, and in the context of this post “prior” means before the shared observation (but possibly after private observations).

royf 14 Jun 2012 6:36 UTC
5 points
on: My Wild and Reckless Youth

Traditional Rationalists can agree to disagree. Traditional Rationality doesn’t have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence.

This is also true of Bayesians. The probability estimate given the evidence is a property of the map, not the territory (hence “estimate”). One correct posterior implies one correct prior. What is this “Ultimate Prior”? There isn’t one.

Possibly, you meant that there’s one correct posterior given the evidence and the prior. That’s correct, but it doesn’t prevent Bayesians from disagreeing, because they do have different priors.

Alternatively, one can point out that the “given evidence” operator is, in expectation, always non-expansive, and contractive when the priors disagree. This means that the beliefs of Perfect Bayesians with shared observations converge (with probability 1) into a single posterior. But this convergence is too slow for humans. Agreeing to disagree is sometimes our only option.

Incidentally, it’s Traditional Rationalists who believed they should never agree to disagree: the set of hypotheses which aren’t “ruled out” by confirmed and repeatable experiments, they argued, is a property of the territory.

royf 18 Sep 2012 3:12 UTC
4 points
on: The Bayesian Agent
If you’re a devoted Bayesian, you probably know how to update on evidence, and even how to do so repeatedly on a sequence of observations. What you may not know is how to update in a changing world. Here’s how:

$B_{t 1} (W_{t 1}$ %3d\Pr(W_{t+1}|O1,\ldots,O{t+1})%3d\frac{\sigma(O{t+1}|W{t+1})\cdot\Pr(W_{t+1}|O_1,\ldots,O_t)}{\sumw\sigma(O{t+1}|w)\cdot\Pr(w|O_1,\ldots,O_t)})

As usual with Bayes’ theorem, we only need to calculate the numerator for different values of $W_{t 1}$ , and the denominator will normalize them to sum to 1, as probabilities do. We know $σ$ as part of the dynamics of the system, so we only need $Pr (W_{t 1} | O_{1}, \dots, O_{t}$ ). This can be calculated by introducing the other variables in the process:

$Pr (W_{t 1} | O_{1}, \dots, O_{t}$ %3d\sum_{W_t,A_t}\Pr(W_t,At,W{t+1}|O_1,\ldots,O_t))

An important thing to notice is that, given the observable history, the world state $W_{t}$ and the action $A_{t}$ are independent—the agent can’t act on unseen information. We continue:

$= \sum_{W_{t}, A_{t}} Pr (W_{t} | O_{1}, \dots, O_{t}$ \cdot\Pr(A_t|O_1,\ldots,Ot)\cdot%20p(W{t+1}|W_t,A_t))

Recall that the agent’s belief $B_{t}$ is a function of the observable history, and that the action only depends on the observable history through its memory $B_{t}$ . We conclude:

$= \sum_{W_{t}, A_{t}} B_{t} (W_{t}$ \cdot\pi(A_t|Bt)\cdot%20p(W{t+1}|W_t,A_t))
What links here?
- The Bayesian Agent by royf (18 Sep 2012 3:23 UTC; 19 points)

royf 23 Jul 2012 15:07 UTC
4 points
in reply to: Oscar_Cunningham’s comment on: The Perception-Action Cycle
Excellent point. It will be a few posts (if the audience is interested) before I can answer you in a way that is both intuitive and fully convincing.

The technical answer is that the belief update caused by an action is deterministically contractive. It never increases the amount of information.

A more intuitive answer (but perhaps not yet convincing) is that, proximally, your action of asking your friend did not change the location of your laptop, only your friend’s mental state. And the effect it had on your friend’s mental state is that you are now less sure of what it is. You took some of the things you knew about it (“she was probably going to keep sitting around”) and made them no longer true.

Regarding information about the future (of the location of your laptop), it is always included in information about the present. Your own mental state is independent of the future given the present. Put another way, you can’t update without more observations. In this case, the location of your laptop merely becomes entangled with information that you already had on your friend’s mental state (“this is where she thinks my desk is”).

royf 28 May 2012 5:16 UTC
4 points
on: How Much Evidence Does It Take?

131,115,985 to 1 [...] this amount of evidence would only be enough to give you an even chance of winning the lottery.

The number of false bleeps is distributed almost exactly Poisson with $λ = 1$ . The important figure is not the expected number of bleeps ( $E x 1$ , which is indeed 2). It’s the expected probability that a random bleep is the true one, $E \frac{1}{x 1}$ . At the moment I can’t find an analytic solution (and a short search suggests none is known), but a computation shows the result is around 63.2%, much better than 50%. Similarly, with 14 boxes (arguably “28 bits of evidence”), the chance of winning is about 79.1% on average, much better than $\frac{2}{3}$ .

royf 21 Jan 2013 22:39 UTC
3 points
in reply to: William_Quixote’s comment on: Update Then Forget
Thanks!

The best book is doubtlessly Elements of Information Theory by Cover and Thomas. It’s very clear (to someone with some background in math or theoretical computer science) and lays very strong introductory foundations before giving a good overview of some of the deeper aspects of the theory.

It’s fortunate that many concepts of information theory share some of their mathematical meaning with the everyday meaning. This way I can explain the new theory (popularized here for the first time) without defining these concepts.

I’m planning another sequence where these and other concepts will be expressed in the philosophical framework of this community. But I should’ve realized that some readers should be interested in a complete mathematical introduction. That book is what you’re looking for.

royf 17 Jan 2013 23:26 UTC
3 points
in reply to: shminux’s comment on: Update Then Forget

an intelligent agent should update on it and then forget it.

Should being the operative word. This refers to a “perfect” agent (emphasis added in text; thanks!).

People don’t do this, as well they shouldn’t, because we update poorly and need the original data to compensate.

If you forget the discarded cards, and later realize that you may have an incorrect map of the deck, aren’t you SOL?

If I remember the cards in play, I don’t care about the discarded ones. If I don’t, the discarded cards could help a bit, but that’s not the heart of my problem.

royf 14 Jun 2012 6:55 UTC
3 points
in reply to: beoShaffer’s comment on: My Wild and Reckless Youth
I’m aware of this result. It specifically requires the two Beyesians to have the same prior. My point is exactly that this doesn’t have to be the case, and in reality is sometimes not the case.

EDIT: The original paper by Aumann references a paper by Harsanyi which supposedly addresses my point. Aumann himself is careful in interpreting his result as supporting my point (since evidently there are people who disagree despite trusting each other). I’ll report here my understanding of the Harsanyi paper once I get past the paywall.

royf 4 Jun 2012 6:13 UTC
3 points
in reply to: CuSithBell’s comment on: Fake Causality
I am saying pretty much exactly that. To clarify further, the words “deliberate”, “conscious” and “wants” again belong to the level of emergent behavior: they can be used to describe the agent, not to explain it (what could not be explained by “the agent did X because it wanted to”?).

Let’s instead make an attempt to explain. A complete control of an agent’s own code, in the strict sense, is in contradiction of Gödel’s incompleteness theorem. Furthermore, information-theoretic considerations significantly limit the degree to which an agent can control its own code (I’m wondering if anyone has ever done the math. I expect not. I intend to look further into this). In information-theoretic terminology, the agent will be limited to typical manipulations of its own code, which will be a strict (and presumably very small) subset of all possible manipulations.

Can an agent be made more effective than humans in manipulating its own code? I have very little doubt that it can. Can it lead to agents qualitatively more intelligent than humans? Again, I believe so. But I don’t see a reason to believe that the code-rewriting ability itself can be qualitatively different than a human’s, only quantitatively so (although of course the engineering details can be much different; I’m referring to the algorithmic level here).

Generally GAIs are ascribed extreme powers around here

As you’ve probably figured out, I’m new here. I encountered this post while reading the sequences. Although I’m somewhat learned on the subject, I haven’t yet reached the part (which I trust exists) where GAI is discussed here.

On my path there, I’m actively trying to avoid a certain degree of group thinking which I detect in some of the comments here. Please take no offense, but it’s phrases like the above quote which worry me: is there really a consensus around here about such profound questions? Hopefully it’s only the terminology which is agreed upon, in which case I will learn it in time. But please, let’s make our terminology “pay rent”.

royf 2 Jun 2012 7:35 UTC
3 points
in reply to: Dmytry’s comment on: Fake Explanations
I agree that the word “why” is perhaps not the best choice, but it really stands for “what is the physical mechanism for”.

Anyway, what you describe is engineering, not science. A scientist explains the mechanism by which a stone ax can kill a mammoth. An engineer uses this understanding to design a better ax. A craftsman builds the ax. A warrior takes the ax, kills the mammoth, and wins the girl.

royf 17 Jul 2015 11:23 UTC
2 points
on: An overall schema for the friendly AI problems: self-referential convergence criteria
It seems that your research is coming around to some concepts that are at the basis of mine. Namely, that noise in an optimization process is a constraint on the process, and that the resulting constrained optimization process avoids the nasty properties you describe.

Feel free to contact me if you’d like to discuss this further.

royf 27 Mar 2015 17:59 UTC
2 points
on: Utility vs Probability: idea synthesis
This is not unlike Neyman-Pearson theory. Surely this will run into the same trouble with more than 2 possible actions.