paulfchristiano(Paul Christiano)

Karma: 27,044

paulfchristiano 15 Aug 2010 5:28 UTC
9 points
on: Welcome to Less Wrong! (2010-2011)
I am an undergraduate mathematician currently headed towards a life of doing theoretical computer science research. Several unrelated friends mentioned LW to me at one point or another in my life, so I read an arbitrary well-liked post every so often for a while. Eventually I concluded that visiting the site somewhat regularly would make me happy (although I have thought enough about how I think, and am easily arrogant enough, to doubt that I will become a better person or learn too much about myself) and so here I am.

I am an (almost) Bayesian utility maximizer when I manage to do what I think I should. My utility is the expected quality of a uniformly random instant of conscious experience (although less flagrantly ill-defined than suggested by such a summary). In practice I am fairly selfish and lazy, but also good at accepting unpalatable arguments.

I am interested mostly in solving problems whose solutions I think would reduce suffering significantly compared to their difficulty, but I also spend a little time thinking about more philosophical issues and questioning my current decision making procedure. I guess a more precise picture of my interests will emerge as I make more comments, if I do, and will be irrelevant, if I don’t.

paulfchristiano 25 Aug 2010 3:18 UTC
6 points
on: Rationality Lessons in the Game of Go
The lesson I learned most effectively from Go was to acknowledge mistakes; when I was a weak player one of my worst faults was being unwilling to admit to my opponent that they had outplayed me. This is particularly clear when compared to chess.

In chess, if you fall behind in material or tempo the correct thing (in my experience; I am a weak chess player) is almost always to see what you gained in exchange and try to exploit that. If you gained nothing in exchange and the loss was significant, your play is typically irrelevant because you lost. This encourages you to search for justifications for your moves: I lost that piece as part of this attack, or that sacrifice improved my position in this way, or so forth.

In contrast, if you are outplayed in Go the correct thing (in my experience; I am about 10k, which is slightly better than I am at chess but still quite weak) is almost always to recognize that you have been outplayed. My original problem was to reason, “If I cannot beat my opponent in this fight, then he is better than me” and then continue a lost fight, or perhaps to assume that the stones would eventually serve some other function (which they typically would not). Eventually I realized that a game of Go consists of many fights, and that you are severely handicapping yourself if you lose the game whenever one of them goes badly. Sometimes, even in deterministic games of perfect information, you will win most but not all fights. The only way I can now win games is by frequently recognizing that my previous moves were errors and abandoning whatever rationalization for those moves I may be tempted to construct.

The difficulty of internalizing this lesson leads me to suspect that I make the same mistake in my life. I am constantly tempted to justify my decisions; the temptation is even greater when I am forced to justify myself to people around me, and the only alternative is to explicitly admit that I made a mistake.
What links here?
- Rationality Lessons in the Game of Go by GreenRoot (21 Aug 2010 14:33 UTC; 49 points)

paulfchristiano 21 Sep 2010 2:47 UTC
13 points
on: Less Wrong Should Confront Wrongness Wherever it Appears
I believe that if Less Wrong users changed the posts they made and the posts/comments they upvoted to try and “confront wrongness wherever it appears,” as you suggest, that the value of Less Wrong to human society would be significantly increased.

Contrary to the principles behind Less Wrong (or what I perceive to be the principles behind Less Wrong), I doubt that most Less Wrong users can get anywhere near as much value by thinking about being rational for a minute as they could by applying their rationality for a minute. Trying to develop consensus among Less Wrong users on difficult issues seems like it is almost certainly the most effective way we can apply our rationality in the context of making posts on Less Wrong, because consensus among Less Wrong users is a valuable thing to reference when making decisions which are actually important about issues which are not, strictly speaking, relevant topics for Less Wrong.

Whether the correct response is to stop reading/posting on Less Wrong or to confront wrongness on Less Wrong is unclear (you can tell from my post count what I have done in the past).

[In general I enjoy Less Wrong and believe its existence is good, but I also tend to agree with some recent posts that for most people it is a distraction rather than a resource and should be treated as such.]
What links here?
- jimrandomh's comment on Less Wrong Should Confront Wrongness Wherever it Appears by jimrandomh (21 Sep 2010 4:10 UTC; 4 points)

paulfchristiano 21 Sep 2010 18:14 UTC
3 points
in reply to: cousin_it’s comment on: Less Wrong Should Confront Wrongness Wherever it Appears
The political issues which might be discussed are not questions of organization or political structure (in general)--they are questions which belong to some other domain and which happen to have political implications. The fact that they are political changes the nature of discussion about the problem, not the nature of the problem.

I think one effect on the nature of discussion is to stop people from breaking off small problems which they can solve or assertions whose truth they can honestly assess. Instead conversations almost always expand until they are unmanageable. For example, starting out with the question “Should I vote for A or B” is exceptionally unlikely to yield useful discussion. Of course, I don’t know about the interminable discussions which have occurred here in the past or why they might not have been productive. I would guess that the problems discussed were much too large for significant progress to be plausible. This is not because we lack a foothold—it is because we tend to (and there are forces encouraging us to) tackle questions much too large.

I believe it is possible to bite off reasonably sized (ie tiny) questions which we can make genuine progress on. If the problem will be around for a long time, such progress might be very valuable. Maybe a small problem takes the form of a simplified hypothetical whose resolution cannot be black-boxed and used in future discussions (because of course that hypothetical will never occur) but which is likely to help us develop general arguments and standards which might be brought to bear on more complex hypotheticals. Maybe it takes the form of some very simple, concrete, assertion about the world whose answer is politically charged but which can be resolved decisively with a little research and care. Even if such assertions have limited implications, at least its starting to get you somewhere.

I am not in agreement with the original topic. I believe this tactic (of responding to wrongness directly regarding controversial issues) is likely to cause us to also bite off much too large a problem.

However, I do believe that the criterion for declaring something on-topic is a good one, and could increase the value of Less Wrong as a resource if it was used to discuss small issues which we could reasonably make progress on. If such small topics are motivated by general failures of the public discourse, it seems possible that we would eventually accumulate enough rigorous understanding to correct those failures “conclusively.” (at least, as conclusively as any discussion on Less Wrong establishes anything)

paulfchristiano 26 Sep 2010 22:04 UTC
7 points
on: What Makes My Attempt Special?
Some questions/answers are hard to think of, not because we are missing knowledge or technology but because they are hard to think of.

I guess this is basically case (1), but I think “vanishingly small” is a bad description in most fields. For example, a significant fraction of advances in theoretical fields are of this form (the exception being for rapidly developing fields). I agree that if you think you have found a proof that P!=NP you should stop and reflect on how many people have thought for a very long time about it, but in general you should not be shocked that you have thought of a question or answer that eluded other researchers, potentially for a rather long time (since there are very many questions and not that many researchers equipped to ask or answer most of them).

I guess you are claiming that the situation is fundamentally different for experimentalists, but I would still be rather shocked if (1) was really negligible. At best there may be some fields where (1) is particularly uncommon (and for people who are still deciding what they want to do with their lives, it seems like this suggests a lot of interesting data which are really hard to extract from the normal discourse about possible fields of study).

paulfchristiano 28 Sep 2010 18:02 UTC
0 points
on: Open Thread September, Part 3
If I live in a universe which is not Turing computable but try and apply Solomonoff induction, I may end up in trouble—I may be unable to accept the existence of a black box for the halting problem, for example, even when believing in such a thing is useful. There are several possible solutions to this problem, but I have not seen any here which I find satisfactory.

My solution is this. Instead of considering a prior on the space of models for the universe (since you can’t really have a prior on the uncountable space of ways the universe could work if it weren’t as restricted as Solomonoff induction thinks it is) consider all computable strategies you could use to achieve whatever goal you are interested in (e.g., to predict what the universe will do next). Rather than disqualifying a model when it fails to conform to our expectations, we penalize a strategy (say by reducing its weight by some constant factor) when it fails to perform well (e.g., makes an incorrect prediction or fails to make a useful prediction) and follow the strategy which currently has the highest weight.

If your goal is predicting the next bit handed to you by the universe, this is the same as Solomonoff induction where every model includes random noise. However, it can be argued convincingly that this is the correct way to incorporate the possibility of e.g. a black box for the halting problem into your world view, while it is not at all clear why Solomonoff induction is reasonable if the universe isn’t computable.

In general you get theoretical guarantees whenever you are trying to solve a problem which doesn’t have too much state—ie, in which it is very difficult to pursue a strategy so bad that it will ruin your current performance and all future performance. The proof is probably not surprising to anyone here, but it is worth pointing out that if you use a multiplicative update rule you get particularly fast convergence to a good prediction strategy (this is an infinite analog of multiplicative weights).

There is some question about why we restrict to computable strategies when implementing our induction procedure is already uncomputable. I don’t really want to include a discussion of this here, but if we are going to think about induction as a useful tool to apply in reality then we should probably adopt a view of it which doesn’t involve us solving the halting problem (for example, as a framework for evaluating suggestions as to what we should believe etc.). In this case it hopefully makes sense to restrict to computable strategies.

paulfchristiano 16 Oct 2010 19:05 UTC
1 point
in reply to: cousin_it’s comment on: Mixed strategy Nash equilibrium
If I was a seller, I would never use what you term the “best possible mechanism”. This notion of mechanism design, and more generally of rational play, is certainly interesting mathematics, but in practice it often leads to mechanisms that consistently perform very badly (sometimes it gives good mechanisms, but that is no thanks to the formalism). I don’t think the revelation principle really has much to say about practical mechanisms. I’m not arguing that mechanism design shouldn’t be approached theoretically; I’m just saying that you might use a different model if you re-examined the maxim “rational play ends at a Nash equilibrium” and its justification.

paulfchristiano 16 Oct 2010 19:18 UTC
12 points
on: Mixed strategy Nash equilibrium
The distribution of sword/armor choices in an MMO will not be the Nash equilibrium with overwhelming probability. In fact, it probably won’t be anywhere close if the choice is at all complicated. If you are playing an MMO with random pvp, populated by people the people who play MMOs, and you choose the Nash equilibrium, you will probably do worse than me.

The point of the game isn’t to do as well as possible against the worst possible distribution over opposing strategies. The goal is to do as well as possible over the actual distribution of strategies your opponents play. Reasoning about the distribution of strategies your opponents play is the correct approach to this problem, not reasoning about the Nash equilibrium. The two notions coincide in a certain rather rare limit, which certainly doesn’t apply to any game played between anonymous people on the internet.

Of course, this doesn’t lead to a very rigorously defined problem. But saying that someone who talks about what their opponents are likely to do is “wrong” is itself quite wrong. If 90% of American generals in the past have chosen to attack West, it is probably wrong to defend East with the “optimal” probability. You should not believe more about Nash equilibrium than is actually true.

paulfchristiano 17 Oct 2010 3:59 UTC
4 points
in reply to: noitanigami’s comment on: Mixed strategy Nash equilibrium
For example, the Abreu-Matsushima mechanism implements essentially any implementable property at the unique rationalizable Nash equilibrium (a much stronger guarantee than anything preserved by the revelation principle). If you actually use the Abreu-Matsushima mechanism you will find that it basically never works unless the players want it to work (and often, not even then) as has been verified empirically.

The VCG mechanism maximizes social welfare whenever the players play undominated strategies. In practice, the possibility of even very weak collusion between even two players destroys this guarantee.

In general, the claim that rational players will always choose a Nash equilibrium has little empirical support, and in fact doesn’t have very good theoretical support either outside of two-player zero sum games (of course in the OP the game is two-player and zero sum; there my complaint is that common knowledge of rationality is a bad assumption).

paulfchristiano 17 Oct 2010 23:34 UTC
0 points
in reply to: Perplexed’s comment on: Mixed strategy Nash equilibrium
By performs badly, I meant that it fails to exhibit the properties its designers imagined, or “proved.” For example, if the designers prove that this mechanism generates the maximum possible revenue and the mechanism ends up generating no revenue when deployed in practice, I would say it performs badly. Similarly, if the mechanism is intended to maximize the social welfare but then selects a pareto inefficient outcome, I would say that it performs badly.

When I say that rational play may not “end” at a Nash equilibrium, I mean that when rational players (fully aware of each others’ rationality, etc.) sit down to play a game, we should not be too confident that they will play a Nash equilibrium. I think my objection to your reasoning is that the players are not sequentially given opportunities to deviate; they choose a strategy and then play it. That is the definition of a strategy; if you are allowed to change your strategy iteratively then you are playing a new game, in which the strategy set has simply been enlarged and to which a similar criticism applies. Here is an example which at least calls into doubt the normal justification.

Suppose that a mechanism with two Nash equilibria, A and B, is deployed commonly at auctions all around the world. Because the mechanism was carefully designed, the goods are allocated efficiently at both equilibria. In Italy, everyone plays equilibrium A. Knowing this, and aware of the rationality of the average Italian, all Italians participating at auctions in Italy select equilibrium A. In America, everyone plays equilibrium B. Knowing this, and aware of the rationality of the average American, all Americans participating at auctions in America select equilibrium B. Now a poor American tourist in Italy participates in an auction, and (ignorant of the world as Americans are) he tries to play equilibrium B. Consequently, the auction fails to allocate goods efficiently—the mechanism made no guarantees about what happened when some individuals played from one equilibrium and others played from a different equilibrium. The American’s failure cannot be attributed to some failure of rationality; after playing, the Italians might also all wish that they had changed their strategy. This is also a real problem for mechanisms; there are certain classes of problems and mechanisms for which you can prove that this sort of thing will always be possible. You can try, as the mechanism designer, to suggest an equilibrium A to the players. But if one equilibrium is better for some players and worse for others, why should the players automatically accept your proposed equilibrium? If they all choose to play another Nash equilibrium B which is better for all of them, are they behaving irrationally?

(Of course this example does not apply to dominant strategy truthful mechanisms.)

paulfchristiano 18 Oct 2010 0:14 UTC
4 points
in reply to: Meni_Rosenfeld’s comment on: Mixed strategy Nash equilibrium
I didn’t really mean to insult your post (although I apparently did :). I was probably just as surprised as you at many of the comments in that thread. I agree that you should understand NE if you want to say anything useful about games (and that they are basically the complete story for two-player zero sum games from a theoretical perspective). The one thing I object to is the sentiment that “if you don’t know exactly what is going on, you should just play NE.” After re-reading your latest post this a bit of a straw man. I agree totally that if you know nothing else then you have nothing to do but play the NE, which is all that you actually said. However, you can put any game that you are faced with into the reference class of “games humans play,” and so I don’t think this fact is very relevant most of the time. In particular, if the question was “what should you do in an MMO with these properties” then there are many acceptable answers other than play the NE. It may or may not be the case that anyone in the thread in question actually gave one.

In particular, because I can put all games I have ever played into the reference class “games that I have ever played,” I can apply, over the course of my entire life, an online learning algorithm which will allow me to significantly outperform the Nash equilibrium. In practice, I can do better in many games than is possible against perfectly rational opponents.

paulfchristiano 18 Oct 2010 1:46 UTC
2 points
in reply to: Perplexed’s comment on: Mixed strategy Nash equilibrium
To clarify: Abreu-Matsushima fails in practice, regardless of whether there is collusion (and certainly it fails entirely if there is a coalition of even two players). VCG is dominant strategy truthful, but fails in the presence of even two colluding players. I agree that VCG is extremely interesting, but I also think that you should not consider the problem solved once you know VCG. Also, there are mechanisms which do much better than competitive markets can hope to. The question now is how well a benevolent dictator can allocate goods (or whatever you are trying to do).

I agree that my example is not a Nash equilibrium. The point was that rational players may not play a Nash equilibrium. If your notion of a reasonable solution is “it works at equilibria” then sure, this isn’t a counterexample. But presumably the minimal thing you would want is “it works when the players are all perfectly rational and don’t conclude” which this example shows isn’t even satisfied if there are multiple Nash equilibria.

Most mechanisms don’t have unique Nash equilibrium. The revelation principle also doesn’t preserve the uniqueness of a Nash equilibrium, if you happened to have one at the beginning.

paulfchristiano 5 Nov 2010 19:02 UTC
6 points
in reply to: JoshuaZ’s comment on: The Curve of Capability
If you believe that FOOM is comparably probable to P = NP, I think you should be breathing pretty easily. Based on purely mathematical arguments it would be extraordinarily surprising if P = NP (and even more surprising if SAT had a solution with low degree polynomial running time), but even setting this aside, if there were a fast (say, quadratic) algorithm for SAT, then it would probably allow a smart human to go FOOM within a few weeks. Your concern at this point isn’t really about an AI, its just that a universe where worst-case SAT can be practically solved is absolutely unlike the universe we are used to. If anyone has doubts about this assertion I would be happy to argue, although I think its an argument that as been made on the internet before.

So I guess maybe you shouldn’t breathe easy, but you should at least have a different set of worries.

In reality I would bet my life against a dollar on the assertion P != NP, but I don’t think this makes it difficult for FOOM to occur. I don’t think the possibility of a FOOMing AI existing on modern computers is even worth debating, its just how likely humans are to stumble upon it. If anyone wants to challenge the assertion that a FOOMing AI can exist, as a statement about the natural world rather than a statement about human capabilities, I would be happy to argue with that as well.

As an aside, it seems likely that a reasonably powerful AI would be able to build a quantum computer good enough to break most encryption used in practice in the near future. I don’t think this is really a serious issue, since breaking RSA seems like about the least threatening thing a reasonably intelligent agent could do to our world as it stands right now.

paulfchristiano 6 Nov 2010 2:08 UTC
2 points
in reply to: JoshuaZ’s comment on: The Curve of Capability
We have very different beliefs about P != NP. I would be willing to make the following wager, if it could be suitably enforced with sufficiently low overhead. If a proof that P != NP reaches general acceptance, you will pay me $10000 with probability 1/1000000 (expectation $.01). If an algorithm provably solves 3SAT on n variables in time O(n^4) or less, I will pay you $1000000.

This bet is particularly attractive to me, because if such a fast algorithm for SAT appears I will probably cease to care about the million dollars. My actual probability estimate is somewhere in the ballpark of 10^-6, though its hard to be precise about probabilities so small.

Perhaps it was not clear what I meant by an individual going FOOM, which is fair since it was a bit of a misuse. I mean just that an individual with access to such an algorithm could quickly amplify their own power and then exert a dominant influence on human society. I don’t imagine a human will physically alter themselves. It might be entertaining to develop and write up a plan of attack for this contingency. I think the step I assume is possible that you don’t is the use of a SAT solver to search for a compact program whose behavior satisfies some desired property, which can be used to better leverage your SAT solver.

A similar thought experiment my friends and I have occasionally contemplated is: given a machine which can run any C program in exactly 1 second (or report an infinite loop), how many seconds would it take you to ?

paulfchristiano 6 Nov 2010 15:20 UTC
3 points
in reply to: rwallace’s comment on: The Curve of Capability
There are no known strong results concerning the relationship between BQP (roughly the analogue of P for quantum computers) and NP. There is strong consensus that BQP does not contain NP, but it is not as strong as the overwhelming consensus that P != NP.

paulfchristiano 7 Nov 2010 16:47 UTC
−2 points
in reply to: JoshuaZ’s comment on: The Curve of Capability
Determining whether white or black wins at GO is certainly not in P (in fact certainly not in NP, I think, if the game can be exponentially long), but in the real world you don’t care whether white or black wins. You care about whether you win in the particular game of Go you are playing, which is in NP (although with bad constants, since you have to simulate whoever you are playing against).

There is a compelling argument to be made that any problem you care about is in NP, although in general the constants will be impractical in the same sense that building a computer to simulate the universe is impractical, even if the problem is in P. In fact, this question doesn’t really matter, because P is not actually the class of problems which can be solved in practice. It is a convenient approximation which allows us to state some theorems we have a chance of proving in the next century.

I think the existence of computational lower bounds is clearly of extreme importance to anything clever enough to discover optimal algorithms (and probably also to humans in the very long term for similar reasons). P != NP is basically the crudest such question, and even though I am fairly certain I know which way that question goes the probability of an AI fooming depends on much subtler problems which I can’t even begin to understand. In fact, basically the only reason I personally am interested in the P vs NP question is because I think it involves techniques which will eventually help us address these more difficult problems.

paulfchristiano 9 Nov 2010 0:54 UTC
−4 points
in reply to: JoshuaZ’s comment on: The Curve of Capability
Suppose I want to choose how to behave to achieve some goal. Either what genes to put in a cell I am growing or what moves to play in a game of go or etc. Presumably I can determine whether any fixed prescription will cause me to attain my goal—I can simulate the universe and check the outcome at the end. Thus checking whether a particular sequence of actions (or a particular design, strategy, etc.) has the desired property is in P. Thus finding one with the desired property is in NP. The same applies to determining how to build a cell with desired properties, or how to beat the world’s best go player, etc. None of this is to say that P = NP sheds light on how easy these questions actually are, but P = NP is the normal theoretical interpretation, and in fact the only theoretical interpretation that makes sense if you are going to stick with the position that P is precisely the class of problems that an AI can solve.

paulfchristiano 9 Nov 2010 17:27 UTC
21 points
on: A note on the description complexity of physical theories
MWI and Copenhagen do not make the same predictions in all cases, just in testable ones. There is a simple program that makes the same predictions as MWI in all cases. There appears to be no comparably simple program that makes the same predictions as Copenhagen in all cases. So, if you gave me some complicated test which could not be carried out today, but on which the predictions of MWI and Copenhagen differed, and asked me to make a prediction about what would happen if the experiment was somehow run (it seems likely that such experiments will be possible at some point in the extremely distant future) I would predict that MWI will be correct with overwhelming probability. I agree that if some other “more complicated” theory made the same predictions as MWI in every case, then K-complexity would not give good grounds to decide between them.

I guess the fundamental disagreement is that you think MWI and Copenhagen are the same theory because discriminating between them is right now far out of reach. But I think the existence of any situation where they make different hypotheses is precisely sufficient to consider them different theories. I don’t know why “testable” (meaning testable in practice, not in theory) was thrown in at the last minute, because it does not seem to appear anywhere in the rest of the post.

If instead you are asserting that MWI and Copenhagen make the same theoretically testable predictions, then I disagree as a matter of fact. MWI asserts that interference should be able to occur on arbitrary scales, in particular on the scale of an entire planet or galaxy (even though such interference is spectacularly difficult to engineer and/or will have a very small effect on probability amplitudes), while Copenhagen seems to imply that it cannot occur on any scale larger than a human observer.

I wouldn’t be surprised if I am wrong on that question of fact, and it would certainly be good for me to fix my error now if I am.
What links here?

paulfchristiano 9 Nov 2010 17:42 UTC
4 points
in reply to: [deleted]’s comment on: A note on the description complexity of physical theories
The length of the program description is not really the measure of how easy it is to make a correct prediction. In fact, the shortest program for predicting is almost never the one you should use to make predictions in practice, precisely because it is normally quite slow. It is also very rarely the program which is easiest to manipulate mentally, since short programs tend to be very hard for humans to reason about.

paulfchristiano 9 Nov 2010 22:44 UTC
7 points
in reply to: Perplexed’s comment on: A note on the description complexity of physical theories
I say “seems to imply” because its not really clear what Copenhagen does or does not imply, because it doesn’t really make predictions in every circumstance.

Copenhagen implies that when we make a measurement, the value we measure is in fact the one true value. In particular, there are not other worlds where the measurement returned other values. This is precisely the thing that distinguishes it from many worlds, which suggests that there are other worlds where the measurement returned other values.

By accepting that only one value of the measurement actually happens, you reject the possibility of one human civilization, where one thing was measured, interfering with a different human civilization, where a different thing was measured (because you don’t believe the other human civilization even exists).