query

Karma: 287

query 3 Apr 2015 21:11 UTC
4 points
in reply to: lerjj’s comment on: The True Prisoner’s Dilemma
I think what might be confusing is that your decision depends on what you know about the paperclip maximizer. When I imagine myself in this situation, I imagine wanting to say that I know “nothing”. The trick is, if you want to go a step more formal than going with your gut, you have to say what your model of knowing “nothing” is here.

If you know (with high enough probability), for instance, that there is no constraint either causal or logical between your decision and Clippy’s, and that you will not play an iterated game, and that there are no secondary effects, then I think D is indeed the correct choice.

If you know that you and Clippy are both well-modeled by instances of “rational agents of type X” who have a logical constraint between your decisions so that you will both decide the same thing (with high enough probability), then C is the correct choice. You might have strong reasons to think that almost all agents capable of paperclip maximizing at the level of Clippy fall into this group, so that you choose C.

(And more options than those two.)

The way I’d model knowing nothing in the scenario in my head would be something like the first option, so I’d choose D, but maybe there’s other information you can get that suggests that Clippy will mirror you, so that you should choose C.

It does seem like implied folk-lore that “rational agents cooperate”, and it certainly seems true for humans in most circumstances, or formally in some circumstances where you have knowledge about the other agent. But I don’t think it should be true in principal that “optimization processes of high power will, with high probability, mirror decisions in the one-shot prisoner’s dilemma”; I imagine you’d have to put a lot more conditions on it. I’d be very interested to know otherwise.

query 4 Apr 2015 3:57 UTC
3 points
in reply to: lerjj’s comment on: The True Prisoner’s Dilemma
I agree it is better if both agents cooperate rather than both defect, and that it is rational to choose (C,C) over (D,D) if you can (as in the TDT example of an agent playing against itself). However, depending on how Clippy is built, you may not have that choice; the counter-factual may be (D,D) or (C,D) [win for Clippy].

I think “Clippy is a rational agent” is the phrase where the details lie. What type of rational agent, and what do you two know about each other? If you ever meet a powerful paperclip maximizer, say “he’s a rational agent like me”, and press C, how surprised would you be if it presses D?

query 19 May 2015 16:42 UTC
8 points
in reply to: Gunnar_Zarncke’s comment on: We Should Introduce Ourselves Differently
Several people I’ve tried to introduce the site to have been turned off at the title: they interpreted “Less Wrong” to mean something akin to “We are a community of arrogant internet atheists who are Less Wrong than all of those stupid people”.

I believe the potshots at religion in major articles also fed into this view for at least one person; he was interested in some of the articles, but used those potshots and selected comments to argue that the community was terrible. He now is interested in many of the ideas, but is semi-actively opposed to the community (e.g. he’ll say derogatory things about it if it’s brought up.)

In it’s current form, I think the site is dangerous to share with anyone who is (a) religious or sensitive to poking fun at religion, or (b) cynical with regard to “atheist communities” which is what LessWrong is at risk of pattern matching as. I’d love to have a good resource for sharing rationality with those friends, although for some of them the well of “rationality” is poisoned by their negative associations with Less Wrong.

query 22 May 2015 15:40 UTC
8 points
0
in reply to: Mark_Friedenbach’s comment on: Leaving LessWrong for a more rational life
What type of evidence would make you think it’s more likely that a self-modifying AGI could “break out of the box”?

I want to understand your resistance to thought-experiments. Are all thought-experiments detached from reality in your book? Are all analogies detached from reality? Would you ever feel like you understood something better and thus change your views because of an analogy or story? How could something like http://lesswrong.com/lw/qk/that_alien_message/ be different in a way that you would theoretically find persuasive?

Perhaps you’re saying that people’s confidence is too strong just based on analogy?

I was going to try and address your comment directly, but thought it’d be a good idea to sort that out first, because of course there are no studies of how AGI behave.

query 23 May 2015 15:05 UTC
3 points
in reply to: Mark_Friedenbach’s comment on: Leaving LessWrong for a more rational life
Ok, I’m still confused as to many of my questions, but let me see if this bit sounds right: the only parameter via which something like “That Alien Message” could become more persuasive to you is by being less fictional. Fictional accounts of anything will NEVER cause you to update your beliefs. Does that sound right?

If that’s right, then I want to suggest why such things should sometimes be persuasive. A perfect Bayesian reasoner with finite computational ability operates not just with uncertainty about the outside world, but also with logical uncertainty as to consequences of their beliefs. So as humans, we operate with at least that difficulty when dealing with our own beliefs. In practice we deal with much much much worse.

I believe the correct form of the deduction your trying to make is “don’t add a fictional story to the reference class of a real analogue for purposes of figuring your beliefs”, and I agree. However, there are other ways a fictional story can be persuasive and should (in my view) cause you to update your beliefs:
- It illustrates a new correct deduction which you weren’t aware of before, whose consequences you then begin working out.
- It reminds you of a real experience you’ve had, which was not present in your mind before, whose existence then figures into your reference classes.
- It changes your emotional attitude toward something, indirectly changing your beliefs by causing you to reflect on that thing differently in the future.
Some of these are subject to biases which would need correcting to move toward better reasoning, but I perceive you as claiming that these should have no impact, ever. Am I interpreting that correctly(I’m going to guess that I’m not somewhere), and if so why do you think that?

query 23 May 2015 15:47 UTC
2 points
in reply to: Mark_Friedenbach’s comment on: Leaving LessWrong for a more rational life
Basically agreed on paragraph 1, but I do want to suggest that then we not say “I will never update on fictional stories.” Taken naively, you then might avoid fictional stories because they’re useless (“I never update on them!”), when of course they might be super useful if they cause you to pull up relevant experiences quite often.

I’ll make an example of how “That Alien Message” could do for me what I illustrated in my 1st bullet point. I think, “Oh, it seems very unlikely that an AI could break out of a box, you just have this shutoff switch and watch it closely and …”. Then That Alien Message suggests the thought experiment of “instead of generalizing over all AI, instead imagine just a highly specific type of AI that may or may not reasonably come to exist: a bunch of really sped up, smart humans.” Then it sparks my thinking for what a bunch of really sped up, smart humans could accomplish with even narrow channels of communication. Then I think “actually, though I’ve seen no new instances of the AI reference class in reality, I now reason differently about how a possible AI could behave since that class (of possibilities) includes the the thing I just thought of. Until I get a lot more information about how that class behaves in reality, I’m going to be a lot more cautious.”

By picking out a specific possible example, it illustrates that my thinking around the possible AI reference class wasn’t expansive enough. This could help break through, for example, an accessibility heuristic: when I think of a random AI, I think of my very concrete vision of how such an AI would behave, instead of really trying to think about what could lie in that huge space.

Perhaps you are already appropriately cautious, and this story sparks/sparked no new thoughts in you, or you have a good reason to believe that communities of sped up humans or anything at least as powerful are excluded from the reference space, or the reference space you care about is narrower, but it seemed like you were making a stronger statement that such stories will never have any impact on you.

query 23 May 2015 16:33 UTC
0 points
in reply to: ChristianKl’s comment on: Leaving LessWrong for a more rational life
This seems promising!

Studies, no. I can’t imagine studies existing today that resolve this, which of course is a huge failure of imagination: that’s a really good thing to think about. For anything high enough level, I expect to run into problems with “do better”, such as “do better at predicting the behavior of AGI” being an accessible category. I would be very excited if there were nearby categories that we could get our hands on though; I expect this is similar to the problem of developing and testing a notion of “Rationality quotient” and proving it’s effectiveness.

I’m not sure where you’re referring to with Mark rejecting cleverness as a way of gathering knowledge, but I think we may be arguing about what the human equivalent of logical uncertainty looks like? What’s the difference in this case between “cleverness” and “thinking”? (Also could you point me to the place you were talking about?)

I guess I usually think of cleverness with the negative connotation being “thinking in too much detail with a blind spot”. So could you say which part of my response you think is bad thinking, or what you instead mean by cleverness?

query 23 May 2015 18:32 UTC
1 point
in reply to: ChristianKl’s comment on: Leaving LessWrong for a more rational life
I see. I do think you can update based on thinking; the human analogue of the logical uncertainty I was talking about. As an aspiring mathematician, this is what I think the practice of mathematics looks, for instance.

I understand the objection that this process may fail in real life, or lead to worse outcomes, since our models aren’t purely formal and our reasoning isn’t either purely deductive or optimally Bayesian. It looks like some others have made some great comments to this article also discussing that.

I’m just confused about which thinking you’re considering bad. I’m sure I’m not understanding, because it sounds to me like “the thinking which is thinking, and not direct empirical observation.” There’s got to be some level above direct empirical observation, or you’re just an observational rock. The internal process you have which at any level approximates Bayesian reasoning is a combination of your unconscious processing and your conscious thinking.

I’m used to people picking apart arguments. I’m used to heuristics that say, “hey you’ve gone too far with abstract thinking here, and here’s an empirical way to settle it, or here’s an argument for why your abstract thinking has gone too far and you should wait for empirical evidence or do X to seek some out” But I’m not used to “your mistake was abstract thinking at all; you can do nothing but empirically observe to gain a new state of understanding”, at least with regard to things like this. I feel like I’m caricaturing, but there’s a big blank when I try to figure out what else is being said.

query 23 May 2015 19:55 UTC
1 point
in reply to: ChristianKl’s comment on: Leaving LessWrong for a more rational life
Ok, I think see the core of what you’re talking about, especially “Whether or not people are simply blinded by fiction or whether it helps reasoning is an empirical question.” This sounds like an outside view versus inside view distinction: I’ve been focused on “What should my inside view look like” and using outside view tools to modify that when possible (such as knowledge of a bias from decision science.) I think you and maybe Mark are trying to say “the inside view is useless or counter-productive here; only the outside view will be of any use” so that in the absence of outside view evidence, we should simply not attempt to reason further unless it’s a super-clear case, like Mark illustrates in his other comment.

My intuition is that this is incorrect, but it reminds me of the Hanson-Yudkowsky debates on outside vs. weak inside view, and I think I don’t have a strong enough grasp to clarify my intuition sufficiently right now. I’m going to try and pay serious attention to this issue in the future though, and would appreciate if you have any references that you think might clarify.

query 24 May 2015 0:36 UTC
0 points
in reply to: Jiro’s comment on: Leaving LessWrong for a more rational life
Agreed; you’d have to figure all of that out separately. For what it’s worth, given the selection of fictional stories I’m usually exposed to and decide to read, I think they’re generally positive value (though probably not the best in terms of opportunity cost.)

query 24 May 2015 16:50 UTC
1 point
in reply to: Nanashi’s comment on: How my social skills went from horrible to mediocre
Thanks for this comment; I don’t know about social interaction being really fucking easy, but I agree wholeheartedly with your recommendations for how to see other people.

Seeing this modeled in the people around me has had a huge positive impact in my life. I’m surrounded by people who, while not skilled at rationality, continually look for the potential and value in other people and openly strategize about how to nurture that viewpoint. (Of course, it would be even better if they were also skilled rationalists, and I’m trying to add that component into my community life as well.)

I’m not sure how many, but I think most people around here are “pro-human” in the sense of thinking every persons life, happiness, and fulfillment is a value to be ultimately pursued (though I make no argument about the opportunity cost of doing so in general at the current time.) Accepting this on an intellectual level is different than emotionally integrating it, and the emotional integration of this has been really fulfilling for me, as well as having the positive impacts on social interaction that you mention.

I think the tendency to feel negative toward “baselines” can be seen as an attribution error in light of those values. You may be annoyed/disgusted/confused by the other person’s lack of understanding in the moment, but the cause of your response can be seen as conditional to that situation, and you can remind yourself of all the good things you would wish for this person given the ability to make them happen.

I’d tentatively recommend anyone finding themselves feeling negative toward “baselines” cultivate a group of people around them who take this view, even if they have bad epistemics. I hear that Unitarian Universalists may be good for this, as they’re open to atheists while having some of the same pro-human community values. My recommendation is tentative, since I think other people may respond differently to the trade-off of community epistemics versus nurturing this viewpoint; but if you have people around you who can satisfy both, then spend time with them!

Edit: I think I want to add a warning/disagreement that you don’t want to end up being condescending or fatherly/motherly unless it’s an appropriate relationship for that. This is one of the ways interaction isn’t easy. But I do find it really helpful to cultivate this as an internal viewpoint.

query 27 May 2015 19:36 UTC
2 points
in reply to: Lumifer’s comment on: Six Ways To Get Along With People Who Are Totally Wrong*
I think your critique of this being only for disagreements that don’t matter is too strong, and your examples miss the context of the article.

This is not a suggested resolution procedure for all humans in all states of disagreement; this is a set of techniques for when you already have and want to maintain some level of cooperative relationship with a person, but find yourself in a disagreement over something. Suggestion 5 above is specifically about disengaging from disagreements that “don’t matter”, and the rest are potentially useful even if it’s a disagreement over something important.

query 27 May 2015 19:48 UTC
2 points
in reply to: Lumifer’s comment on: Six Ways To Get Along With People Who Are Totally Wrong*
I don’t know what you mean, but I think I see a lot of people “being polite” but failing at one of these when it would be really useful for them.

For example, you can be polite while internally becoming more suspicious and angry at the other person (#3 and #4) which starts coming out in body language and the direction of conversation. Eventually you politely end the conversation in a bad mood and thinking the other person is a jerk, when you could’ve accomplished a lot more with a different internal response.

query 27 May 2015 19:57 UTC
6 points
in reply to: lululu’s comment on: Six Ways To Get Along With People Who Are Totally Wrong*
Seconded! Another phrase (whose delivery might be hard to convey in text) is “Look, I dunno, but anyways...”

Maybe the big idea is to come across as not expressing much interest in the claim, instead of opposing the claim? I think most people are happy to move on with the conversation when they get a “move on” signal, and we exchange these signals all the time.

I also like that this is an honest way to think about: I really am not interested in what I expect will happen with that conversation (even if I am interested in the details of countering your claim.)

query 27 May 2015 20:12 UTC
0 points
in reply to: Lumifer’s comment on: Six Ways To Get Along With People Who Are Totally Wrong*
Agreed on the 2nd paragraph.

Optimally, you’d be have an understanding of the options available, how you work internally, and how other people respond so you could choose the appropriate level of anger, etc. Thus it’s better to explore suggestions and see how they work than to naively apply them in all situations.

query 29 May 2015 17:38 UTC
2 points
in reply to: [deleted]’s comment on: Six Ways To Get Along With People Who Are Totally Wrong*
I’ve enlarged my social circles, or the set of circles I can comfortably move in, and didn’t end up on that model. I think I originally felt that way a lot, and I worked on the “feeling like doing a dramatic facepalm” by reflecting on it in light of my values. When dramatic face palms aren’t going to accomplish anything good, I examine why I have that feeling and usually I find out it’s because my political brain is engaged even when this isn’t a situation where I’ll get good outcomes from political-style conversation. You can potentially change your feelings so that other responses that you value become natural.

A warning though, it took me a long time to learn how to do this in a manner that didn’t make me feel conflicted. There’s always the danger of “I was nice to the person even after they said X” priming you in a bad way. You could also do even worse than before in terms of your values, simply because you’re inexperienced with the non-facepalm response and do it badly. I think it was worth it for me, but depending on your situation it might be dangerous to mess with.

query 5 Jun 2015 16:16 UTC
0 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
A mathematical model of what this might look like: you might have a candidate class of formal models U that you think of as “all GAI” such that you know of no “reasonably computable”(which you might hope to define) member of the class (corresponding to an implementable GAI). Maybe you can find a subclass F in U that you think models Friendly AI. You can reason about these classes without knowing any examples of reasonably computable members of either. Perhaps you could even give an algorithm for taking an arbitrary example in U and transforming it via reasonable computation into an example of F. Then, once you actually construct an arbitrary GAI, you already know how to transform it into an FAI.

So the problem may be factorable such that you can solve a later part before solving the first part.

So, I’d agree it might be hard to understand F without understanding U as a class of objects. And lets leave aside how you would find and become certain of such definitions. If you could, though, you might hope that you can define them and work with them without ever constructing an example. Patterns not far off from this occur in mathematical practice, for example families of graphs with certain properties known to exist via probabilistic methods, but with no constructed examples.

Does that help, or did I misunderstand somewhere?

(edit: I don’t claim an eventual solution would fit the above description, this is just I hope a sufficient example that such things are mathematically possible)

query 5 Jun 2015 16:48 UTC
0 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
Oh yes, it sounds like I did misunderstand you. I thought you were saying you didn’t understand how such a thing could happen in principle, not that you were skeptical of the currently popular models. The classes U and F above, should something like that ever come to pass, need not be AIXI-like (nor need they involve utility functions).

I think I’m hearing that you’re very skeptical about the validity of current toy mathematical models. I think it’s common for people to motte and bailey between the mathematics and the phenomena they’re hoping to model, and it’s an easy mistake for most people to make. In a good discussion, you should separate out the “math wank” (which I like to just call math) from the transfer of that wank to reality that you hope to model.

Sometimes toy models are helpful and some times they are distractions that lead nowhere or embody a mistaken preconception. I see you as claiming these models are distractions, not that no model is possible. Accurate?

query 5 Jun 2015 17:04 UTC
0 points
in reply to: query’s comment on: FAI Research Constraints and AGI Side Effects
I offered the transform as an example how things can mathematically factor, so like I said, that may not be what the solution looks like. My feeling is that it’s too soon to throw out anything that might look like that pattern though.

query 5 Jun 2015 17:15 UTC
0 points
in reply to: AeroRails’s comment on: FAI Research Constraints and AGI Side Effects
Formally, you don’t. Informally, you might try approximate definitions and see how they fail to capture elements of reality, or you might try and find analogies to other situations that have been modeled well and try to capture similar structure. Mathematicians et al usually don’t start new fields of inquiry from a set of definitions, they start from an intuition grounded in reality and previously discovered mathematics and iterate until the field takes shape. Although I’m not a physicist, the possibly incorrect story I’ve heard is that Feynman path integrals are a great example of this.