Open Thread: June 2009
I provide our monthly place to discuss Less Wrong topics that have not appeared in recent posts. Work your brain and gain prestige by doing so in E-prime (or not, as you please).
I provide our monthly place to discuss Less Wrong topics that have not appeared in recent posts. Work your brain and gain prestige by doing so in E-prime (or not, as you please).
One of the things I worry about the LW community is that we’ll get too enamored of signaling rationality and become afraid to bounce ideas off each other that we’re not quite sure are rigorous enough. This thought is brought on by the seemingly small number of top level posts that people try to make (given how many members we have). Are people refraining from posting after they see that one such thread gets voted down? I know how discouraging that can be, and it seems to me that it would be worse for the budding rationalist who is unsure of themselves. To some extent these monthly open threads help, but even in these comments seem pretty conservative. What say you LW? Are we capable of entertaining ideas without adopting them or is the risk of diluting the signal to noise ratio too high?
edit: it occurs to me that maybe I’d like to see another tab up top called “passing thoughts” or some such. The S:N ratio of the regular posts could be kept up and shorter posts could go there. We could all agree simply to not hold it against each other if we make a gaffe in these shorter posts (obvious mistake). I think this would be valuable because even if an idea is flawed it can generate great discussion (one of the reasons I enjoy hacker news). As an added bonus, this could serve as a space for “Ask LW” posts without disrupting anything.
Having posted five top-level posts so far and having two half-written ones pending, I’d estimate that writing a good post takes me about 20 times as much time and effort as writing a long comment. Many people simply can’t commit that much time, or can’t maintain focus for that long. I don’t think fear of rejection is the problem.
I take nazgulnarsil’s comment as suggesting that there may be value to more people writing posts that aren’t necessarily “good”… in which case that sort of rejection may not be optimal.
I like the “passing thoughts” segment idea. “Ask LW” sounds great too—a community advice column without the Yahoo! Answers nonsensical noise could be a great draw for new contributors, too.
Apropos my comment here, what do you guys think of making The Simple Math of Everything a reality?
The LW community probably has a diverse enough group of scholars to cover most of the major fields, and if not, we all have contacts. Splitting it up into sections for different individuals to write would make the project much easier to complete, provided someone is coordinating everything. What do you guys think?
Write a call for top-level-post essays as a top-level post, stating clearly what kind of articles you see in such a collection, and see if enough accumulates for an edited volume (I think it’s unrealistic at this point).
I have always liked this idea. Could it be done on the Less Wrong wiki?
Instead of writing a collection of articles, could we just put together a collection of links? It couldn’t be sold, but it would be ready a lot sooner.
Something like this?
I like this idea, actually. I’m sure there’s already a lot of great material out there, we just need to put it together.
Agreed as well, I always thought this was a great idea.
An edited volume might be overly ambitious, but I think a wikibook could be a feasible.
Another way to test the waters is to write a top level post calling for suggestions on topics or format. If there’s enough interest, put out a call for essays accompanied by a brief style guide.
We’ll have to make sure to ask Kalid Azad to participate.
Speaking of improving the LW website, I’d like to see little homepages for each user to provide a self-introduction, which I know has been suggested before, but in particular I’d like to see an implementation of userboxes as Wikipedia does (see my homepage there). This would allow people a standardized and easy-to-reference way to show their opinions on various common issues on the site.
Here’s an open topic: why did Overcoming Bias and Less Wrong split?
It certainly looks to me like the technical shift was just an excuse for certain parts of that community to split away.
If this social explanation is true, what implications does this have?
It’s worth noting that LW has a lower barrier to participation and a more community-centric focus, while OB has shifted to be explicitly Hanson’s blog; these are clearly pursuing distinctly different goals. The simplest hypothesis is that Hanson and Yudkowsky disagreed on direction and decided to part ways.
In the About page on OB, Hanson says:
The phrasing here (especially the strongly complimentary reference to Eliezer) is a sort that, in my experience, often indicates a professional disagreement after which collaboration is ended, but with no real hard feelings over it and a desire to signal such to third parties. This suggests to me a lack of ulterior motives.
It does seem to be the case that the readers and commenters of LW and OB have some amount of non-overlap, but I suspect that is mostly incidental.
I have the strong impression that Eliezer and Robin were better together. I’ve not yet felt that either individual blog has the same draw as the old OB.
And why couldn’t that goal have been pursued within a LW-powered OB? It’s not as if there is all that much competing content, and surely we can expect further technical additions like tags. Entirely separate sites, with all the implied overhead and loss of network effects, to pursue separate goals only makes sense if the goals are contradictory.
Robin said he was apprehensive about using new software to power his blog. It’s almost as if Robin had a bad experience with over-ambitious hypertext software, at some point in his murky past...
Was that a Xanadu reference?
It was; well done.
I need relationship advice and I trust the wisdom and honesty of this community more than most of my friends. I created a new account to ask this question.
I’m with an incredibly compassionate, creative woman. She excels at her job, which is a “helping profession,” and one which I believe improves social welfare far more than most. The sex is outstanding.
But she loves magical thinking, she is somewhat averse to expected-utility calculations, my atheism, etc. She is, by her own admission, subject to strong swings of emotion and at greater than average risk of longer-lasting depression. We love each other but are scared that our differences may be too great.
How would you personally feel about a relationship like this? How should I go about deciding whether to continue this?
Added: We have been together more than 6 months. She has learned a decent amount about my way of thinking, but I have not pushed it on her. I frequently mention how great rationality is (but also mock myself to make sure we’re all having fun).
I wish I had confidence that trying to convert her to my way of thinking would have net-benefits for her and for the world long-term, but I don’t. Not that I’m convinced trying to convert her is a bad idea on utilitarian grounds either, it just seems risky.
With science!
Specifically, the science of John Gottman. Short version: irreconcilable differences of viewpoint are not an intrinsic bar to a long-lasting relationship. The most potent relationship poison is contempt.
I was pretty amazed the first time I saw this, and even though I’m pretty confident in my relationship, it seems like this test would still be worth quite a bit.
Does anybody still run these microexpression tests or would you have to convince the researchers to get back into it for a one off thing?
When I looked, I couldn’t find anyone offering to run this test. I hadn’t gone far enough as to contact the original researchers and see if they’d be willing to do it, but if there are other people interested, it might be worth a shot.
I assume you’re talking about the test where the researchers infer the likely fate of the marriage by tracking the “trajectory” of a 20-minute-or-so conversation. My impression is that proper interpretation of the test requires training.
Gottman has written popular relationship advice guides based on his research; I’d recommend that anyone interested in maintaining the health of their current relationship have a look. His advice makes the assumption that certain correlations he’s observed are causative, but that assumption seems reasonable for the most part. Research into the actual causal effect of his suggested interventions is ongoing.
Yes, that is what I was talking about. Proper interpretation may require training, but we know those people exist, at least.
Thanks for the link Cyan.
It is possible to make a relationship work in which each party has a role in the general neighborhood of “the rational one” or “the emotional one”, as long as the relative places and merits of these roles are acknowledged by both parties. Since you say she’s prepared to admit to her mood swings, this may be doable. My proposed checklist:
If she suffers from an extended period of depression, is she prepared to address that (therapy, antidepressants, ice cream & sad movies, whatever she finds works for her), or would she let it greatly interfere with her life and your lives together?
How averse to your atheism is she? How averse to her (presumable) theism are you? Do you have enough else to talk about and enough ability to skirt the topic that it can avoid being a major point of contention between you? If you want kids eventually, can you come to an agreement about how to raise them re: religion?
Does her magical thinking lead her to do anything profoundly instrumentally stupid, or does it mostly just make her sound a little silly occasionally?
Can you respect her, as well as love her, in spite of her failures of rationality? Or would you be hoping in the back of your mind forever and always that she’d eventually wise up and be a more rational version of herself that you could respect?
Are you compatible on other long-term axes? (Financially, politically, life goals, desire to live in a particular location, opinions on family and homemaking, etc.)
If you can give the “right” answer to all of those questions (I think it should be obvious in each case which answer would be best) then go for it and the best of luck to you. If you can’t, you either need to address the situation and fix it, or move on.
Alicorn, Thanks for responding—see my “Added” to the original comment.
She seems to take a fairly reasonable approach to dealing with / working around her emotional issues and tries hard not to let me suffer because of them.
The atheism / theism divide could be much worse. I’m not sure her beliefs even have net-negative consequences. At present, the main issue is that we each have important beliefs that we don’t think we can share. RE: children, we could probably both accept me having the right to be honest about my beliefs but not pushing them, or going into detail unless they really want to hear it or reached 18.
She doesn’t generally do things that are obviously and profoundly instrumentally stupid, but its probably fair to worry about whether she might in the future. She’d need some money to spend on her friends more lavishly than I would, and to give to inefficient charities, but it seems unlikely she’d want to spend more than I could indulge.
Being able to share differing beliefs has more to do with whether you can both remain civil about important things than whether you agree. I regularly and enthusiastically pick apart minute disagreements between myself and my friends, and would feel as though something were lacking if I couldn’t—but we can switch topics from politics to polenta when someone gets fed up and there are no hard feelings. If you can’t do that with your girlfriend, that indicates a deeper-running incompatibility than merely disagreeing on rationality. Even if you agreed on all the big issues, it would be miraculous for you to make it through life without ever arguing, and being able to argue without it having it destroy your relationship is an essential skill.
A big part of the issue is that I’m not sure whether in depth discussions of my views will a) convince her, or b) help her live a good and happy life, or c) the relationship between a) and b).
Regardless, I’ll need to push a little more conversation of LWish topics before doing anything crazy like getting married. She realizes this as well.
Let me explain that sentence a bit more. As you know, preference utilitarianism comes with quite a bit of bullet-swallowing and while I may be less hard core than some, I swallow bullets she seems very hesitant to. Perhaps equally or more importantly, like most people, she doesn’t seem to like to taste the bullets, i.e. ponder uncomfortable thoughts, accept uncertainty, etc. I, on the other hand, seem to take some perverse pleasure in thinking and talking about such topics. From her perspective, I sometimes “analyze things e.g. a poem, a play, the proper emotional response to situation X, to death to the point of being distracted from their inherent value.”
For me, any (serious) talk of “overanalyzing” or “overthinking” things would be a huge red flag. But maybe I’m unusual in that.
Using the word “overanalysis” isn’t always a red flag for unwillingness to analyze where appropriate. Sometimes it just means recognizing that it is not worth six hours of nutrition research and blind taste tests to decide what cereal is the optimal breakfast. In a pinch, you can just grab the generic crisped rice or a prettily-packaged granola and call it good.
Of course. To clarify a bit, it’s obviously possible to give things more thought than they deserve. But someone who habitually makes accusations of overthinking, to my mind, is indicating a contempt for thought itself, which is about the most horrible quality I can think of. (I believe I first came to this conclusion when I read this webcomic, though on looking back at it I’m not sure.)
Some people just aren’t very good at getting right answers through deliberate reasoning, but can get by using implicit reasoning. Combine that with the typical mind fallacy and you get someone who sees “overthinking” everywhere they look. But I think the problem here isn’t so much about these implicit reasoners, but rather about contemptuous people in general, with contemptuous implicit reasoners as a special case.
So the program of understanding each other doesn’t make progress. I agree with Alicorn, it’s essential to establish a mode of communication where you can steadily work on disagreements, with the goal of ultimately resolving them in full. The arguments shouldn’t turn into color politics, polarizing and alienating.
A bit of advice, based on my experience, for a long-term conversion strategy:
Work on understanding your own position better, make sure you know why you believe what you believe before trying to convince another person to change one’s mind. Maybe you are wrong.
Make the mode of interaction and your goals clear when you are arguing, distinguish analysis from social interaction.
Educate the person about fundamentals, thus steadily crafting tools for making deeper arguments in specific discussions.
Prefer shifting the discussion towards education about more general mistake that (might have) contributed to a specific mistake or confusion. In long term, it’s more important than resolving a specific problem, and it’s easier on the other person’s feelings, as you are educating on an abstract theme, rather than attacking a conviction.
Don’t argue the objects of emotional attachment, ever (unless the person is ready for that, at which point you are probably done with bootstrapping). Instead, work on finding an angle of approach (as suggested above, maybe something more fundamental) that allows you to make progress without directly confronting the issue.
Not everyone is going to change, some people are too dim or too shallow or persistently not interested.
Thank you, this sounds like very good advice for how to lead someone down the path.
But given that she is reluctant to go down the path, do I want to lead her down it? She already believes that I can defend my views better than she can her’s. She probably even believes that my views are closer to the truth.
My guess is that she is reluctant to discuss and evaluate the fundamental facts of existence and our values, precisely because she cherishes certain aspects of her current worldview that she correctly believes she is likely to lose. I think its plausible that she’ll end up less happy, and maybe less productive, after hearing about the preference utilitarianism and the opportunity cost of spending $80 to have flowers delivered to a friend (note: I’d never try to stop her from doing it, I’d just like to explain why I’m not going to) or after explaining why the idea that people have souls is incoherent (note: I would never say something that strongly. As you suggest I’d want to build up to it slowly, by asking questions and letting the conclusions fall out of the discussion.)
Religious people report being happier. By many measures they also do more “good works.” I wouldn’t be surprised if the same were true of deontologists vs. consequentialists.
Do I really have reason to believe she’ll benefit from serious detailed discussion of our respective worldviews?
I’d be curious to know what sort of power dynamic you have. My spouse believes I am more rational and intelligent than s/he is—but s/he’s still the one who makes the decisions. I advise—my spouse decides. We both like it that way and we’ve had a successful and happy relationship for more than a decade. Now that I think about it, this is reminiscent of Eliezer’s “Three Worlds Collide”. You want to keep the rationalist cultists around—but you don’t want them in charge :p
In “Three Worlds Collide”, the rational one does have the power to override if necessary, which I think is very important. If you cant agree, you’re doing it wrong- but it still happens occasionally. You’ll get better results if you defer to the person that is more rational under those circumstances.
In general, it seems like the right policy is to let whomever has harder to communicate data decide. This way, the decision maker is as informed as possible.
Actually, in 3WC the Confessor is supposed to be strictly charged with sedating people who depart the bounds of sanity. He goes outside this bound, which is completely against all the rules, and afterward he can no longer be called a Confessor.
I don’t know about the rest of the audience, but I’d really appreciate a worldbuilding writeup, or maybe even just a glossary, explaining the cultural/technological backdrop of 3WC in more detail than the story provides.
There are some worlds for which I have devised huge cultural, technological, and historical backdrops but this is not one of them.
I was referring to the part where the president went crazy and her confessor sedated her “and recommend to the government that they carry out the evacuation without asking further questions of your ship”.
If that doesn’t count as the “power to override if necessary”, then I’m missing a distinction somewhere.
Well, part of the point there was that their President would have been universally recognized by her own society as crazy, at that point, just as if she’d said she was hearing voices from her teapot. In contrast to say our own society where this would be considered perfectly normal madness in a politician. The reason her Confessor then needs to advise the government is that her Confessor was the only one to listen to an extremely classified conversation; in other words she has private info which she must summarize/convey somehow to the government.
Thank you very much for this data point.
It’s impossible to make someone reason if they don’t wish to. It’s impossible to force someone to acknowledge the truth if they don’t want to acknowledge it.
You don’t need to lead her down any path, even if she were willing to follow. She probably already knows what the rational approach is and doesn’t choose to implement it.
In the event that she doesn’t, teach her the method of rationality—not just the result—if she comes to you for help. Don’t, otherwise.
If she’s comfortable with letting you be reasonable, and you’re comfortable with letting her have her magical thinking, I’d say everything is fine.
Part of what motivates this post is that research on happiness suggests that people have a hard time predicting how happy they will be in various possible futures. Gilbert has suggested that introspection is so poor that we better off just asking people in that situation how they feel.
In my case, it was worth it. There may be a stubborn reluctance to “give in” or other lower level things that get in the way of believing the truth, but if at the top level she would really rather believe the truth, you’re probably fine. If you can’t get her to say in full honesty “I would rather believe the truth than made up ‘feel good’ stories”, you’re probably hosed.
My girlfriend started out as a creationist christian and is now pretty much atheist. Overall, she is much more reasonable and can make the right decision on the types of things she previously did not. She seems to be about as happy as before, and when she’s not she can recognize the cause so it’s not nearly as damaging. I’d call that a success so far.
In general, I have a pretty strict policy of calling each other on any BS. I attempt to get the point across in a nice way, but will persist until she understands even if it does make her upset in the short term. The one exception was when I found out that she believed in creationism. That was too big of a bite, so we left it as “agree to disagree”, though I made it obvious that I did not respect her belief. I never made an attempt to deconvert her, but it did happen on its own.
There’s probably a better way to do it that can make the same progress without ever upsetting the person, but in my experience it ended up being worth it to push through anyway.
I’m in a situation which seems sort of the opposite of yours. I’m with a woman, who’s more rational than any other I personally know. But the sex is just not very good, and I find myself getting physically drawn to other women a bit too much. I’ve struggled for weeks, trying to decide whether to continue or not. I’ve tried hard to think what I really want. And I think that if I were sexually satisfied, I would be very happy with the relationship because everything else seems perfect. So, I’m trying to work on that now. I’m paying more attention to being a loving and sensuous partner. Let’s say I’m experimenting on the weak aspects of my relationship.
If I were in your place, I’d take each point of disagreement on its own merit. If it’s decisions where the results can be seen clearly I wouldn’t argue but just politely point to the results. As far as religious beliefs are concerned, the more I think about it the more I feel, that defining myself as an ‘atheist’ is only useful in saying that I don’t believe in God. Beyond that, it doesn’t add anything valuable to my personality. It can’t because it’s a negative definition. So, I would try and deal with specific issues rather than try to convince my partner that theism is wrong. If she believes in magic, playful humour might lighten things up a bit.
I also think it would be useful if you learnt more about her way of thinking, just like she has learnt about yours.
My advice is first, to talk to her a lot about sex and make it clear how important that is to you.
If that doesn’t work, consider asking her for permission to sleep with other women. That option would satisfy me in your situation temporarily, but I’d have to think about whether it would satisfy me longer term.
With respect to making your decision, I would advise you to just spend more time with her. It usually took me about 18 months to figure out how I would finally feel about someone, long-term, that I was initially attracted to. After that period of time, differences were either sources of annoyance (or something more neutral) or sources of contempt. If the latter, for either one of you, you’re “too different”. (Things are complicated by the fact that there’s always a little contempt in a relationship, but the contempt I’m talking about will tend to grow and feel more important than everything else over time.)
In other words, I don’t believe, a priori, that in order to be consistently rational, a rationalist should seek out another rationalist. You’re probably seeking someone that complements you, and pushes you to more fully experience life, and that’s why opposites attract. As long as you have the same core values about what matters to you both. It takes time to determine if you share those.
Regarding the magical thinking that perplexes you: it doesn’t seem to me that most rationalists actually understand what it is that magical thinkers believe. For example, if you think about it in terms of scientifically true and false, you’re probably not thinking about it the right away. Magical thinkers know they’re not making scientific statements. For example, it’s not false to believe you have a soul. Whatever she really means by having a soul, she does have. That’s why rational arguments don’t work. I think it’s a matter of communication: she’s not really expressing what she means by soul, and you’re not really arguing (if you were to) that what she doesn’t have is the soul she’s talking about. Her description of a soul may be naive and if she’s says anything about it that is scientifically falsifiable, then she is confused about what she means. But whenever she gives a description that is not scientifically falsifiable, and you see it as false, you are probably interpreting the words in too literal a way. Magical thinkers make the same mistake as well, and there is a downward spiral of overly literal thinking when the original belief was just that there is some concept of self that is related in some complex way with the physical body, that has value to her and her community (religion).
I don’t think establishing a rule “you won’t try to convert her” is going to work well, as I personally wouldn’t want to always (or ever) have to repress what I think. That’s not respectful of either of you. Instead, another suggestion: use your rationalist skills to approach it as a research project, but expand your sources. Trying reading books written by really good theologians (for example, very logical ones) on whatever they have to say about the soul—you may be able to find a description of “soul” for example, that you both agree on. And then the difference may just end up being that she has a context for valuing that concept, while you don’t.
I am pretty sure that most strong male rationalists are better off learning how the typical woman thinks than holding out for a long-term relationship with a strong female rationalist. Since this point is probably of general interest, I put it in a top-level post.
Converting her to your worldview sounds like a bad idea in general. An additional consideration that applies in your particular situation is that converting a helping professional from deontologism to consequentialism will more likely than not make her less effective at work (because helping professionals need all the help they can get to care enough about their patients and clients, and worldview is definitely one source of significant help in that regard).
Nobody has responded to the following:
I, too, will refrain from commenting because you probably mean “strong swings of mood” and I do not have romantic experience with a moody woman. I do have romantic experience with a fiery woman, i.e., a woman easily aroused to strong negative emotions, but I doubt that is what you mean: in what I am calling a “fiery” woman, the emotion always dissipates quickly—usually in a few minutes.
You say,
I would consider that a very positive sign in a prospective sexual partner—maybe an extremely positive sign (the reason for my uncertainty being that I have never been with a woman whose expected global utility was as high as you describe) -- a sign that would make me pursue the woman much more keenly. The fact that you use language such as “would have net-benefits for her and for the world long-term” (emphasis mine) suggests to me that you are like me in the relevant characteristics and consequently should take it to be a very positive sign, too.
The most I can say about the global expected utility (i.e., expected effect on the world in the long term) of any of my girlfriends up to now is that (1) she has many close friendships of long duration, and she is very caring and helpful to those friends or that (2) she is a resourceful and clearly productive member of the labor force and does not harm anyone unless you consider the occasional cheating of the government a harm. If I were with a woman whose expected global utility was much higher than any of my girlfriends up to now, there is a good chance that I could become much more unconditionally loving to her than I have been to any of my girlfriends up to now. By “unconditionally loving” I mean being helpful and caring to her without any regard for how much she has done for me or is expected to do for me.
So, that is why I would consider what you wrote a very positive sign: lack of expected global utility is my best current guess as to what has been holding me back from being more unconditionally loving to my girlfriend up to now. (Why I even want to become more unconditionally loving to my girlfriend is a long story.)
And yeah, I know that “expected global utility of the girlfriend” is an odd and cold phrasing, but if that oddness or coldness is enough to prevent you from reading this comment, then we are probably too different for the advice in this comment to be of any use to you.
Thanks very much for your thoughts, and for making a top level post on the topic. Yes, her contribution to social welfare is something I find very attractive, and you help me remember just how important and rare that is.
Rationality sometimes goes badly wrong, when important details don’t fit into a neat reasoning structure or a fatal flaw in argument goes undetected. Emotional reasoning sometimes goes badly wrong, when it deals with corner cases or situations too far from the environments it was evolved to handle. Rationality goes wrong less often, but crucially, they go wrong in different and mostly non-overlapping circumstances. If you have a different world view and reasoning style than your partner, then this might produce conflict, but it also gives you both far better sanity checking than any like-minded person could. You can’t transform her mind, but you can act as a rationalist oracle. You speak of this as though it were only a flaw, but in fact it has both an upside and a downside. Use the upside, and mitigate the downside.
Gaining rational framework doesn’t deprive you of emotional reasoning, but allows you to guide it in normally confusing situations.
How strong is your own understanding of rationality? Why are you an atheist, (why) do you believe science works, what is the difference between one person who is actually right and another person who is merely confused, etc.? How smart is she (that is, how easy it’ll be to positively apply more confused and complex arguments, not requiring of you great feats of persuasion)?
I’ll be interested in what other people think about applying the corpus of Overcoming Bias directly (doesn’t seem like a generally good idea to me, requires a person to be of a kind that’ll maintain focus for extended period of time).
I think I’m quite rational and have a decent understanding of aspects of rationality that I haven’t managed to implement yet. I think karma is a very imperfect measure, but I’ll note that I have more than 100 and less than 400.
She is probably one standard deviation above average in terms of IQ, and would score more highly when considering other kinds of intelligence. The main problem in convincing her to think more rationally is emotional resistance.
Thank you for responding—see my “Added” to the original comment.
Huh? Karma on this site primarily shows positive contribution/participation (if you are able to contribute positively), treating it as a “measure of rationality” is a bizarre perspective. Please try to outline your position on the specific questions I suggested, or something of that kind, it’s hard to make such a decision in a well-known case, but yet harder to construct fully general advice.
For another example, why do you think it’s important to get away from magical thinking? Is it? What is your motivation for thinking about rationality, and for dispelling the other person’s confusion? “Compatibility” of worldviews?
I would probably give you a response you liked better if I understood why you were asking what you were asking.
Because the evidence favors atheism and suggests science leads to truth more often than other approaches to belief formation? I could link to arguments but I don’t see the point in trying to explain these things in my own words. Does it help to know that I usually agree with your comments and with the LW consensus, where it exists? Is the implication that the more rational I am, the more of a problem my partners rationality will be?
I don’t think I understand this question.
I think the importance of getting away from magical thinking varies across people and contexts. I’m not confident I know how important it is, or even whether its helpful, for some people. Its clear that getting away from magical thinking can sometimes help people achieve their personal goals and help make the world a better place.
I enjoy the process regardless of the consequences. But I also hope that it will help me in my career and help me contribute to the world.
I think my partner and I both experience some level of discomfort at knowing that our worldviews are in significant conflict, even though this conflict seems to coexist with a high degree of respect for how the accomplishments of the other. It is unfortunate that we basically have to avoid certain topics of conversation that we both find important and that our emotional reaction to things often differs.
You might check my responses to Alicorn to learn more. Once again, thank you very much for responding.
This is a delicate topic, but I think Vladimir is trying to tell whether you really use rationality to the degree you claim, or whether you rather accept certain opinions of people you see as rationalists, and wish others shared them. In the latter case, it doesn’t matter that the clash is between rationalist and irrationalist opinions: the conflict is isomorphic to any relationship between people of different religions or political parties, and much of the advice for those people should work for you. It’s the former case that would require more particular advice.
I’m afraid that, in the absence of seeing your thought process, much of this looks like guessing the teacher’s password to me. I’d be happy to be corrected, though.
EDIT: Wow, that sounds really tactless and dismissive of me. I retract my accusation, on the basis of (1) not having any real justification and (2) it would set a bad precedent, especially for the sort of reception newcomers get.
Its interesting that people seem to a) be as skeptical of my rationality as they seem to be, and b) think that is the crux of the matter.
Regarding a), if someone tells me that they’ve been reading OB/LW for quite a while and that they think they are considerably more rational than their romantic partner, I think it is very likely that they are correct. But maybe if I was on the other side I would react differently. If I knew of an easy way to prove my rationality I would, but I don’t. Even writing an original rational essay wouldn’t prove much because I could easily be irrational in other domains.
Regarding b), I’m not sure exactly how important it is that potential advice-givers have a very accurate estimate of my rationality (and my girlfriend’s rationality). Perhaps it would be helpful to focus on more specific aspects of our beliefs and approaches to experiencing and acting in the world.
I lean towards preference utilitarianism, though I don’t walk the walk as well as I should. I attempt to calculate the costs and benefits of various choices, she does this too sometimes, but doesn’t like applying it reflexively. She believes in spirits, I’m into Dennett and Dawkins (though I see positive aspects to religion/spirituality)
My partner and I both agree that: She is much more emotional and I am more rational. She is more prone to depression. She has more faith in intuition, I’m more skeptical of it.
Lets say you’ve read everything I’ve written here and you think I’m probably no more rational than my partner. ok, that’s fine, I’d be happy to hear advice that works for two equally irrational people with different beliefs/values/approaches to experiencing and acting in the world.
So the program of understanding each other doesn’t make progress. I agree with Alicorn, it’s essential to establish a mode of communication where you can steadily work on disagreements, with the goal of ultimately resolving them in full. The arguments shouldn’t turn into color politics, polarizing and alienating.
A bit of advice, based on my experience, for a long-term conversion strategy:
Work on understanding your own position better, make sure you know why you believe what you believe before trying to convince another person to change one’s mind. Maybe you are wrong.
Make the mode of interaction and your goals clear when you are arguing, distinguish analysis from social interaction.
Educate the person about fundamentals, thus steadily crafting tools for making deeper arguments in specific discussions.
Prefer shifting the discussion towards education about more general mistake that (might have) contributed to a specific mistake or confusion. In long term, it’s more important than resolving a specific problem, and it’s easier on the other person’s feelings, as you are educating on an abstract theme, rather than attacking a conviction.
Don’t argue the objects of emotional attachment, ever. Instead, work on finding an angle of approach (as suggested above, maybe something more fundamental) that allows you to make progress without directly confronting the issue.
Not everyone is going to change, some people are too dim or too shallow or persistently not interested.
it would appear to the average person that most rational types are only moderately successful while all the extremely wealthy people are irrational. due to not seeing the whole sample space (that larger proportion of rational people enjoy moderate success vs a tiny fraction of irrational people who enjoy major success) I don’t think a lot of our arguments gain traction with people. Most people infer from outliers as a matter of course.
Now combine this with the idea that signaling rationality is also signaling that we think we deserve more status and decision making capability than we have (remember in politics people act as if we live in bands of 100 and political ideas actually mean something) and it starts making sense why we make people nervous and they might reject us out of hand.
So am I just ragging on rationality, trotting out a flawed reason not to be rational? No. I’m saying that something we know a lot about applies to us as well: successful ideologies are the ones that allow people to signal palatable goals while pursuing their selfish top level goal (grab resources, have children) (and that applies even if they’re unaware of their top level goal vis-a-vis adaptation executor not fitness maximizer). We have to keep this in mind while proselytizing. A bare appeal to “achieving your goals more effectively” doesn’t work if the person knows on some level that their stated goals are not their actual goals. They don’t need a system for achieving their stated goals.
This only makes sense if you consider “rational” to equal “geeky Spock-wannabe”, in which case the correlation is reasonable. Bill Gates comes close to that stereotype, though, except for being irrationally passionate about controlling everything and yelling profanity a lot.
I do know a few wealthy guys—some are more stereotypically rational, others are only moderately instrumentally rational in the context of “do what works, stop doing what doesn’t”, while being utterly wacko with respect to everything else.
Of course, these guys are only moderately wealthy—businesses with 8-figure annual sales, net worth below $100 mill, so they’re certainly not in BillG’s league.
Anyway… I can see normal people associating people who like to signal an ideal of rationality with a lack of success and wealth, as it seems to me there’s a correlation there. However, being rational—at least instrumentally—doesn’t require even a fraction of the sort of mathematical rationality promoted here.
See, all of the wealthy people I know, got that way because they don’t mind losing thousands or even tens of thousands of dollars trying things that might not work, and which they have no guaranteed way to evaluate in advance, and no science to study. They don’t mind being wrong, because they understand that they’re in a business where the black swans are on the upside, not the downside.
We’re talking here about perception, not reality, and I’m sorry to say that “geeky Spock-wannabe” probably does equate to the average person’s perception of “most rational types” .
(Most people take a concept to mean whatever most uniquely distinguishes it from other concepts—so ‘rational’ means whatever, in the characteristics they associate with rationality, is most unique and different from the other concepts they have. i.e. Spock-like).
Not only that, often people’s goals require irrational thinking. If you’re hoping to find a mate in a religious community, or if you’re a businessman bringing the free market to the boonies there’s an obvious rational incentive to believe irrational things.
A few related hypotheses: Rationality is highly correlated intelligence. Intelligence is highly correlated to proneness to boredom (in the range IQ >= 90). Boredom is likely to make you fail at the task at hand. The most reliable ways to get rich are perceived as boring to intelligent people.
It seems to me that boredom occurs when the task at hand is perceived as not leading towards any highly valued goal. Rationality will gladly give you reasons why some of your goals should not be highly valued. For example, I would like to have millions of euros. I have recently seen evidence that people lead happy lives no matter what their circumstances are. I have discovered that nearly anything I think would be great fun costs less than a thousand euros. I am convinced money does not make one more attractive to women. If I was less analytical about these issues, I think I might perceive my goal as having higher value, and working towards it would be less boring.
According to research K.E. Stanovich, this is not the case:
See http://www.project-syndicate.org/commentary/stanovich1
It would be useful if everyone used quote markup more often; it can be difficult to figure out what someone’s agreeing to when they say “I agree” and there are a dozen posts separating the agreement from the original post.
Is anybody aware of an academic course which tries to teach OB / LW type stuff? I might someday have the opportunity to teach such a class.
Has anyone made and organized a list of important posts that could serve as the rough draft for a syllabus?
No, I’ve never heard of such a course. Depending on what you mean by that ‘type stuff’, I’d expect it to be in the psychology, anthropology, economics, or polisci departments.
One quick question—quantum game theory—useful or a mathematical gimmick? I can follow the math, but it’s not clear to me if it makes any sense in the real world or not. Can anybody who took a closer look at it tell me what they think?
Suppose I decide that I’m going to partake in a pleasurable activity. How far removed must the decision be from the activity before the decision is no longer reinforced by operant conditioning?
Educate me, LW hive mind. Robin Hanson has mentioned that prediction markets can give not just probability assessments on discrete sets of outcomes but also probability distributions over such assessments that let us know how (un)certain the market is concerning a particular assessment (or at least, that’s how I interpreted his words). Does anyone have links to descriptions of such methods?
The key phrase is “scoring rules.” Another phrase is “combinatorial prediction markets,” but I think that’s something else.
del
In case it hasn’t already been posted by somebody, here’s a nice talk about irrational behaviour and loss aversion in particular.
Perhaps this is a known issue, but since I haven’t seen it discussed, I thought I’d mention that images don’t seem to work in some (all?) of the old posts imported from Overcoming Bias. See for example:
Timeless Physics
The first few pics in that particular post is available from an external server if you click them, but I don’t see them inline. The last picture seems to have been hosted at Overcoming Bias, and is no longer accessible.
Work is underway, apparently.
Has anyone read: “3:16 Bible Texts Illuminated by Donald E. Knuth”
http://www.amazon.com/3-16-Bible-Texts-Illuminated/dp/0895792524
...?
Would you all please recommend books on many-worlds? I liked The End of Time but I thought the treatment of MWI was too cursory.
Hi,I have a question I haven’t seen adressed after a qucik search. A friend of mine has been diagnosed with mild paranoid schizophrenia after he attacked his brother and got hospitalized thereafter. this was 2 years ago.he got (and still gets) medical treatment (some sort of neuroleptica, I suppose), but not much more. it sort of helped, he has a nice job and some superficial friendships (he never had great interest in things social). Now, the paranoia has surfaced again. I guess it was there all along, but nobody knew for sure. We’re afraid it’ll get worse soon. Question is, what to do? I adress this here because my friend is higly intelligent and seems still responsive to reason, indeed, he helds rational thinking in high regard. Doctors advise not to mention his delusions in order to not “manifest the paranoia”, so i don’t know much besides i’m part of some sort of minor conspiracy revolving around him. this conspiracy is the only thing that irritates and bothers him—he doesn’t hear voices or anything like that. he rejects any form of therapy or hospitalization- according to him, it was a very bad and traumatic experience. So. Might it be possible to talk him out of it? any thoughts and questions for clarification welcome.
I guess you’d need an expert psychiatrist rationalist to benefit from this angle. Any obvious advice runs too high a risk of being invalidated by more solid facts known about people in such conditions, so one would need to start with that knowledge.
Thanks for the quick response. I have some trouble understanding, probably due to the language barrier. Do you mean 1) if at all, olny an “expert psychiatrist rationalist” might talk him out of it, or, 2) I should seek knowledge from such a person? if 2), where to find them? Any suggestions?
I’m not aware of anyone here being a professional psychiatrist, so can’t help with that. I’m only warning that seeking advice from people who know nothing about a phenomenon is generally a bad idea, even if those people have some orthogonal advantage. For example, it’s (much) better to seek construction advice from an ordinary engineer that from a genius who is not an engineer.
And for another suggestion for the site itself, it should be possible to tag posts (especially articles, possibly comments) by language, and let users pick what languages they want to see. The interface wouldn’t necessarily have to be translated; it would just be nice to have some support for multilingualism.
there’s a thread for requested features.
Thank you.
I’m considering donating to World Vision UK. Does anyone know much about them?
More generally, is there an easy way to find out how good a charity is? Are there reviews done by third parties?
I recommend you check out GiveWell, they’re doing this sort of thing.
This might not be the right place to ask, but I’ll try anyway:
I read an online paper/article on global dictatorship / totalitarianism as an existential threat a while ago, but I can’t find it anymore. I’ve probably found it on OB / SIAI’s website or something like that in the first place, but can’t find it there now. Would anyone know of such an article (or any good article on the topic, for that matter)?
That was probably Bryan Caplan’s chapter The Totalitarian Threat for the GCR book.
Yup, that’s the one, thanks!
Bryan Caplan’s chapter in Global Catastrophic Risks
Two links that might foster discussion:
http://www.philosophersnet.com/games/
Fun online rationality and anti-bias oriented games. I particularly enjoyed “Staying Alive” (testing conceptions of selfhood). And
http://bloggingheads.tv/diavlogs/20086
Great discussion, I hadn’t seen Gendler before but Bloom is always good. Reminded me a little of the IAT discussion here a few months ago.
Any fellow OB/LW-ers attending OSCON this year? If you are interested in meeting up there let me know. Perhaps respond to this comment or email me directly and we’ll see what we can work out. (You can contact me via gmail at user name evtujo).
What a critic might say about Less Wrong:
1) The purpose of the pursuit of rationality is to increase an individual’s understanding of and power over their environment and the people in it.
2) The only way to establish rational thinking in a group of people not otherwise disposed towards it* is to establish a group norm of praising rational thinkers and shaming the irrational, by an established standard of rationality.
Therefore:
Rationalists are power-seekers, and the pursuit of rationality is inherently elitist and exclusionary.
*That is to say, the vast majority of people.
Number two is flawed. You can also establish rationality in people not otherwise interested in it by convincing them that it will be useful to them in pursuing other goals that they’re interested in.
But recognizing that rationality is useful requires the application of rationality. So you’ll only convince people that already agree, to the degree that they do agree.
The real obstacle is that people often don’t want to apply rationality and prefer to use magical thinking instead. Rationalists aren’t those who know how to be rational, they’re the ones who choose rationality over magic when push comes to shove.
You don’t have to wait until they already agree, just until they come up against a situation where their magical thinking is obviously not working. Show them that rationality fixes the problem in a few of those situations, and there’s a decent chance that they’ll start trying it on their own.
That’s not nearly enough. To work, you have to pick a situation where the emotional obstacles to thinking rationally are weak, the tendency to avoid extended thought is most likely to be overridden because the outcome is important, and then present the rational strategy in a way that doesn’t denigrate magical thinking.
It’s difficult to locate such moments, as they are very rare. Better by far simply to find people who genuinely want to be right.
A situation where the person has already figured out that their way of handling things isn’t working, and they’re looking for advice, fits those criteria. They’re still unlikely to ask for advice from someone who’s more likely to mock them than to be useful, though, which is probably why you hear about those situations so infrequently.
But such people usually don’t conclude that the magical thinking itself isn’t working. They just want help finding a specific magical argument that will end up with their getting what they want.
I don’t think I’ve ever known anyone who was actually against rationality altogether. Most people just want a solution that works, and if the one they wind up using is rational, they don’t have a problem with that. And if you can teach them some rationalist skills while you’re giving them that solution, all the better. It takes time to bootstrap people that way, but it’s far from being impossible.
The imaginary critic points out that convincing wholly irrational people to be rational is unlikely to be possible using rational arguments. A rationalist would need power over the irrationalist in order to change their mind.
Or, we could just use the same systems that less-rational people use to convince each other of things, but in a way that in in line with the rationalist ethos.
Since when does valuing rationality limit what other skills we can use?
Critic: “‘Other skills’ means skills of manipulation, rationalists readily abandon honesty in favour of power.”
I just want to clarify that this is not what I think, but I have seen comments of this kind made about Overcoming Bias (on other blogs), though never about Less Wrong. I have never written anything in reply. I was wondering how anyone here would respond to this kind social criticism. Would you even care enough to respond?
Anticipating critics and responses to them is largely a waste of time, if they are determined to be against you. Whatever you say will only be fodder for the next attack, and you are wasting precious time and energy being pinned down by their fire.
What we want is responses for people who are not the critics, but may have heard the critics’ arguments. That’s a considerably less-demanding audience.
This is a key distinction which all rationalists trying to talk to the public should keep in mind in all places at all times.
Power and honesty (or, more accurately, Friendliness?) aren’t mutually exclusive, but yes, that’s a major issue. Maybe we need to spend more time talking about Friendliness here, even though it’s not exactly a rationalist topic?
I’m highly skeptical of the idea that rationalism isn’t a strategy for getting laid. It’s just a strategy that says the direct methods of using manipulation are worse than using rationality to get rich (or save the world, or something in between). This is because we recognize that most manipulators fall prey to their own shoddy reasoning and are ultimately handicapping themselves even if they have more success in the short term.
Of course such rationalizing about short term vs long term success can be thought of as a natural response from high IQ nerds who find little success with women in day-to-day life. I still think this is true even if it turns out to be correct that long term planning pays off (as is usually the case, unless you get hit by a truck :)
bring on the downvotes.
Rationalism, as discussed on LW, is not particularly helpful for getting laid.
Perhaps what you mean is that our interest in rationalism is motivated by mental circuits which are indirectly oriented towards us getting laid. This is true, but this could also be said of peoples interest in sports, dance, conversation, literature and the arts, etc. Is there any reason to believe it is more true of rationalism?
Well, if you filter out the epistemic stuff and focus on various instrumental-rationality bits like:
Willingness to accept unpleasant or unpopular ideas
Willingness to try things you see others succeeding with, even if they seem to be based on ideas that are absurd or impossibly wrong
Willingness to suspend disbelief while you are doing something, separating evaluating from doing
Observing reality to see what works, rather than imagining you are more (or less) successful than you actually are, by devising as-objective-as-practical test/success criteria in advance
Accepting others’ beliefs and worldviews at face value, without judging them “good” or “bad”
Then yeah, you will find some useful things here, though of course perhaps not nearly as useful things as studying some domain-specific materials on the topic. But the above ideas will serve you well in any domain that involves influencing human behavior, whether it’s your own behavior or someone else’s.
How to design utility functions for safe AIs?
Make a utility function which will only emit positive values if the AI is disabled at the moment the solution to your precise problem is found. Ensure that the utility function will emit smaller values for solutions which took longer. Ensure the function will emit higher values for world which are more similar to the world as it would have been without the AI interfering.
This will not create friendly AI, but an AI which tries to minimize its interference with the world. Depending on the weights applied to the three parts, it might spontaneously deactivate though.
“AI is disabled” and “world more similar to the world as it would have been without the AI interfering” are both magical categories. Your qualitative ontology has big, block objects labeled “AI” and “world” and an arrow from “AI” to “world” that can be either present or absent. The real world is a borderless, continuous process of quantum fields in which shaking one electron affects another electron on the opposite side of the universe.
I understand the general point, but “AI is disabled” seems like a special case, in that an AI able to do any sort of reasoning about itself, allocate internal resources, etc. (I don’t know how necessary this is for it to do anything useful), will have to have concepts in its qualitative ontology of, or sufficient to define, its disability – though perhaps not in a way easily available for framing a goal system (e.g. if it developed them itself, assuming it could build up to them in their absence), and probably complicated in some other ways that haven’t occurred to me in two minutes.
Suppose the AI builds devices in the environment, especially computational devices designed to offload cognitive labor. What do you want to happen when the AI is “switched off”? Hence, magical category.
Interesting, I didn’t think of this situation. How do you define “lack of intelligence” or “removal of the effect of intelligence” in the environment, so that an AI can implement that state? How is this state best achieved?
Once the system is established, the world will ever be determined by a specific goal system, even if the goal is for the world to appear as if no AI is present, starting from a certain time. The best solution is for AI to pretend of not being present, “pulling the planets along their elliptic orbits”.
D’oh. Yes, of course, that breaks it.
As an aside, “waiting for Eliezer to find a loophole” probably does not constitute a safe and effective means of testing AI utility functions. This is something we want provable from first principles, not “proven” by “well, I can’t think of a counterexample”.
Of course, hence ”...and probably complicated in some other ways that haven’t occurred to me in two minutes.”.
Right. I know you realize this, and the post was fine in the context of “random discussion on the internet”. However, if someone wants to actually, seriously specify a utility function for an AI any description that starts with “here’s a high-level rule to avoid bad things” and then works from there looking for potential loopholes is deeply and fundamentally misguided completely independently of the rule proposed.
This might make a fun game actually.
Player One sets up a hypothetical AI (utility function + restraints), and Player Two, or the rest of the group, must find a situation where a super intelligence with that utility function would lead to a dystopia. Here’s my entry:
1) Utility is obtained by ‘votes’. Every person can either give +1 utility or −1 utility to the AI each day. To avoid issues of uncertainty over what constitutes a person, before the AI is turned on everyone is given a voting button. Additional voting buttons that may be created provide no utility.
2) Strongly discount future votes, using U(t) = 1 / 2^t for the new utility, where t is time in days.This way a constant vote for all time yields a finite utility.
3) The AI can not take any action until one year after it has been turned on, except for text outputs to satisfy (4)
4) The AI has an oracle component. Any voter can ask how their life on a particular future day will vary from the present day. The AI must answer to the best of its abilities.
I ask it a question as in (4), and it tells me “build this machine [specifications included] and you will get [something the AI believes it can tempt me with]”. It exerts the best of its ability in persuading me. Unknown to me, the machine that it wants me to build takes command of all the voting buttons and jams them into giving the AI +1 every day.
I don’t think Player One can win this game.
I should have been more clear about what I meant by an oracle. The only utility the AI takes into account when answering a question is accuracy. It must be truthful and only answers questions about differences in the days of a voter. A valid question could be something like “A year from now, if you do [horrendous act] would my voter button still vote you down?” If not, I’d vote the AI down for the whole year until it begins acting.
While I recognize the huge threat of unfriendly AI, I’m not convinced Player One can’t win. I’d like to see the game played on a wider scale (and perhaps formalized a bit) to explore the space more thoroughly. It might also help illuminate the risks of AI to people not yet convinced. Plus it’s just fun =D
You can’t get anything useful out of AI directed to leave the future unchanged, as any useful thing it does will (indirectly) make an impact on the future, through its application by people. Trying to define what kind of impact really results from the intended useful thing produced by AI brings you back to the square one.
Minimize is not “reduce to zero”. If the weighting is correct, the optimal outcome might very well be just the solution to your problem and nothing else. Also, this gives you some room for experiments. Start with a function which only values non-interference, and then gradually restart the AI with functions which include ever larger weights for solution finding, until you arrive at the solution.
Or until everyone is dead.
If the solution to your problem is only reachable by killing everybody, yes.
Um, then this is not a “safe” AI in any reasonable sense.
I claim that it is, as it is averse to killing people as a side effect. If your solution does not require killing people it would not.
Stop, and read The Hidden Complexity of Wishes again. To us, killing a person or lobotomizing them feels like a bigger change than (say) moving a pile of rock; but unless your AI already shares your values, you can’t guarantee it will see things the same way.
Your AI would achieve its goal in the first way it finds that matches all the explicit criteria, interpreted without your background assumptions on what make for a ‘reasonable’ interpretation. Unless you’re sure you’ve ruled out every possible “creative” solution that happens to horrify you, this is not a safe plan.
If it can understand “I have had little effect on the world”, it can understand “I am doing good for humanity”. A “safe” utility function would be no easier and less desirable than a Friendly one.
No easier? There’s a lot of hidden content in “effect on the world”, but presumably not all of Fun Theory, the entire definition of “person”, etc. (or shorter descriptions that unfold into these things). An Oracle AI that worked for humans would probably work just as well for Babyeaters or Superhappies (in terms of not automatically destroying things they value; obviously, it’d make alien assumptions about cognitive style, concepts, etc.).
I agree with that much, but the question is whether there’s enough hidden content to force development of a general theory of “learning what the programmers actually meant” that would be sufficient unto full-scale FAI, or sufficient given 20% more work.
Does moving a few ounces of matter from one location to another count as a significant “effect on the world”?
Does it matter to you whether that matter is taken from 1) a vital component of the detonator on a bomb in a densely populated area or 2) the frontal lobe of your brain?
If it does matter to you, how do you propose to explain the difference to an AI?
In general, yes; you can and should be much more conservative here than would fully reflect your preferences, and give it a principle implying your (1) and (2) are both Very Bad.
But, the waste heat from its computation will move at least a few ounces of air.
Maybe you can get around this by having it not worry (so to speak) about effects other than through I/O, but this is unsafe if it can use channels you didn’t think of to deliberately influence the world. Certainly other problems, too – but (it seems to me) problems that have to be solved anyway to implement CEV, which is sort of a special case of Oracle AI.
Quite so. The waste heat, of course, has very little thermodynamically significant direct impact on the rest of the world—but by the same token, removing someone’s frontal lobe or not has a smaller, more indirect impact on the world than preventing the bomb from detonating or not.
Now, suppose the AI’s grasp of causal structure is sufficient that it will indeed only take actions that truly have minimal impact vs. nonaction; in this case it will be unable to communicate with humans in ways that are expected to result in significant changes to the human’s future behavior, making it a singularly useless oracle.
My intuition here is that the insights required for any specification of what causal results of action are acceptable is roughly equivalent to what is necessary to specify something like CEV (i.e., essentially what Warrigal said above) in that both require the AI have, roughly speaking, the ability to figure out what people actually want, not what they say they want. If you’ve done it right, you don’t need additional safeguards such as preventing significant effects; if you’ve done it wrong, you’re probably screwed anyways.
That’s pretty condensed. One of my video/essays discusses the underlying idea. To quote:
“One thing that might help is to put the agent into a quiescent state before being switched off. In the quiescent state, utility depends on not taking any of its previous utility-producing actions. This helps to motivate the machine to ensure subcontractors and minions can be told to cease and desist. If the agent is doing nothing when it is switched off, hopefully, it will continue to do nothing.
Problems with the agent’s sense of identity can be partly addressed by making sure that it has a good sense of identity. If it makes minions, it should count them as somatic tissue, and ensure they are switched off as well. Subcontractors should not be “switched off”—but should be tracked and told to desist—and so on.”
http://alife.co.uk/essays/stopping_superintelligence/
I haven’t thought this through very deeply, but couldn’t the working of the machine be bounded by hard safety constraints that the AI was not allowed to change, rather than trying to work safety into the overall utility function?
e.g. the AI is not allowed to construct more resources for itself. No matter how inefficient it may be, the AI has to ask a human for more hardware and wait for the hardware to be installed by humans.
What I would want of a super-intellient AI is more or less what I would want of a human who has power over me: don’t do things to me or my stuff without asking, don’t coerce me, don’t lie to me.
I don’t know how you would code all that, but if we can’t design simple constraints, we can’t correctly design more complex ones. I’m thinking layers of simple constraints would be safer than one unprovably-friendly utility function.
There’s been a great deal of discussion here on such issues already; you might be interested in some of what’s been said. As for your initial point, I think you have a model of a ghost in the machine who you need to force to do your bidding; that’s not quite what happens.
Nope, I’m a software engineer, so I don’t have that particular magical model of how computer systems work.
But suppose you design even a conventional software system, to run something less dangerous than a general AI, like say a nuclear reactor. Would you have every valve and mechanism controlled only by the software, with no mechanical failsafes or manual overrides, trusting the software to have no deadly flaws?
Designing that software and proving that it would work correctly in every possible circumstance would be a rich and interesting research topic, but never be completed.
One difference with AI is that it is theoretically capable of analyzing your failsafes and overrides (and their associated hidden flaws) more thoroughly than you. Manual, physical overrides aren’t yet amenable to rigorous, formal analysis, but software is. If we employ a logic to prove constraints on the AI’s behavior, the AI shouldn’t be able to violate its constraints without basically exploiting an inconsistency in the logic, which seems far less likely than the case where, e.g., it finds a bug in the overrides or tricks the humans into sabotaging them.
What makes you think these constraints are at all simple?
New rationalist blog: “gullibility is bad for you”:
http://gullibilityisbadforyou.blogspot.com/