There seem to be more and more people who think sacrificing their life to help build FAI is an ethical imperative. If I were Eliezer, I would run screaming in the other direction the moment I saw the first such person
You mean when he saw himself in the mirror? :)
Seriously, do you think sacrificing one’s life to help build FAI is wrong (or not necessarily wrong but not an ethical imperative either), or is it just bad PR for LW/SI to be visibly associated with such people?
I think it’s not an ethical imperative unless you’re unusually altruistic.
Also I feel the whole FAI thing is a little questionable from a client relations point of view. Rationality education should be about helping people achieve their own goals. When we meet someone who is confused about their goals, or just young and impressionable, the right thing for us is not to take the opportunity and rewrite their goals while we’re educating them.
It’s hard not to rewrite someone’s goals while educating them, because one of our inborn drives is to gain the respect and approval of people around us, and if that means overwriting some of our goals, well that’s a small price to pay as far as that part of our brain is concerned. For example, I stayed for about a week at the SIAI house a few years ago when attending the decision theory workshop, and my values shift in obvious ways just by being surrounded by more altruistic people and talking with them. (The effect largely dissipated after I left, but not completely.)
I think it’s not an ethical imperative unless you’re unusually altruistic.
Presumably the people they selected for the rationality mini-camp were already more altruistic than average, and the camp itself pushed some of them to the “unusually altruistic” level. Why should SIAI people have qualms about this (other than possible bad PR)?
I think it’s not an ethical imperative unless you’re unusually altruistic.
I don’t think “unusually altruistic” is a good characterization of “doesn’t value personal preferences about some life choices more than the future of humanity”...
Do you believe most people are already quite altruistic in that sense? Why? It seems to me that many people give lip service to altruism, but their actions (e.g. reluctance to donate to highly efficient charities) speak otherwise. I think rationality education should help people achieve the goals they’re already trying to achieve, not the goals that the teacher wants them to achieve.
I think rationality education should help people achieve the goals they’re already trying to achieve, not the goals that the teacher wants them to achieve.
False dichotomy. Humans are not automatically strategic, we often act on urges, not goals, and even our explicitly conceptualized goals can be divorced from reality, perhaps more so than the urges. There are general purpose skills that have an impact on behavior (and explicit goals) by correcting errors in reasoning, not specifically aimed at aligning students’ explicit goals with those of their teachers.
Rationality is hard to measure. If LW doesn’t make many people more successful in mundane pursuits but makes many people subscribe to the goal of FAI, that’s reason to suspect that LW is not really teaching rationality, but rather something else.
(My opinions on this issue seem to become more radical as I write them down. I wonder where I will end up!)
I didn’t say anything about “rationality”. Whether the lessons help is a separate question from whether they’re aimed at correcting errors of reasoning or at shifting one’s goals in a specific direction. The posts I linked also respond to the objection about people “giving lip service to altruism” but doing little in practice.
Yes, the reasoning in the linked posts implies that deep inside, humans should be as altruistic as you say. But why should I believe that reasoning? I’d feel a lot more confident if we had an art of rationality that made people demonstrably more successful in mundane affairs and also, as a side effect, made some of them support FAI. If we only get the side effect but not the main benefit, something must be wrong with the reasoning.
Yes, the reasoning in the linked posts implies that deep inside, humans should be as altruistic as you say.
This is not what the posts are about, even if this works as one of the conclusions. The idea that urges and goals should be distinguished, for example, doesn’t say what your urges or goals should be, it stands separately on its own. There are many such results, and ideas such as altruism or importance of FAI are only few among them. Do these ideas demonstrate comparatively more visible measurable effect than the other ideas?
Rationality is hard to measure. If LW doesn’t make many people more successful in mundane pursuits but makes many people subscribe to the goal of FAI, that’s reason to suspect that LW is not really teaching rationality, but rather something else.
if prediction markets were legal, we could much more easily measure if LW helped rationality. Just ask people to make n bets or predictions per month and see 1) it they did better than the population average and 2) if they improved over time.
In fact, trying to get intrade legal in the US might be a very worthwhile project for just this reason ( beyond all the general social reasons to like prediction markets)
There is no need to wish or strive for regulatory changes that may never happen: I’ve pointed out in the past that non-money prediction markets generally are pretty accurate and competitive with money prediction markets; so money does not seem to be a crucial factor. Just systematic tracking and judgment.
(Being able to profit may attract some people, like me, but the fear of loss may also serve as a potent deterrent to users.)
I have written at length about how I believe prediction markets helped me but I have been helped even more by the free active you-can-sign-up-right-now-and-start-using-it,-really,-right-now http://www.PredictionBook.com
I routinely use LW-related ideas and strategies in predicting, and I believe my calibration graph reflects genuine success at predicting.
If other people have suggested this before, there may be enouph background support to make it worth following up on this idea.
When I get home from work, I will post in the discussion forum to see if people would be interested in working to legalize prediction markets ( like intrade) it the US.
[EDITED: shortly after making this post, I saw Gwern’s post above suggesting that an alternative like prediction book would be just as good. As a result I did not make a post about legalizing prediction markets and instead tried prediction book for a month and a half. After this trial, I still think that making a push to legalize predictions markets would be worthwhile]
It doesn’t sound like you know all that many humans, then. In most times and places, the “future of humanity” is a signal that someone shouldn’t be taken seriously, not an actual goal.
I don’t think this calculation works out, actually. If you’re purely selfish (don’t care about others at all), and the question is whether to devote your whole life to developing FAI, then it’s not enough to believe that the risk is high (say, 10%). You also need to believe that you can make a large impact. Most people probably wouldn’t agree to surrender all their welfare just to reduce the risk to themselves from 10% to 9.99%, and realistically their sacrifice won’t have much more impact than that, because it’s hard to influence the whole world.
Funny in which way? Do you want to avoid an automatic “makro-of-denial”-invocation or are you afraid of them joining Eliezers evergrowing crowd of memetically subverted FAI-lers ?
If I teach rationality and deliberately change my students’ goals, that means I fail as a teacher. It’s even worse if their new goal happens to be donating all their money to my organization.
You mean when he saw himself in the mirror? :)
Seriously, do you think sacrificing one’s life to help build FAI is wrong (or not necessarily wrong but not an ethical imperative either), or is it just bad PR for LW/SI to be visibly associated with such people?
I think it’s not an ethical imperative unless you’re unusually altruistic.
Also I feel the whole FAI thing is a little questionable from a client relations point of view. Rationality education should be about helping people achieve their own goals. When we meet someone who is confused about their goals, or just young and impressionable, the right thing for us is not to take the opportunity and rewrite their goals while we’re educating them.
It’s hard not to rewrite someone’s goals while educating them, because one of our inborn drives is to gain the respect and approval of people around us, and if that means overwriting some of our goals, well that’s a small price to pay as far as that part of our brain is concerned. For example, I stayed for about a week at the SIAI house a few years ago when attending the decision theory workshop, and my values shift in obvious ways just by being surrounded by more altruistic people and talking with them. (The effect largely dissipated after I left, but not completely.)
Presumably the people they selected for the rationality mini-camp were already more altruistic than average, and the camp itself pushed some of them to the “unusually altruistic” level. Why should SIAI people have qualms about this (other than possible bad PR)?
Pointing out that religious/cultic value rewriting is hard to avoid hardly refues the idea that LW is a cult.
I don’t think “unusually altruistic” is a good characterization of “doesn’t value personal preferences about some life choices more than the future of humanity”...
Do you believe most people are already quite altruistic in that sense? Why? It seems to me that many people give lip service to altruism, but their actions (e.g. reluctance to donate to highly efficient charities) speak otherwise. I think rationality education should help people achieve the goals they’re already trying to achieve, not the goals that the teacher wants them to achieve.
False dichotomy. Humans are not automatically strategic, we often act on urges, not goals, and even our explicitly conceptualized goals can be divorced from reality, perhaps more so than the urges. There are general purpose skills that have an impact on behavior (and explicit goals) by correcting errors in reasoning, not specifically aimed at aligning students’ explicit goals with those of their teachers.
Rationality is hard to measure. If LW doesn’t make many people more successful in mundane pursuits but makes many people subscribe to the goal of FAI, that’s reason to suspect that LW is not really teaching rationality, but rather something else.
(My opinions on this issue seem to become more radical as I write them down. I wonder where I will end up!)
I didn’t say anything about “rationality”. Whether the lessons help is a separate question from whether they’re aimed at correcting errors of reasoning or at shifting one’s goals in a specific direction. The posts I linked also respond to the objection about people “giving lip service to altruism” but doing little in practice.
Yes, the reasoning in the linked posts implies that deep inside, humans should be as altruistic as you say. But why should I believe that reasoning? I’d feel a lot more confident if we had an art of rationality that made people demonstrably more successful in mundane affairs and also, as a side effect, made some of them support FAI. If we only get the side effect but not the main benefit, something must be wrong with the reasoning.
This is not what the posts are about, even if this works as one of the conclusions. The idea that urges and goals should be distinguished, for example, doesn’t say what your urges or goals should be, it stands separately on its own. There are many such results, and ideas such as altruism or importance of FAI are only few among them. Do these ideas demonstrate comparatively more visible measurable effect than the other ideas?
if prediction markets were legal, we could much more easily measure if LW helped rationality. Just ask people to make n bets or predictions per month and see 1) it they did better than the population average and 2) if they improved over time.
In fact, trying to get intrade legal in the US might be a very worthwhile project for just this reason ( beyond all the general social reasons to like prediction markets)
There is no need to wish or strive for regulatory changes that may never happen: I’ve pointed out in the past that non-money prediction markets generally are pretty accurate and competitive with money prediction markets; so money does not seem to be a crucial factor. Just systematic tracking and judgment.
(Being able to profit may attract some people, like me, but the fear of loss may also serve as a potent deterrent to users.)
I have written at length about how I believe prediction markets helped me but I have been helped even more by the free active you-can-sign-up-right-now-and-start-using-it,-really,-right-now http://www.PredictionBook.com
I routinely use LW-related ideas and strategies in predicting, and I believe my calibration graph reflects genuine success at predicting.
Very nice idea, thanks! After some googling I found someone already made this suggestion in 2009.
If other people have suggested this before, there may be enouph background support to make it worth following up on this idea.
When I get home from work, I will post in the discussion forum to see if people would be interested in working to legalize prediction markets ( like intrade) it the US.
[EDITED: shortly after making this post, I saw Gwern’s post above suggesting that an alternative like prediction book would be just as good. As a result I did not make a post about legalizing prediction markets and instead tried prediction book for a month and a half. After this trial, I still think that making a push to legalize predictions markets would be worthwhile]
It doesn’t sound like you know all that many humans, then. In most times and places, the “future of humanity” is a signal that someone shouldn’t be taken seriously, not an actual goal.
I was talking about the future of humanity, not the “future of humanity” (a label that can be grossly misinterpreted).
… or you estimate the risk to be significant and you want to live past the next N years.
I don’t think this calculation works out, actually. If you’re purely selfish (don’t care about others at all), and the question is whether to devote your whole life to developing FAI, then it’s not enough to believe that the risk is high (say, 10%). You also need to believe that you can make a large impact. Most people probably wouldn’t agree to surrender all their welfare just to reduce the risk to themselves from 10% to 9.99%, and realistically their sacrifice won’t have much more impact than that, because it’s hard to influence the whole world.
Funny in which way? Do you want to avoid an automatic “makro-of-denial”-invocation or are you afraid of them joining Eliezers evergrowing crowd of memetically subverted FAI-lers ?
The latter, I think.
If I teach rationality and deliberately change my students’ goals, that means I fail as a teacher. It’s even worse if their new goal happens to be donating all their money to my organization.