This seems fair. But I am unconvinced that the marginal benefit of understanding AGI now is even close to the marginal benefit of spreading and training rationality now. Unless you believe that either FAI research is very non-parallelizable or that AGI will be feasible in the near future, it seems likely that spreading rationality is more important today (even to the instrumental goal of understanding FAI before AGI is feasible). But if you do believe that AGI will be feasible in the near future, and in particular before long-term efforts to produce more rationalists will have a significant effect on the world, then working on FAI directly (or, more likely, funding current research on FAI, or attempting to influence the AI research community, or trying to do research which is likely to lead to understandable AGI, etc.) is urgent.
So I concede that understanding FAI should precede the feasibility of AGI, but without some additional argument about the timescales on which AGI is feasible, the difficulty of parallelizing FAI research, or some unexpected obstruction to spreading rationality, I am not yet convinced that I should work on or fund FAI research or do anything related to AI.
I am very skeptical about causes that engage exclusively in spreading awareness. By directing the efforts of a small proportion of the rationalists we produce towards direct work on FAI, we validate that we are in fact producing people capable of working on the problem, as opposed to merely having an exponentially growing group of people who profess “Yay FAI!”.
I am very skeptical about causes that engage exclusively in spreading awareness.
As am I. However, here are some things I believe about the SIAI and FAI:
To the average well-educated person, the efforts of the SIAI are indistinguishable from a particularly emphatic declaration of “Yay FAI!” To the average person who cares strongly about FAI, the performance of the SIAI still does not validate that “we are in fact producing people capable of working on the problem,” because there are essentially no standards to judge against, no concrete theoretical results in evidence, and no suggestion that impressive theoretical advances are forthcoming. Saying “the problem is difficulty” is a perfectly fine defense, but it does not give the work being done any more value as validation.
The average intelligent (and even abnormally rational) non-singulatarian has little respect for the work of the SIAI, to the extent that the affiliation of the SIAI with outreach significantly reduces its credibility with the most important audience, and the (even quite vague) affiliation of an individual with SIAI makes it significantly more difficult for that individual to argue credibly about the future of humanity.
It is not at all obvious that FAI is the most urgent technical problem currently in view. For example, pushing better physical understanding of the brain, better algorithmic understanding of cognition, and technology for interfacing with human brains all seem like they could have a much larger effect on the probability of a positive singularity. The real argument for normal humans working on FAI is extremely complicated and uncertain.
I place fairly little value on an exponentially growing group of people interested in FAI, except insofar as they can be converted into an exponentially large group of people who care about the future of humanity and act rationally on that preference. I think there are easier ways to accomplish this goal; and on the flip side I think “merely” having an exponentially large group of rational people who care about humanity is incredibly valuable.
My main concern in the direction you are pointing is the difficulty of effective outreach when the rationality on offer appears to be disconnected from reality (in particular the risk that what you are spreading will almost certainly cease to be “rationality” without some good grounding). I believe working on FAI is a uniquely bad way to overcome this difficulty, because most of the target audience (really smart people whose help is incredibly valuable) considers work on FAI even more disconnected from reality than rationality outreach itself, and because the quality or relevance of work on FAI is essentially impossible for almost anyone not directly involved with that work to assess.
Ideally, Friendly AI should be understood long before AGI is feasible. Like, 200 years ago. It’ll be too late when AGI is possible.
This seems fair. But I am unconvinced that the marginal benefit of understanding AGI now is even close to the marginal benefit of spreading and training rationality now. Unless you believe that either FAI research is very non-parallelizable or that AGI will be feasible in the near future, it seems likely that spreading rationality is more important today (even to the instrumental goal of understanding FAI before AGI is feasible). But if you do believe that AGI will be feasible in the near future, and in particular before long-term efforts to produce more rationalists will have a significant effect on the world, then working on FAI directly (or, more likely, funding current research on FAI, or attempting to influence the AI research community, or trying to do research which is likely to lead to understandable AGI, etc.) is urgent.
So I concede that understanding FAI should precede the feasibility of AGI, but without some additional argument about the timescales on which AGI is feasible, the difficulty of parallelizing FAI research, or some unexpected obstruction to spreading rationality, I am not yet convinced that I should work on or fund FAI research or do anything related to AI.
I am very skeptical about causes that engage exclusively in spreading awareness. By directing the efforts of a small proportion of the rationalists we produce towards direct work on FAI, we validate that we are in fact producing people capable of working on the problem, as opposed to merely having an exponentially growing group of people who profess “Yay FAI!”.
As am I. However, here are some things I believe about the SIAI and FAI:
To the average well-educated person, the efforts of the SIAI are indistinguishable from a particularly emphatic declaration of “Yay FAI!” To the average person who cares strongly about FAI, the performance of the SIAI still does not validate that “we are in fact producing people capable of working on the problem,” because there are essentially no standards to judge against, no concrete theoretical results in evidence, and no suggestion that impressive theoretical advances are forthcoming. Saying “the problem is difficulty” is a perfectly fine defense, but it does not give the work being done any more value as validation.
The average intelligent (and even abnormally rational) non-singulatarian has little respect for the work of the SIAI, to the extent that the affiliation of the SIAI with outreach significantly reduces its credibility with the most important audience, and the (even quite vague) affiliation of an individual with SIAI makes it significantly more difficult for that individual to argue credibly about the future of humanity.
It is not at all obvious that FAI is the most urgent technical problem currently in view. For example, pushing better physical understanding of the brain, better algorithmic understanding of cognition, and technology for interfacing with human brains all seem like they could have a much larger effect on the probability of a positive singularity. The real argument for normal humans working on FAI is extremely complicated and uncertain.
I place fairly little value on an exponentially growing group of people interested in FAI, except insofar as they can be converted into an exponentially large group of people who care about the future of humanity and act rationally on that preference. I think there are easier ways to accomplish this goal; and on the flip side I think “merely” having an exponentially large group of rational people who care about humanity is incredibly valuable.
My main concern in the direction you are pointing is the difficulty of effective outreach when the rationality on offer appears to be disconnected from reality (in particular the risk that what you are spreading will almost certainly cease to be “rationality” without some good grounding). I believe working on FAI is a uniquely bad way to overcome this difficulty, because most of the target audience (really smart people whose help is incredibly valuable) considers work on FAI even more disconnected from reality than rationality outreach itself, and because the quality or relevance of work on FAI is essentially impossible for almost anyone not directly involved with that work to assess.