This model seems quite a bit different from mine, which is that FAI research is about reducing FAI to an AGI problem, and solving AGI takes more work than doing this reduction.
More concretely, consider a proposal such as Paul’s reflective automated philosophy method, which might be able to be implemented using epsiodic reinforcement learning. This proposal has problems, and it’s not clear that it works—but if it did, then it would have reduced FAI to a reinforcement learning problem. Presumably, any implementations of this proposal would benefit from any reinforcement learning advances in the AGI field.
Of course, even if we a proposal like this works, it might require better or different AGI capabilities from UFAI projects. I expect this to be true for black-box FAI solutions such as Paul’s. This presents additional strategic difficulties. However, I think the post fails to accurately model these difficulties. The right answer here is to get AGI researchers to develop (and not publish anything about) enough AGI capabilities for FAI without running a UFAI in the meantime, even though the capabilities to run it exist.
Assuming that this reflective automated philosophy system doesn’t work, it could still be the case that there is a different reduction from FAI to AGI that can be created through armchair technical philosophy. This is often what MIRI’s “unbounded solutions” research is about: finding ways you could solve FAI if you had a hypercomputer. Once you find a solution like this, it might be possible to define it in terms of AGI capabilities instead of hypercomputation, and at that point FAI would be reduced to an AGI problem. We haven’t put enough work into this problem to know that a reduction couldn’t be created in, say, 20 years by 20 highly competent mathematician-philosophers.
In the most pessimistic case (which I don’t think is too likely), the task of reducing FAI to an AGI problem is significantly harder than creating AGI. In this case, the model in the post seems to be mostly accurate, except that it neglects the fact that serial advances might be important (so we get diminishing marginal progress towards FAI or AGI per additional researcher in a given year).
Transcript:
Question: Are you as afraid of artificial intelligence as your Paypal colleague Elon Musk?
Thiel: I’m super pro-technology in all its forms. I do think that if AI happened, it would be a very strange thing. Generalized artificial intelligence. People always frame it as an economic question, it’ll take people’s jobs, it’ll replace people’s jobs, but I think it’s much more of a political question. It would be like aliens landing on this planet, and the first question we ask wouldn’t be what does this mean for the economy, it would be are they friendly, are they unfriendly? And so I do think the development of AI would be very strange. For a whole set of reasons, I think it’s unlikely to happen any time soon, so I don’t worry about it as much, but it’s one of these tail risk things, and it’s probably the one area of technology that I think would be worrisome, because I don’t think we have a clue as to how to make it friendly or not.