There are too many solved problems that take years of learning to understand, more to understand the solution, and history of humankind’s effort to solve.
Your objection partially defeats itself. Eliezer suspects that FAI is indeed one of those problems that would normally take many decades of effort from a whole civilization to conquer, and he wants to do it in a fraction of the time, using many fewer people. That looks pretty impossible, by any meaning of the word. We know enough about the problem space to put a lower bound on how much we don’t know, and that lower bound is still in the “impossible” range.
On the other hand, once we eliminate enough confusion to be able to put better estimates on things things, we already understand them well enough that they no longer seem impossible. So, is the very act of judging something to be impossible, itself impossible?
Your objection partially defeats itself. Eliezer suspects that FAI is indeed one of those problems that would normally take many decades of effort from a whole civilization to conquer, and he wants to do it in a fraction of the time, using many fewer people. That looks pretty impossible, by any meaning of the word. We know enough about the problem space to put a lower bound on how much we don’t know, and that lower bound is still in the “impossible” range.
On the other hand, once we eliminate enough confusion to be able to put better estimates on things things, we already understand them well enough that they no longer seem impossible. So, is the very act of judging something to be impossible, itself impossible?