Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don’t know how much my opinions actually changed (there’s some well-known biases about that).
So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don’t understand, already requires solving the hard part?
Current estimate says yes. There would still be an inscrutable problem to solve too, but I don’t think it would have quite the same impenetrability about it.
Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?
Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don’t know how much my opinions actually changed (there’s some well-known biases about that).
Current estimate says yes. There would still be an inscrutable problem to solve too, but I don’t think it would have quite the same impenetrability about it.
Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?
Yes.