if we can reduce one FAI problem to another FAI or AGI problem, which we know has to be solved anyway, that counts as solving the former problem
Setting aside what counts as a ‘solution’, merging two problems counts as progress on the problem only when the merged version is easier to solve than the unmerged version. Or when the merged version helps us arrive at an important conceptual insight about the unmerged version. You can collapse every FAI problem into a single problem that we need to solve anyway by treating them all as components of its utility function or action policy, but it’s not clear that represents progress, and it’s very clear it doesn’t represent a solution.
I guess I was interpreting RobbBB’s sequence of posts as describing a narrower problem than your “naturalized induction”.
Naturalized induction is the problem of defining an AGI’s priors, from the angle of attack ‘how can we naturalize this?’. In other words, it’s the problem of giving the AGI a reasonable epistemology, as informed by the insight that AGIs are physical processes that don’t differ in any fundamental way from other physical processes. So it encompasses and interacts with a lot of problems.
That should be clearer in my next couple of posts on naturalized induction. I used Solomonoff induction as my entry point because it keeps the sequence grounded in the literature and in a precise formalism. (And I used AIXI because it makes the problems with Solomonoff induction, and some other Cartesian concerns, more vivid and concrete.) It’s an illustration of how and why being bad at reductionism can cripple an AGI, and a demonstration of how easy it is to neglect reductionism while specifying what you want out of an AGI. (So it’s not a straw problem, and there isn’t an obvious cure-all patch.)
I’m also going to use AIXI as an illustration for some other issues in FAI (e.g., self-representation and AGI delegability), so explaining AIXI in some detail now lays gets more people on the same page for later.
doesn’t solving “naturalized induction” get us most of the way to being able to build an AGI already?
You may not need to solve naturalized induction to build a random UFAI. To build a FAI, I believe Eliezer thinks the largest hurdle is getting a recursively self-modifying agent to have stable specifiable preferences. That may depend on the AI’s decision theory, preferences, and external verifiability, or on aspects of its epistemology that don’t have much to do with the AI’s physicality.
Setting aside what counts as a ‘solution’, merging two problems counts as progress on the problem only when the merged version is easier to solve than the unmerged version. Or when the merged version helps us arrive at an important conceptual insight about the unmerged version. You can collapse every FAI problem into a single problem that we need to solve anyway by treating them all as components of its utility function or action policy, but it’s not clear that represents progress, and it’s very clear it doesn’t represent a solution.
Naturalized induction is the problem of defining an AGI’s priors, from the angle of attack ‘how can we naturalize this?’. In other words, it’s the problem of giving the AGI a reasonable epistemology, as informed by the insight that AGIs are physical processes that don’t differ in any fundamental way from other physical processes. So it encompasses and interacts with a lot of problems.
That should be clearer in my next couple of posts on naturalized induction. I used Solomonoff induction as my entry point because it keeps the sequence grounded in the literature and in a precise formalism. (And I used AIXI because it makes the problems with Solomonoff induction, and some other Cartesian concerns, more vivid and concrete.) It’s an illustration of how and why being bad at reductionism can cripple an AGI, and a demonstration of how easy it is to neglect reductionism while specifying what you want out of an AGI. (So it’s not a straw problem, and there isn’t an obvious cure-all patch.)
I’m also going to use AIXI as an illustration for some other issues in FAI (e.g., self-representation and AGI delegability), so explaining AIXI in some detail now lays gets more people on the same page for later.
You may not need to solve naturalized induction to build a random UFAI. To build a FAI, I believe Eliezer thinks the largest hurdle is getting a recursively self-modifying agent to have stable specifiable preferences. That may depend on the AI’s decision theory, preferences, and external verifiability, or on aspects of its epistemology that don’t have much to do with the AI’s physicality.