At a past Singularity Summit, Juergen Schmidhuber thought that “improve compression of sensory data” would motivate an AI to do science and create art.
It’s true that, relative to doing nothing to understand the environment, doing science or creating art might increase the degree to which sensory information can be compressed.
But the maximum of this utility function comes from creating environmental subagents that encrypt streams of all 0s or all 1s, and then reveal the encryption key. It’s possible that Schmidhuber’s brain was reluctant to really actually search for an option for “maximizing sensory compression” that would be much better at fulfilling that utility function than art, science, or other activities that Schmidhuber himself ranked high in his preference ordering.
Specifically, parts of the talk suggest that nice things like affection, friendship, love, play, curiosity, anger, envy, democracy, liberalism, etc. are the global maxima of competitive forces in a post-AGI age (or at least, might be), whereas I think they aren’t. Merging / making copies [I sometimes call this “zombie dynamics”, in that an AGI that gets more chips will get more copies of itself to go after more resources, like a zombie horde making more zombies] has a lot to do with that, but so does simply being strategic. That gets us to a different issue:
In my mind, there are two quite different dichotomies:
The first dichotomy is:
(1A) “I intrinsically care about X [e.g. friendship] for its own sake”, versus
(1B) “X-type behaviors are instrumentally useful for accomplishing some other goal”
The second dichotomy is:
(2A) “I figured out a while ago that X-type behaviors are instrumentally useful for accomplishing some other goal, and now just carry on with X-type behaviors in such-and-such situation without thinking too hard, because there’s no point in reinventing the wheel every time”, versus
(2B) “I reason from first principles each time that X-type behaviors are instrumentally useful for accomplishing some other goal”.
You seem to use the term “amortised inference” to lump these two dichotomies together, whereas I would prefer to use that term just for (2A). Or if you like:
the first dichotomy is between (1A) Things figured out by evolution, versus (1B) Things figured out within a lifetime; while
the second dichotomy is between (2A) things figured out earlier in life, versus (2B) things figured out just now.
I think the (2A) things are extremely fragile, totally different from the (1A) things which are highly robust. For example, when I was a kid, I learned a (2A) heuristic that I should ask my parents to drive me places. Then I got older, and that heuristic stopped serving me well, so I almost immediately stopped using it. Likewise, I used a certain kind of appointment calendar for 5 years, and then someone suggested that I should switch to a different kind of appointment calendar, and I thought about it a bit, and decided they were right, and switched. I have a certain way of walking, that I’ve been using unthinkingly for decades, but if you put me in high heels, I would immediately drop that habit and learn a new one. These things are totally routine.
The discussion of “amortised inference” in the post makes it sound like a tricky thing that requires superintelligence, but it’s not. The dumbest person you know uses probably thousands or millions of implicit heuristics every day, and is able to flexibly update any of them, or add exceptions, when the situation changes such that the heuristic stops being instrumentally useful.
…Then there’s a normative dimension to all this. If people are cooperating on (2B) grounds versus (2A) grounds, I really don’t care, at least not in itself. If the situation changes such that cooperating stops being instrumentally useful, the (2B) people will immediately stab their former allies in the back, whereas the (2A) people might take a bit longer before the idea pops into their heads that it’s a great idea to stab their former allies in the back. I don’t really care, neither of these is real friendship. By contrast, real friendship has to be (1A), and I do care about real friendship existing into the distant future.
I’ve been talking about cooperation, but it’s equally true that once a trained AI surpasses some level of strategic and metacognitive competence, it doesn’t need curiosity (see Soares post), or play, anger, etc. It can figure out that it should do all those things strategically, and I don’t think it’s any harder than figuring out quantum mechanics etc.
Parts of this post seem close to an error that Yudkowsky accused Schmidhuber of making:
Specifically, parts of the talk suggest that nice things like affection, friendship, love, play, curiosity, anger, envy, democracy, liberalism, etc. are the global maxima of competitive forces in a post-AGI age (or at least, might be), whereas I think they aren’t. Merging / making copies [I sometimes call this “zombie dynamics”, in that an AGI that gets more chips will get more copies of itself to go after more resources, like a zombie horde making more zombies] has a lot to do with that, but so does simply being strategic. That gets us to a different issue:
In my mind, there are two quite different dichotomies:
The first dichotomy is:
(1A) “I intrinsically care about X [e.g. friendship] for its own sake”, versus
(1B) “X-type behaviors are instrumentally useful for accomplishing some other goal”
The second dichotomy is:
(2A) “I figured out a while ago that X-type behaviors are instrumentally useful for accomplishing some other goal, and now just carry on with X-type behaviors in such-and-such situation without thinking too hard, because there’s no point in reinventing the wheel every time”, versus
(2B) “I reason from first principles each time that X-type behaviors are instrumentally useful for accomplishing some other goal”.
You seem to use the term “amortised inference” to lump these two dichotomies together, whereas I would prefer to use that term just for (2A). Or if you like:
the first dichotomy is between (1A) Things figured out by evolution, versus (1B) Things figured out within a lifetime; while
the second dichotomy is between (2A) things figured out earlier in life, versus (2B) things figured out just now.
I think the (2A) things are extremely fragile, totally different from the (1A) things which are highly robust. For example, when I was a kid, I learned a (2A) heuristic that I should ask my parents to drive me places. Then I got older, and that heuristic stopped serving me well, so I almost immediately stopped using it. Likewise, I used a certain kind of appointment calendar for 5 years, and then someone suggested that I should switch to a different kind of appointment calendar, and I thought about it a bit, and decided they were right, and switched. I have a certain way of walking, that I’ve been using unthinkingly for decades, but if you put me in high heels, I would immediately drop that habit and learn a new one. These things are totally routine.
The discussion of “amortised inference” in the post makes it sound like a tricky thing that requires superintelligence, but it’s not. The dumbest person you know uses probably thousands or millions of implicit heuristics every day, and is able to flexibly update any of them, or add exceptions, when the situation changes such that the heuristic stops being instrumentally useful.
…Then there’s a normative dimension to all this. If people are cooperating on (2B) grounds versus (2A) grounds, I really don’t care, at least not in itself. If the situation changes such that cooperating stops being instrumentally useful, the (2B) people will immediately stab their former allies in the back, whereas the (2A) people might take a bit longer before the idea pops into their heads that it’s a great idea to stab their former allies in the back. I don’t really care, neither of these is real friendship. By contrast, real friendship has to be (1A), and I do care about real friendship existing into the distant future.
I’ve been talking about cooperation, but it’s equally true that once a trained AI surpasses some level of strategic and metacognitive competence, it doesn’t need curiosity (see Soares post), or play, anger, etc. It can figure out that it should do all those things strategically, and I don’t think it’s any harder than figuring out quantum mechanics etc.
I agree, and would also point out that since:
...this intrinsic value [friendship] is in place and leads to cooperation (an instrumental value).
Very different than the model that says: competition → cooperation → the value [friendship].