My question is more pre-takeoff. Specifically, how does an AI that is making additional AIs even know when it has one “better” than itself? What does it even mean to be “better” at making an AI?
And none of that addresses the completely separate question of an AI picking which task to do. A program that recursively writes programs that write programs that are faster/smaller/some other objective metric makes sense. But what good is having the smallest, fastest, small+fast program writer possible?
“Better” in this case depends on the AI’s original utility function. An AI is “better” than its predecessor if it can score more highly on that utility function. As you said, better is arbitrary, but “better with respect to a particular goal” isn’t.
In addition, things like speed, efficiency, correctness, having more copies of yourself, and cognitive capacity are all useful for a wide variety of goals—an AI that is ten times as powerful, ten times faster, significantly more accurate, or capable of understanding more sophisticated concepts as compared to another AI with the same goal is likely to be better at achieving it. This is true for almost any goal you can imagine, whether it be “Create paperclips” or “Maximise human flourishing”.
So, again, how can an AI know which child AI will best fulfill its utility function without being able to observe them in action, which for any sort of long-term goal is necessarily going to be slow?
If the goal is “make the most paperclips per day” the minimum cycle time to evaluate the performance of a child AI is one day. Maybe more if it wants an n > 1 to sample from each AI. And where is this AI getting the resources to try multiple AIs worth of systems? And also, too, if the goal is something complex like “acquire power and/or money” you now have multiple child AIs competing with each-other to better extract resources, which is a different environment than a single AI working to maximize its power/$.
My question is more pre-takeoff. Specifically, how does an AI that is making additional AIs even know when it has one “better” than itself? What does it even mean to be “better” at making an AI? And none of that addresses the completely separate question of an AI picking which task to do. A program that recursively writes programs that write programs that are faster/smaller/some other objective metric makes sense. But what good is having the smallest, fastest, small+fast program writer possible?
“Better” in this case depends on the AI’s original utility function. An AI is “better” than its predecessor if it can score more highly on that utility function. As you said, better is arbitrary, but “better with respect to a particular goal” isn’t.
In addition, things like speed, efficiency, correctness, having more copies of yourself, and cognitive capacity are all useful for a wide variety of goals—an AI that is ten times as powerful, ten times faster, significantly more accurate, or capable of understanding more sophisticated concepts as compared to another AI with the same goal is likely to be better at achieving it. This is true for almost any goal you can imagine, whether it be “Create paperclips” or “Maximise human flourishing”.
So, again, how can an AI know which child AI will best fulfill its utility function without being able to observe them in action, which for any sort of long-term goal is necessarily going to be slow? If the goal is “make the most paperclips per day” the minimum cycle time to evaluate the performance of a child AI is one day. Maybe more if it wants an n > 1 to sample from each AI. And where is this AI getting the resources to try multiple AIs worth of systems? And also, too, if the goal is something complex like “acquire power and/or money” you now have multiple child AIs competing with each-other to better extract resources, which is a different environment than a single AI working to maximize its power/$.