(Somewhat rambly response.) You’re definitely on to something here. I’ve long felt that while deep learning may not be AGI, the machine learning industry sure seems ‘general’ in its ability to solve almost any well defined problem, and the only limitation is that the most difficult problems cannot be so easily put into a series of testcases. Once there is some metric to guide them, the ML guys can arrive at seemingly any destination in a relatively short order. But there are still some problems outside their reach where we can at least attempt to define a score:
Nethack and other complex games. The ideal game for testing autonomy under nostalgebraist’s definition would show a lot of information, most of which is irrelevant, but at least some of which is very important, and the AI has to know which information to remember (and/or how to access relevant information e.g. via game menus or the AI’s own database of memories)
Strong versions of the Turing Test, especially where the challenge is to maintain a stable personality over a long conversation, with a consistent (fake) personal history, likes and dislikes, things the personality knows and doesn’t know, etc.
Autonomously create a complex thing: a long novel, a full-length movie, or a complex computer program or game. A proof of success would be getting a significant amount of attention from the public such that you could plausibly earn money using AI-generated content.
The Wozniak Test: make and present a cup of coffee using the tools of an unfamiliar kitchen. Given that pick-and-place is now relatively routine, this seems within reach to me, but it would require chaining together multiple pick-and-place tasks in a way that robotics AIs really struggle with today. I would not be surprised if, within the next year, there is an impressive demonstration of this kind of a robomaid task, seeming passing the Wozniak Test, but then the robot never arrives for consumers because the robomaid only works in the demo home that they set up, or only works 20% of the time, or takes an hour to make coffee, or something.
I think there are also two notions of intelligence that need to be differentiated here. One is the intelligence that does pattern matching on what seems to be totally arbitrary data, and generating some kind of response. This kind of intelligence has an abstract quality to it. The other kind of intelligence is being able to be instructed (by humans, or an instruction manual), or to do transfer learning (understanding that technology will make your nation in Civilization VI more powerful from having trained on language data of the history of real civilization, for example).
I saw someone ask recently, “Do humans really do anything zero shot?” If we can’t actually zero shot things, then we shouldn’t expect an AI to be able to do so. (It might be impossible under a future Theory of Intelligence.) If we actually can zero shot anything, we must either have been instructed or were able to reason from context. By definition, you can’t do supervised learning from zero examples. Humans probably can do the more abstract, pattern matching type of intelligence, but my guess is that is somewhat rare and only happens in totally new domains, and only after reasoning has failed, because it’s essentially guesswork.
(Somewhat rambly response.) You’re definitely on to something here. I’ve long felt that while deep learning may not be AGI, the machine learning industry sure seems ‘general’ in its ability to solve almost any well defined problem, and the only limitation is that the most difficult problems cannot be so easily put into a series of testcases. Once there is some metric to guide them, the ML guys can arrive at seemingly any destination in a relatively short order. But there are still some problems outside their reach where we can at least attempt to define a score:
Nethack and other complex games. The ideal game for testing autonomy under nostalgebraist’s definition would show a lot of information, most of which is irrelevant, but at least some of which is very important, and the AI has to know which information to remember (and/or how to access relevant information e.g. via game menus or the AI’s own database of memories)
Strong versions of the Turing Test, especially where the challenge is to maintain a stable personality over a long conversation, with a consistent (fake) personal history, likes and dislikes, things the personality knows and doesn’t know, etc.
Autonomously create a complex thing: a long novel, a full-length movie, or a complex computer program or game. A proof of success would be getting a significant amount of attention from the public such that you could plausibly earn money using AI-generated content.
The Wozniak Test: make and present a cup of coffee using the tools of an unfamiliar kitchen. Given that pick-and-place is now relatively routine, this seems within reach to me, but it would require chaining together multiple pick-and-place tasks in a way that robotics AIs really struggle with today. I would not be surprised if, within the next year, there is an impressive demonstration of this kind of a robomaid task, seeming passing the Wozniak Test, but then the robot never arrives for consumers because the robomaid only works in the demo home that they set up, or only works 20% of the time, or takes an hour to make coffee, or something.
I think there are also two notions of intelligence that need to be differentiated here. One is the intelligence that does pattern matching on what seems to be totally arbitrary data, and generating some kind of response. This kind of intelligence has an abstract quality to it. The other kind of intelligence is being able to be instructed (by humans, or an instruction manual), or to do transfer learning (understanding that technology will make your nation in Civilization VI more powerful from having trained on language data of the history of real civilization, for example).
I saw someone ask recently, “Do humans really do anything zero shot?” If we can’t actually zero shot things, then we shouldn’t expect an AI to be able to do so. (It might be impossible under a future Theory of Intelligence.) If we actually can zero shot anything, we must either have been instructed or were able to reason from context. By definition, you can’t do supervised learning from zero examples. Humans probably can do the more abstract, pattern matching type of intelligence, but my guess is that is somewhat rare and only happens in totally new domains, and only after reasoning has failed, because it’s essentially guesswork.