My impression is that act-based approaches are good for human replacements, but not good for human meta-replacements. That is, if we consider the problem of fulfilling orders in an Amazon warehouse, we have a number of different problems:
Move a worker and a crate of goods adjacent to each other.
Move a single good from the crate to the order box.
Organize the facility to make the above two as cost-efficient as possible.
The first is already replaced by robots, the second seems like a good candidate for imitation (we need robot eyes and robot hands that are about as good as human eyes and hands, and they can work basically the same way) / low-level control theory.
But the third is a problem of cost functions, models, simuation, calculation, and possibility enumeration. It’s where creativity most comes into play, and that’s something I’m pessimistic about getting out of interpolative systems instead of extrapolative systems.
There are still major gains made by the reduced scope—a warehouse management system seems easier to make safe than a generic moneymaker—but I think there’s a fundamental connection between the opportunity and danger of AI (the ability to extrapolate into previously unseen realms).
Some things can be done by imitation based on our current understanding (and will get better as machine learning improves). The interesting part of the project is figuring out how to do the trickier things, which will require new ideas.
It’s not clear that imitation impairs your ability to generalize to new domains. An RL agent faces the question: in this new domain, how should I behave to receive rewards? It has not been trained in the domain, but must learn to reason about the domain and figure out what policies will work well. An imitation learner faces the question: in this new domain, how would the expert behave / what behavior would they approve of? The two questions seem similar in difficulty, and indeed you could use the same algorithmic ingredients.
It’s also not clear that it’s relevant if a task involves thinking about cost functions, models, simulation, calculation, etc… These are techniques one could apply either to achieve a high reward, or to produce actions the expert would approve of / like / do themselves. You might say that at that point these rich internal behaviors must be guided by some non-trivial internal dynamic. But then we will just have the same discussion a level lower.
I agree that those directions look promising.
My impression is that act-based approaches are good for human replacements, but not good for human meta-replacements. That is, if we consider the problem of fulfilling orders in an Amazon warehouse, we have a number of different problems:
Move a worker and a crate of goods adjacent to each other.
Move a single good from the crate to the order box.
Organize the facility to make the above two as cost-efficient as possible.
The first is already replaced by robots, the second seems like a good candidate for imitation (we need robot eyes and robot hands that are about as good as human eyes and hands, and they can work basically the same way) / low-level control theory.
But the third is a problem of cost functions, models, simuation, calculation, and possibility enumeration. It’s where creativity most comes into play, and that’s something I’m pessimistic about getting out of interpolative systems instead of extrapolative systems.
There are still major gains made by the reduced scope—a warehouse management system seems easier to make safe than a generic moneymaker—but I think there’s a fundamental connection between the opportunity and danger of AI (the ability to extrapolate into previously unseen realms).
Some things can be done by imitation based on our current understanding (and will get better as machine learning improves). The interesting part of the project is figuring out how to do the trickier things, which will require new ideas.
It’s not clear that imitation impairs your ability to generalize to new domains. An RL agent faces the question: in this new domain, how should I behave to receive rewards? It has not been trained in the domain, but must learn to reason about the domain and figure out what policies will work well. An imitation learner faces the question: in this new domain, how would the expert behave / what behavior would they approve of? The two questions seem similar in difficulty, and indeed you could use the same algorithmic ingredients.
It’s also not clear that it’s relevant if a task involves thinking about cost functions, models, simulation, calculation, etc… These are techniques one could apply either to achieve a high reward, or to produce actions the expert would approve of / like / do themselves. You might say that at that point these rich internal behaviors must be guided by some non-trivial internal dynamic. But then we will just have the same discussion a level lower.