I’m unsure about what’s the most important reason that explains the lack of significant progress in general-purpose robotics, even as other fields of AI have made great progress. I thought I’d write down some theories and some predictions each theory might make. I currently find each of these theories at least somewhat plausible.
The sim2real gap is large because our simulations differ from the real world along crucial axes, such as surfaces being too slippery. Here are some predictions this theory might make:
We will see very impressive simulated robots inside realistic physics engines before we see impressive robots in real life.
The most impressive robotic results will be the ones that used a lot of real-world data, rather than ones that had the most pre-training in simulation
Simulating a high-quality environment is too computationally expensive, since it requires simulations of deformable objects and liquids among other expensive-to-simulate features of the real world environment. Some predictions:
The vast majority of computation for training impressive robots will go into simulating the environment, rather than the learning part.
Impressive robots will only come after we figure out how to do efficient but passable simulations of currently expensive-to-simulate objects and environments.
Robotic hardware is not good enough to support agile and fluid movement. Some predictions:
We will see very impressive simulated robots before we see impressive robots in real life, but the simulated robots will use highly complex hardware that doesn’t exist in the real world
Impressive robotic results will only come after we have impressive hardware, such as robots that have 100 degrees of freedom.
People haven’t figured out that the scaling hypothesis works for robotics yet. Some predictions:
At some point we will see a ramp-up in the size of training runs for robots, and only after that will we see impressive robotics results
After robotic training runs reach the large-scale, real-world data will diminish greatly in importance, and approaches that leverage human domain knowledge like those from Boston Dynamics will quickly become obsolete
I like this list. Some other nonexclusive possibilities:
General purpose robotics need very low failure rates (or at least graceful failure) without supervision. Every application which has taken off (ChatGPT, Copilot, Midjourney) has human supervision, so failure is ok. So it is an artifact of none of AI handling failure well, rather than something to do with robots. Predictions:
—Even non-robot apps intended to have zero human supervision will have problems, i.e., maybe why adept.ai hasn’t launched?
Most of this progress is in SF. There’s just more engineers good at HPC and ML than at robots, and engineers are the bottleneck anyhow.
—Predicts Shenzhen or somewhere might start to do better.
I’m unsure about what’s the most important reason that explains the lack of significant progress in general-purpose robotics, even as other fields of AI have made great progress. I thought I’d write down some theories and some predictions each theory might make. I currently find each of these theories at least somewhat plausible.
The sim2real gap is large because our simulations differ from the real world along crucial axes, such as surfaces being too slippery. Here are some predictions this theory might make:
We will see very impressive simulated robots inside realistic physics engines before we see impressive robots in real life.
The most impressive robotic results will be the ones that used a lot of real-world data, rather than ones that had the most pre-training in simulation
Simulating a high-quality environment is too computationally expensive, since it requires simulations of deformable objects and liquids among other expensive-to-simulate features of the real world environment. Some predictions:
The vast majority of computation for training impressive robots will go into simulating the environment, rather than the learning part.
Impressive robots will only come after we figure out how to do efficient but passable simulations of currently expensive-to-simulate objects and environments.
Robotic hardware is not good enough to support agile and fluid movement. Some predictions:
We will see very impressive simulated robots before we see impressive robots in real life, but the simulated robots will use highly complex hardware that doesn’t exist in the real world
Impressive robotic results will only come after we have impressive hardware, such as robots that have 100 degrees of freedom.
People haven’t figured out that the scaling hypothesis works for robotics yet. Some predictions:
At some point we will see a ramp-up in the size of training runs for robots, and only after that will we see impressive robotics results
After robotic training runs reach the large-scale, real-world data will diminish greatly in importance, and approaches that leverage human domain knowledge like those from Boston Dynamics will quickly become obsolete
I like this list. Some other nonexclusive possibilities:
General purpose robotics need very low failure rates (or at least graceful failure) without supervision. Every application which has taken off (ChatGPT, Copilot, Midjourney) has human supervision, so failure is ok. So it is an artifact of none of AI handling failure well, rather than something to do with robots. Predictions: —Even non-robot apps intended to have zero human supervision will have problems, i.e., maybe why adept.ai hasn’t launched?
Most of this progress is in SF. There’s just more engineers good at HPC and ML than at robots, and engineers are the bottleneck anyhow. —Predicts Shenzhen or somewhere might start to do better.