I agree with the key point, but for followup work, I’d ask—what is the pitch to robotics teams who are using diffusion for planning about how to constrain their objectives to not lose these benefits?
To restate the core ai safety point in current context, a very strong diffusion planner may be able to propagate structured logical consequences from much further in the future, depending on how strongly generalizing it is, and then be subject to goal mis-generalization. While the main generative objective does not specifically incentivize this, the approximate constraint satisfaction of diffusion would, by default, become more and more able to reason about long term consequences as it becomes able to do internal tokenization of those long term consequences. This sort of generalizing far out of training distribution is not something I’d expect from any sequential autoregressive model, because sequential autoregressive models only need to pull their activation manifolds into shapes that predict the most likely next element; but a diffusion model’s activation manifold is pushed to be stable in the face of all noise directions, which means a multiscale diffusion planner of the kind used in robotics could much more easily distill across examples.
I’ve actually never heard of diffusion for planning. Do you have a reference?
A diffusion model for text generation (like Diffusion-LM) still has the training objective to produce text from the training distribution, optimizing over only the current episode—in this case a short text.
I agree with the key point, but for followup work, I’d ask—what is the pitch to robotics teams who are using diffusion for planning about how to constrain their objectives to not lose these benefits?
To restate the core ai safety point in current context, a very strong diffusion planner may be able to propagate structured logical consequences from much further in the future, depending on how strongly generalizing it is, and then be subject to goal mis-generalization. While the main generative objective does not specifically incentivize this, the approximate constraint satisfaction of diffusion would, by default, become more and more able to reason about long term consequences as it becomes able to do internal tokenization of those long term consequences. This sort of generalizing far out of training distribution is not something I’d expect from any sequential autoregressive model, because sequential autoregressive models only need to pull their activation manifolds into shapes that predict the most likely next element; but a diffusion model’s activation manifold is pushed to be stable in the face of all noise directions, which means a multiscale diffusion planner of the kind used in robotics could much more easily distill across examples.
I’ve actually never heard of diffusion for planning. Do you have a reference?
A diffusion model for text generation (like Diffusion-LM) still has the training objective to produce text from the training distribution, optimizing over only the current episode—in this case a short text.
https://diffusion-planning.github.io/mobile.html