My pet AGI strategy, as a 12 year old in ~2018, was to build sufficiently advanced general world models (from YT videos etc.), then train an RL policy on said world model (to then do stuff in the actual world). A steelman of 12-year-old me would point out that video modeling has much better inductive biases than language modeling for robotics and other physical (and maybe generally agentic) tasks, though language modeling fundamentally is a better task for teaching machines language (duh!) and reasoning (mathematical proofs aren’t physical objects, nor encoded in the laws of physics).
OpenAI’s Sora models (and also DeepMind’s Genie and similar) very much seems like a backup investment in this type of AGI (or at least transformative narrow robotics AI), so I don’t think this is good for reducing OpenAI’s funding (robots would be a very profitable product class), nor influence (obv. a social network gives a lot of influence, to e.g. prevent an AI pause or to move the public towards pro-AGI views).
In any scenario, Sora 2 seems to me as a net-negative activity for AI safety:
if LLMs are the way to AGI (which I believe is the case), then we will probably die, but with a more socially influential OpenAI that potentially has robots (than if Sora 2 didn’t exist); the power OpenAI would have in this scenario to prevent an AI pause seems to outweigh the slowdown that would be caused by the marginal amounts of compute Sora 2 uses
if LLMs aren’t the way to AGI (unlikely), but world modeling based on videos is (also unlikely), then Sora 2 is very bad—you would want OpenAI to train more LLMs and not invest in world models which lead to unaligned AGI/ASI.
if neither LLMs or world modeling is the way to AGI (also unlikely), then OpenAI probably isn’t using any compute to do ‘actual’ AGI research (what else do they do?); so Sora 2 wouldn’t be affecting the progress of AGI, but it would be increasing the influence of OpenAI; and having highly influential AI companies is probably bad for global coordination over AGI safety. Also, OpenAI may have narrow (and probably safe) robotics AI in this scenario, but progress in AI alignment probably isn’t constrained in any measurable way by physically moving or doing things; though maybe indirect impacts from increased economic growth could cause slightly faster AI alignment progress, by reducing funding constraints?
I think that the path to AGI involves LLMs/automated ML research, and the first order effects of diverting compute away from this still seem large. I think OpenAI is bottlenecked more by a lack of compute (and Nvidia release cycles), than by additional funding from robotics. And I hope I’m wrong, but I think the pause movement won’t be large enough to make a difference. The main benefit in my view comes if it’s a close race with Anthropic, where I think slowing OpenAI down seems net positive and decreases the chances we die by a bit. If LLMs aren’t the path to AGI, then I agree with you completely. So overall it’s hard to say, I’d guess it’s probably neutral or slightly positive still.
Of course, both paths are bad, and I wish they would invest this compute into alignment research, as they promised!
My pet AGI strategy, as a 12 year old in ~2018, was to build sufficiently advanced general world models (from YT videos etc.), then train an RL policy on said world model (to then do stuff in the actual world).
A steelman of 12-year-old me would point out that video modeling has much better inductive biases than language modeling for robotics and other physical (and maybe generally agentic) tasks, though language modeling fundamentally is a better task for teaching machines language (duh!) and reasoning (mathematical proofs aren’t physical objects, nor encoded in the laws of physics).
OpenAI’s Sora models (and also DeepMind’s Genie and similar) very much seems like a backup investment in this type of AGI (or at least transformative narrow robotics AI), so I don’t think this is good for reducing OpenAI’s funding (robots would be a very profitable product class), nor influence (obv. a social network gives a lot of influence, to e.g. prevent an AI pause or to move the public towards pro-AGI views).
In any scenario, Sora 2 seems to me as a net-negative activity for AI safety:
if LLMs are the way to AGI (which I believe is the case), then we will probably die, but with a more socially influential OpenAI that potentially has robots (than if Sora 2 didn’t exist); the power OpenAI would have in this scenario to prevent an AI pause seems to outweigh the slowdown that would be caused by the marginal amounts of compute Sora 2 uses
if LLMs aren’t the way to AGI (unlikely), but world modeling based on videos is (also unlikely), then Sora 2 is very bad—you would want OpenAI to train more LLMs and not invest in world models which lead to unaligned AGI/ASI.
if neither LLMs or world modeling is the way to AGI (also unlikely), then OpenAI probably isn’t using any compute to do ‘actual’ AGI research (what else do they do?); so Sora 2 wouldn’t be affecting the progress of AGI, but it would be increasing the influence of OpenAI; and having highly influential AI companies is probably bad for global coordination over AGI safety.
Also, OpenAI may have narrow (and probably safe) robotics AI in this scenario, but progress in AI alignment probably isn’t constrained in any measurable way by physically moving or doing things; though maybe indirect impacts from increased economic growth could cause slightly faster AI alignment progress, by reducing funding constraints?
Thanks, these are good points!
I think that the path to AGI involves LLMs/automated ML research, and the first order effects of diverting compute away from this still seem large. I think OpenAI is bottlenecked more by a lack of compute (and Nvidia release cycles), than by additional funding from robotics. And I hope I’m wrong, but I think the pause movement won’t be large enough to make a difference. The main benefit in my view comes if it’s a close race with Anthropic, where I think slowing OpenAI down seems net positive and decreases the chances we die by a bit. If LLMs aren’t the path to AGI, then I agree with you completely. So overall it’s hard to say, I’d guess it’s probably neutral or slightly positive still.
Of course, both paths are bad, and I wish they would invest this compute into alignment research, as they promised!