I was more certain of this before, but I’m less certain after Claude 4.6 Opus. Opus seems like “doing normal RL well just keeps working, even though the pretraining prior is strong”. If this approach just keeps working, then we could get ASI without the scenario I outlined happening.
If Anthropic has a secret technique that trains dense information into the model which explains Opus’ success, then that would be worrying. But considering how much effort they are putting into Personas, it seems like they believe pretraining is storng.
I guess im confused about this conceptually. To me AGI/ASI by definition does continual learning / long-long-horizon planning. What would it even mean otherwise?
Claude Opus 4.6 and other frontier models have gotten really, impressively good without continual learning, so it is possible that isn’t strictly necessary.
If continual learning is required for AGI, then there’s a lot of understudied (potentially unstudyable?) risk there.
This is Not Only Possible, this is Definitely Going To Happen- the only question is When.
Will most work that people do look hopelessly myopic and streetlighty? Yeah pretty much
I was more certain of this before, but I’m less certain after Claude 4.6 Opus. Opus seems like “doing normal RL well just keeps working, even though the pretraining prior is strong”. If this approach just keeps working, then we could get ASI without the scenario I outlined happening.
If Anthropic has a secret technique that trains dense information into the model which explains Opus’ success, then that would be worrying. But considering how much effort they are putting into Personas, it seems like they believe pretraining is storng.
I guess im confused about this conceptually. To me AGI/ASI by definition does continual learning / long-long-horizon planning. What would it even mean otherwise?
Claude Opus 4.6 and other frontier models have gotten really, impressively good without continual learning, so it is possible that isn’t strictly necessary.
If continual learning is required for AGI, then there’s a lot of understudied (potentially unstudyable?) risk there.