This post provides a good overview of some topics I think need attention by the ‘AI policy’ people at national levels. AI policy (such as the US and UK AISI groups) has been focused on generative AI and recently agentic AI to understand near-term risks. Whether we’re talking LLM training and scaffolding advances, or a new AI paradigm, there is new risk when AI begins to learn from experiments in the world or reasoning about its own world model. In child development, imitation learning focuses on learning from examples, while constructivist learning focuses on learning by reflecting on interactions with the world. Constructivist learning is, I expect, key to push past AGI to ASI and caries obvious risks to alignment beyond imitation learning.
In general, I expect something LLM-like (i.e. transformer models or an improved derivative) to be able to reach ASI with a proper learning-by-doing structure. But I also expect ASI could find and implement a more efficient intelligence algorithm once ASI exists.
1.4.1 Possible counter: “If a different, much more powerful, AI paradigm existed, then someone would have already found it.”
This paragraph tries to provide some data for a probability estimate of this point. AI as a field has been around at least since the Dartmouth conference in 1956. In this time we’ve had Eliza, Deep Blue, Watson, and now transformer-based models including OpenAI o3-pro. In support of Steven’s position, one could note that AI research publications are much higher now that during the previous 70 years, but at the same time many AI ideas have been explored and the current best results are with models based on the 8-year-old “Attention is all you need” paper. To get a sense for the research rate, we can note that the doubling time for AI/ML research papers per month was about 2 years between 1994 and 2023 according to this Nature paper. Hence, every 2 years we have about as many papers as created in the last 70 years. I don’t expect this doubling can continue forever, but certainly many new ideas are being explored now. If a ‘simple model’ for AI exists and it’s discovery is, say, randomly positioned on a given AI/ML research paper published between 1956 and ASI achievement then one could estimate the probability of the paper’s position using this simplistic research model. If ASI is only 6 years out and the doubling every 2 years continues, then almost 90% of the AI/ML research papers before ASI are still in the future. Even though many of these papers are LLM focused, there is still active work in alternative areas. But even though the foundational paper for ASI may yet be in our future, I would expect something like a ‘complex’ ML model will win out (for example, Yann LeCun’s ideas involve differentiable brain modules). And the solution may or may not be more compute-intensive than current models. The brain compute estimates vary widely and the human brain has been optimized by evolution for many generations. In short, it seems reasonable to expect another key idea before ASI, but I would not expect it to be a simple model.
This post provides a good overview of some topics I think need attention by the ‘AI policy’ people at national levels. AI policy (such as the US and UK AISI groups) has been focused on generative AI and recently agentic AI to understand near-term risks. Whether we’re talking LLM training and scaffolding advances, or a new AI paradigm, there is new risk when AI begins to learn from experiments in the world or reasoning about its own world model. In child development, imitation learning focuses on learning from examples, while constructivist learning focuses on learning by reflecting on interactions with the world. Constructivist learning is, I expect, key to push past AGI to ASI and caries obvious risks to alignment beyond imitation learning.
In general, I expect something LLM-like (i.e. transformer models or an improved derivative) to be able to reach ASI with a proper learning-by-doing structure. But I also expect ASI could find and implement a more efficient intelligence algorithm once ASI exists.
This paragraph tries to provide some data for a probability estimate of this point. AI as a field has been around at least since the Dartmouth conference in 1956. In this time we’ve had Eliza, Deep Blue, Watson, and now transformer-based models including OpenAI o3-pro. In support of Steven’s position, one could note that AI research publications are much higher now that during the previous 70 years, but at the same time many AI ideas have been explored and the current best results are with models based on the 8-year-old “Attention is all you need” paper. To get a sense for the research rate, we can note that the doubling time for AI/ML research papers per month was about 2 years between 1994 and 2023 according to this Nature paper. Hence, every 2 years we have about as many papers as created in the last 70 years. I don’t expect this doubling can continue forever, but certainly many new ideas are being explored now. If a ‘simple model’ for AI exists and it’s discovery is, say, randomly positioned on a given AI/ML research paper published between 1956 and ASI achievement then one could estimate the probability of the paper’s position using this simplistic research model. If ASI is only 6 years out and the doubling every 2 years continues, then almost 90% of the AI/ML research papers before ASI are still in the future. Even though many of these papers are LLM focused, there is still active work in alternative areas. But even though the foundational paper for ASI may yet be in our future, I would expect something like a ‘complex’ ML model will win out (for example, Yann LeCun’s ideas involve differentiable brain modules). And the solution may or may not be more compute-intensive than current models. The brain compute estimates vary widely and the human brain has been optimized by evolution for many generations. In short, it seems reasonable to expect another key idea before ASI, but I would not expect it to be a simple model.