I agree that (B) never happened. Another way of saying this, is that “algorithms for discovering algorithms” have only ever been written by humans, and never directly discovered by another “algorithm for discovering algorithms.”
The LLM+RL “algorithm for discovering algorithms” is far less powerful than the simple core of intelligence, but far more powerful than any other “algorithm for discovering algorithms” we ever had before. Since it has discovered the algorithms for solving IMO level math problems.
Meanwhile, the simple core of intelligence may also be the easiest “algorithm for discovering algorithms” to discover (by another such algorithm). This is because evolution found it (and the entire algorithm fits inside the human genome), and the algorithm seems to be simple. The first time (B) happens, may be the only time (B) happens (before superintelligence).
I think it’s both plausible that the simple core of intelligence is found by human researchers, and that it just emerges inside a LLM with much greater effective scale (due to being both bigger and more efficient), subject to much greater amounts of chain-of-thought RL.
To be honest I’m very unsure about all of this.
I agree that (B) never happened. Another way of saying this, is that “algorithms for discovering algorithms” have only ever been written by humans, and never directly discovered by another “algorithm for discovering algorithms.”
The LLM+RL “algorithm for discovering algorithms” is far less powerful than the simple core of intelligence, but far more powerful than any other “algorithm for discovering algorithms” we ever had before. Since it has discovered the algorithms for solving IMO level math problems.
Meanwhile, the simple core of intelligence may also be the easiest “algorithm for discovering algorithms” to discover (by another such algorithm). This is because evolution found it (and the entire algorithm fits inside the human genome), and the algorithm seems to be simple. The first time (B) happens, may be the only time (B) happens (before superintelligence).
I think it’s both plausible that the simple core of intelligence is found by human researchers, and that it just emerges inside a LLM with much greater effective scale (due to being both bigger and more efficient), subject to much greater amounts of chain-of-thought RL.