maybe our last hope for understanding what these things are doing before they turn into completely opaque superintelligence, please God let it keep working.”
*
We got unbelievably, preposterously, cosmically lucky with Chain-of-Thought.
Actually there is an important point to make here. We didn’t get so lucky, of course.
Rather CoT was the logical next step, because, as you just explained, it is more powerful to build chains of reasoning than optimize every single output ad infinitum.
We will get far, but never get to SI, with only an LLM, exactly because they lack integrated learnings and continuous states.
An LLM designing the start of a new SI-seed architecture paradigm is one (speculative) thing, but neither whole-models nor latent mesa-optimizers inside them, can accidentally turn into SI on their own (with or without agentic scaffolding)
You can scale for some time, probably even to strong AI using CoT. But for SI you need self-optimization that can scale. Functional, fluid intelligence requires reasoning over time, not just endless optimization. It requires continuity, integration, and, for some endeavours, stakes (to assess feedback).
Actually there is an important point to make here. We didn’t get so lucky, of course.
Rather CoT was the logical next step, because, as you just explained, it is more powerful to build chains of reasoning than optimize every single output ad infinitum.
We will get far, but never get to SI, with only an LLM, exactly because they lack integrated learnings and continuous states.
An LLM designing the start of a new SI-seed architecture paradigm is one (speculative) thing, but neither whole-models nor latent mesa-optimizers inside them, can accidentally turn into SI on their own (with or without agentic scaffolding)
You can scale for some time, probably even to strong AI using CoT. But for SI you need self-optimization that can scale. Functional, fluid intelligence requires reasoning over time, not just endless optimization. It requires continuity, integration, and, for some endeavours, stakes (to assess feedback).