You seem overly anchored on COT as the only scaffolding system in the near-mid future (2-5 years). While I’m uncertain what specific architectures will emerge, the space of possible augmentations (memory systems, tool use, multi-agent interactions, etc.) seems vastly larger than current COT implementations.
COT (and particularly the extension tree of thoughts) seems like the strongest of those to me, probably because I can see an analogy to Solomonoff induction → AIXI. I am curious whether you have some particular more sophisticated memory system in mind?
My point is that these are all things that might work, but there is no strong reason to think they will—particularly to the extent of being all that we need. AI progress is usually on the scale of decades and often comes from unexpected places (though for the main line, ~always involving a neural net in some capacity).
Something like that seems like it would be a MVP of “actually try and get an LLM to come up with something significantly economically valuable. I expect that the lack of this type of experiment existing is because major AI labs feel like that would be choosing to exploit while there are still many gains to be made from exploring further architectural and scaffolding-esque improvements.
I find this kind of hard to swallow—a huge number of people are using and researching LLMs, I suspect that if something like this “just works” we would know by now. I mean, it would certainly win a lot of acclaim for the first group to pull it off, so the incentives seem sufficient—and it doesn’t seem that hard to pursue this in parallel to basic research on LLMs. Plus, the two investments are synergistic; for example, one would probably learn about the limitations of current models by pursuing this line. Maybe Anthropic is too small and focused to try it, but GDM could easily spin off a team.
Where you say “Certainly LLMs should be useful tools for coding, but perhaps not in a qualitatively different way than the internet is a useful tool for coding, and the internet didn’t rapidly set off a singularity in coding speed.”, I find this to be untrue both in terms of the impact of the internet (while it did not cause a short takeoff, it did dramatically increase the amount of new programmers and the effective transfer of information between them. I expect without it we would see computers having <20% of their current economic impact), and in terms of the current and expected future impact of LLM’s (LLM’s simply are widely used by smart/capable programmers. I trust them to evaluate if it is noticeably better than StackOverflow/the rest of the internet).
I expect LLMs to offer significant advantages above the internet. I am simply pointing out that not every positive feedback loop is a singularity. I expect great coding assistants (essentially excellent autocomplete) but not drop-in replacements for software engineers any time soon. This is one factor that will increase the pace of AI research somewhat, but also Moore’s law is running out, which will definitely slow the pace. Not sure which one wins out directionally.
COT (and particularly the extension tree of thoughts) seems like the strongest of those to me, probably because I can see an analogy to Solomonoff induction → AIXI. I am curious whether you have some particular more sophisticated memory system in mind?
My point is that these are all things that might work, but there is no strong reason to think they will—particularly to the extent of being all that we need. AI progress is usually on the scale of decades and often comes from unexpected places (though for the main line, ~always involving a neural net in some capacity).
I find this kind of hard to swallow—a huge number of people are using and researching LLMs, I suspect that if something like this “just works” we would know by now. I mean, it would certainly win a lot of acclaim for the first group to pull it off, so the incentives seem sufficient—and it doesn’t seem that hard to pursue this in parallel to basic research on LLMs. Plus, the two investments are synergistic; for example, one would probably learn about the limitations of current models by pursuing this line. Maybe Anthropic is too small and focused to try it, but GDM could easily spin off a team.
I expect LLMs to offer significant advantages above the internet. I am simply pointing out that not every positive feedback loop is a singularity. I expect great coding assistants (essentially excellent autocomplete) but not drop-in replacements for software engineers any time soon. This is one factor that will increase the pace of AI research somewhat, but also Moore’s law is running out, which will definitely slow the pace. Not sure which one wins out directionally.