i’m starting to think recursive self improvement is basically already possible with LLMs, even without anymore training ever. I’m pretty shocked with how much better my coding LLMs have become just by taking care to give the LLMs the right meta-context and information systems. I feel like I’ve moved from prompting, to figuring out what context is needed in addition to the prompt, to spending a bunch of time/effort building a knowledge structure so that the LLM can figure out its own context to get whatever done, and thats moved me from having LLMs write functions and scripts to large multi-file chunks of entire repositories. And I’m continually having the thought that’s like “ok but now I’m building this knowledge system that it can traverse and decide its own relevant context, but why can’t the LLM do that too? what would i need to setup for it to do that?” and i’m starting to feel like that’s a never ending thing.
After thinking about this a bit more, the main point I’d want to make is less about recursive self improvement and more that just there’s a lot more capability in these models than people realize.
Whether that capacity is enough for recursive self improvement is another question that I’m not certain about either way but I think it’s at least plausible that it might be. I will note that humanity improves in its knowledge and capability without architectural change. That’s a rough analogy to the type of improvement I’m imagining.
i’m starting to think recursive self improvement is basically already possible with LLMs, even without anymore training ever. I’m pretty shocked with how much better my coding LLMs have become just by taking care to give the LLMs the right meta-context and information systems. I feel like I’ve moved from prompting, to figuring out what context is needed in addition to the prompt, to spending a bunch of time/effort building a knowledge structure so that the LLM can figure out its own context to get whatever done, and thats moved me from having LLMs write functions and scripts to large multi-file chunks of entire repositories. And I’m continually having the thought that’s like “ok but now I’m building this knowledge system that it can traverse and decide its own relevant context, but why can’t the LLM do that too? what would i need to setup for it to do that?” and i’m starting to feel like that’s a never ending thing.
After thinking about this a bit more, the main point I’d want to make is less about recursive self improvement and more that just there’s a lot more capability in these models than people realize.
Whether that capacity is enough for recursive self improvement is another question that I’m not certain about either way but I think it’s at least plausible that it might be. I will note that humanity improves in its knowledge and capability without architectural change. That’s a rough analogy to the type of improvement I’m imagining.