I’m expecting a future paradigm shift, and I don’t think it matters terribly much whether that next paradigm of AI is invented by humans, or by old-paradigm AIs, or by the two working together (although for the record I expect it to be mainly humans). At least, it doesn’t matter much for technical alignment. I guess it’s relevant for timelines.
Okay, the last time I was having this argument with a (different) someone, it was primarily about timelines, and they specifically disbelieved LLM-descendants would be capable of inventing the next paradigm. Sounds like you don’t have a strong take on that?
I totally agree there will be a new paradigm by the time we get to overwhelming superintelligence. I don’t think there is necessarily a new paradigm by the time we get to “human-reasoning-complete” AGI. Is that something you have a strong belief on?
I think LLMs are (and will always be) worse than humans at the kind of research involved in inventing a new AI paradigm. I think they’ll be helpful in the same way that PyTorch and GitHub and arXiv are helpful, but not involved in the way that’s similar to massively multiplying the quantity or quality of intellectual labor working on the problem. So when I think about timelines, I kinda assume business as usual, and am thinking loosely about historical analogies and how things seem to be going, and I wind up saying … I dunno, probably 5–25 years, or maybe more than 25 years, who knows, or maybe less than 5 years, who knows. I have a bit of discussion in §1.9 here (+ §1.4.1).
I think LLMs are (and will always be) worse than humans at the kind of research involved in inventing a new AI paradigm. I think they’ll be helpful in the same way that PyTorch and GitHub and arXiv are helpful, but not involved in the way that’s similar to massively multiplying the quantity or quality of intellectual labor working on the problem.
I basically agree with that description, as applied to the LLMs of 2026. However, I think that there’s a smooth-ish path in design space from here to actually-dangerous AGI. It seems plausible to me that the AI companies will be able to follow that path to its destination over the course of several years.
I’m expecting a future paradigm shift, and I don’t think it matters terribly much whether that next paradigm of AI is invented by humans, or by old-paradigm AIs, or by the two working together (although for the record I expect it to be mainly humans). At least, it doesn’t matter much for technical alignment. I guess it’s relevant for timelines.
Okay, the last time I was having this argument with a (different) someone, it was primarily about timelines, and they specifically disbelieved LLM-descendants would be capable of inventing the next paradigm. Sounds like you don’t have a strong take on that?
I totally agree there will be a new paradigm by the time we get to overwhelming superintelligence. I don’t think there is necessarily a new paradigm by the time we get to “human-reasoning-complete” AGI. Is that something you have a strong belief on?
I think LLMs are (and will always be) worse than humans at the kind of research involved in inventing a new AI paradigm. I think they’ll be helpful in the same way that PyTorch and GitHub and arXiv are helpful, but not involved in the way that’s similar to massively multiplying the quantity or quality of intellectual labor working on the problem. So when I think about timelines, I kinda assume business as usual, and am thinking loosely about historical analogies and how things seem to be going, and I wind up saying … I dunno, probably 5–25 years, or maybe more than 25 years, who knows, or maybe less than 5 years, who knows. I have a bit of discussion in §1.9 here (+ §1.4.1).
I basically agree with that description, as applied to the LLMs of 2026. However, I think that there’s a smooth-ish path in design space from here to actually-dangerous AGI. It seems plausible to me that the AI companies will be able to follow that path to its destination over the course of several years.