I think a better question is, what does that mean? So many people throw around “GPT doesn’t really understand xyz” as if there’s a well-defined, unambiguous, universally-accepted notion of “understanding” that’s also separate from performance. Perhaps this kind of understanding is a coherent concept, but it’s not trivial!
I agree Rafael, and I apologize for carelessly using the term “understanding” as if it was an obvious term. I’ve tried to clarify my position in my comment below.
Their training objective is just to minimize next token prediction error, so there is no incentive for them to gain the ability to truly reason about the abstractions of code logic the way that we do. Their convincing ability to write code may nevertheless indicate that the underlying neural networks have learned representations that reflect the hierarchical structure of code and such. Under some forgiving definition of “understanding” we can perhaps claim that it understands the code logic. Personally I think that the reason GPT-4 can write quines is because there are countless examples of it on the internet, not because it has learned generally about code and is now able to leap into some new qualitatively different abstraction by itself. Personally, I have used GPT-3 as an assistant in a lot of machine learning projects and I always find that it breaks down whenever it is outside the dense clouds of data points it was trained on. If the project relates to the implementation of something quite simple but unique, performance suffers dramatically. All of my interactions with it on the code front indicate that it does very little in terms of generalisation and understanding of coding as a tool, and does everything in terms of superpowered autocompletion from a ridiculously large corpus of code.
How do we know that?
I think a better question is, what does that mean? So many people throw around “GPT doesn’t really understand xyz” as if there’s a well-defined, unambiguous, universally-accepted notion of “understanding” that’s also separate from performance. Perhaps this kind of understanding is a coherent concept, but it’s not trivial!
I agree Rafael, and I apologize for carelessly using the term “understanding” as if it was an obvious term. I’ve tried to clarify my position in my comment below.
Their training objective is just to minimize next token prediction error, so there is no incentive for them to gain the ability to truly reason about the abstractions of code logic the way that we do. Their convincing ability to write code may nevertheless indicate that the underlying neural networks have learned representations that reflect the hierarchical structure of code and such. Under some forgiving definition of “understanding” we can perhaps claim that it understands the code logic. Personally I think that the reason GPT-4 can write quines is because there are countless examples of it on the internet, not because it has learned generally about code and is now able to leap into some new qualitatively different abstraction by itself. Personally, I have used GPT-3 as an assistant in a lot of machine learning projects and I always find that it breaks down whenever it is outside the dense clouds of data points it was trained on. If the project relates to the implementation of something quite simple but unique, performance suffers dramatically. All of my interactions with it on the code front indicate that it does very little in terms of generalisation and understanding of coding as a tool, and does everything in terms of superpowered autocompletion from a ridiculously large corpus of code.