Then on X, I noticed that today’s news included an announcement by Anthropic that one of their employees had last week managed to radically improve a bound related to the Riemann hypothesis
Apparently that result is also not an instance of “truly creative math research”, much like none of OpenAI’s latest ones. There appears to be some intuitive distinction between math results that are just “recombinations of known ideas” (possibly very sophisticated recombinations!), and math results that feel “truly novel”, and the mathematical consensus is apparently that none of AI math results so far are of the latter type.
It’s possible this is just a particularly memetically fit/convergent cope. Perhaps not, though; perhaps this intuition is pointing at something real. Notably, this is a take apparently shared by Noam Brown, who only expects LLMs to produce “entirely new mathematical machinery” “within two years”.
I don’t claim to understand what’s going on there, or why LLMs would be limited in this manner (whatever “this manner” even means). As bearish on LLMs as I am, I don’t see why RLVR wouldn’t suffice to let them “solve” formal mathematics. Yet, something’s up with that, I think.
My guess is that Sol wasn’t being “truly creative” in your case as well.
Apparently that result is also not an instance of “truly creative math research”, much like none of OpenAI’s latest ones.
This I tend to agree with.
“truly novel”, and the mathematical consensus is apparently that none of AI math results so far are of the latter type
I’d mildly disagree, in that the recent counterexamples (Jacobian Conjecture and Period-Index conjecture, the former purely AI, the latter very much Human-AI collaboration) are a third category that AI seems surprisingly good at: finding examples of stuff. Not novel theory, but personally I find it hard to deny there’s a novel capability there. At least useful.
For context; my PhD research was partly aimed at verifying cases of BSD, and I care(d?) a fair amount about period-index.
There appears to be some intuitive distinction between math results that are just “recombinations of known ideas” (possibly very sophisticated recombinations!), and math results that feel “truly novel”, and the mathematical consensus is apparently that none of AI math results so far are of the latter type.
I think the conclusion is correct here but the model is not quite right. Most (or all) ideas originate from previous concepts and information, passed through a variable amount of reprocessing and abstraction. People understood the concepts of slope and area before the invention of calculus. Of course calculus took plenty of “creativity” to find, which I believe should be judged by the degree of transformation. It appears that LLMs are pretty good at combining known mathematical techniques in straightforward “right-out-of-the-packaging” ways e.g. a property of object X satisfies the assumptions of theorem Y, so X is an example of conjecture Z (exactly what happened with non-sofic groups btw). These can still be surprises, since the LLM has access to such an incredibly large array of techniques! As far as I can tell, however, they haven’t demonstrated the ability to abstract ideas further to the extent of being considered “new perspectives”.
I don’t claim to understand what’s going on there, or why LLMs would be limited in this manner (whatever “this manner” even means). As bearish on LLMs as I am, I don’t see why RLVR wouldn’t suffice to let them “solve” formal mathematics. Yet, something’s up with that, I think.
RLVR for mathematics is not amenable to the same kind of exploration that you can do for board games like Chess and Go. In those cases you can basically harvest endless data from MCTS self-play, and thereby empirically discover winning patterns and new ideas that can be used at runtime. For math, the analogue would require the training process to somehow harvest all of the ideas that the model would need at inference to solve whatever research problem it might be posed with—essentially it would need to do the operative part of that original research ahead of time. This does not seems remotely plausible to me. Indeed RLVR for LLMs appears to be limited to improving sampling of known ideative pathways that appear in the pretraining. This last point was discussed really nicely by both Steven Byrnes and Beren Millidgerecently, with some links to the relevant papers.
My view is that LLMs seem to be restricted in certain important aspects of math and other formal subjects by a lack of “conceptual fluidity”. I think this is the same blocker that you’ve described quite cogently in your expectations for LLM progress in hard-to-verify research. Actually I don’t think that the formal character of math makes that much of a difference in this regard! Sure, it makes it easier for the models to practice, but the most important driver of progress in math is still the development of qualitative concepts, and we don’t seem to have a way of generating such concepts in training (except for massive-scale empirics in the case of board games).
This is just a continuation of the “God of the gaps” argument: “If a computer can do it, it’s not intelligence.” Overtime, the difference between what a computer can do and “true intelligence” becomes exponentially smaller, then ephemeral, then truly esoteric. I think the conflict at its core, is a theological conflict. A direct struggle with the theological doctrine that all true intelligence and true creativity comes from a divine spark in our meat brains, from God.
Apparently that result is also not an instance of “truly creative math research”, much like none of OpenAI’s latest ones. There appears to be some intuitive distinction between math results that are just “recombinations of known ideas” (possibly very sophisticated recombinations!), and math results that feel “truly novel”, and the mathematical consensus is apparently that none of AI math results so far are of the latter type.
It’s possible this is just a particularly memetically fit/convergent cope. Perhaps not, though; perhaps this intuition is pointing at something real. Notably, this is a take apparently shared by Noam Brown, who only expects LLMs to produce “entirely new mathematical machinery” “within two years”.
I don’t claim to understand what’s going on there, or why LLMs would be limited in this manner (whatever “this manner” even means). As bearish on LLMs as I am, I don’t see why RLVR wouldn’t suffice to let them “solve” formal mathematics. Yet, something’s up with that, I think.
My guess is that Sol wasn’t being “truly creative” in your case as well.
This I tend to agree with.
I’d mildly disagree, in that the recent counterexamples (Jacobian Conjecture and Period-Index conjecture, the former purely AI, the latter very much Human-AI collaboration) are a third category that AI seems surprisingly good at: finding examples of stuff. Not novel theory, but personally I find it hard to deny there’s a novel capability there. At least useful.
For context; my PhD research was partly aimed at verifying cases of BSD, and I care(d?) a fair amount about period-index.
Right. Non-sofics would also go there, I think? My loose understanding is that those are arguably the biggest AI-derived conceptual result to date...
Could well be true, but far from my field, so I can’t judge.
I think the conclusion is correct here but the model is not quite right. Most (or all) ideas originate from previous concepts and information, passed through a variable amount of reprocessing and abstraction. People understood the concepts of slope and area before the invention of calculus. Of course calculus took plenty of “creativity” to find, which I believe should be judged by the degree of transformation. It appears that LLMs are pretty good at combining known mathematical techniques in straightforward “right-out-of-the-packaging” ways e.g. a property of object X satisfies the assumptions of theorem Y, so X is an example of conjecture Z (exactly what happened with non-sofic groups btw). These can still be surprises, since the LLM has access to such an incredibly large array of techniques! As far as I can tell, however, they haven’t demonstrated the ability to abstract ideas further to the extent of being considered “new perspectives”.
RLVR for mathematics is not amenable to the same kind of exploration that you can do for board games like Chess and Go. In those cases you can basically harvest endless data from MCTS self-play, and thereby empirically discover winning patterns and new ideas that can be used at runtime. For math, the analogue would require the training process to somehow harvest all of the ideas that the model would need at inference to solve whatever research problem it might be posed with—essentially it would need to do the operative part of that original research ahead of time. This does not seems remotely plausible to me. Indeed RLVR for LLMs appears to be limited to improving sampling of known ideative pathways that appear in the pretraining. This last point was discussed really nicely by both Steven Byrnes and Beren Millidge recently, with some links to the relevant papers.
My view is that LLMs seem to be restricted in certain important aspects of math and other formal subjects by a lack of “conceptual fluidity”. I think this is the same blocker that you’ve described quite cogently in your expectations for LLM progress in hard-to-verify research. Actually I don’t think that the formal character of math makes that much of a difference in this regard! Sure, it makes it easier for the models to practice, but the most important driver of progress in math is still the development of qualitative concepts, and we don’t seem to have a way of generating such concepts in training (except for massive-scale empirics in the case of board games).
This is just a continuation of the “God of the gaps” argument: “If a computer can do it, it’s not intelligence.” Overtime, the difference between what a computer can do and “true intelligence” becomes exponentially smaller, then ephemeral, then truly esoteric. I think the conflict at its core, is a theological conflict. A direct struggle with the theological doctrine that all true intelligence and true creativity comes from a divine spark in our meat brains, from God.