Afaict, this is (mostly?) true, which is partially why I haverepeatedlycommunicated that these results are not as groundbreaking to AI progress as folks claim it to be.
And regarding the unit distance problem:
Yes, it seems that my idea of OOD is harsher than what many folks seem to implicitly share. It is like this because I am projecting into the future to understand what capabilities an AI would need to undergo an intelligence explosion and produce novel breakthroughs that are out-of-paradigm. I am commenting on a potential fundamental wall wrt LLMs.
It’s not just about no human involvement. For the unit distance problem, it seems largely a matter of mixing two well-known fields in non-standard ways, partly because humans have difficulty specializing in many things.
i asked claude, deepseek, chatGPT, and gemini about “top 50 most important unsolved math problems”, and they all ranked JC as being in the #10 to #20 rank. that feels pretty reasonable to justify it as legitimately “groundbreaking”
saying “well it’s not that impressive cause they just exhaustively found a finite example” feels like yet another goalpost-move, and when we get eg. the twin prime conjecture proved in generality (or disproved, because it can’t be proved/disproved with a finite example) then there will be a new cope for that.
But this and other concepts are not understood and agreed on. A coarser concept like “difficult math problem / conjecture” doesn’t support the relevant inferences. We can’t infer from “solves a difficult math problem” to “has the beginnings of ASI”, because “solves a difficult math problem” describes a huge range of things, many of which do and many of which don’t have the beginnings of ASI.
it’s true that i’m arguing a bit towards the “generalized AI skeptic” and lumping many positions together and claiming hypocracy. it’s like the easiest mistake to make on the internet and i hate when i do it. sorry about that.
Part of the issue, though is that we realized that AIs had a jagged frontier, and more generally one of the takeaways is that capabilities are more fragmented/there’s less of a necessary unifying core of intelligence than expected.
It’s also worth noting that humans are also rather jagged in their abilities, showing strengths and weaknesses, and while general intelligence, often shortened to the g-factor/IQ is real, it’s also limited in it’s predictive power.
I will somewhat grant the argument that the jagged frontier affects AI more, but I can’t fully agree with claims that suggest that AI can’t do something big/dangerous enough to be an x-risk for example because of jaggedness.
Jaggedness is countering a simple inference from “big capability on dimension X and Y” ----[enthymeme: non-jaggedness]----> “big capabilities on everything including Z = destroy the world”. It’s not that “more jagged implies less dangerous”. It matters which capabilities you have.
The ‘rank’ doesn’t really matter, you are missing the point. What matters is which cognitive moves were required for the agent to arrive at an answer to those problems and what that allows us to predict about future progress.
Please focus on the specific underlying capabilities instead of assuming I am a “goalpost-mover.” These LLMs are in fact continuing to solve the type of problems I’ve come to expect they will be good at and have so far failed at the type I expect matters even more for AI timelines.
ok, so your model is that AI won’t be able to do capability X until they’re able to do capability Y, and so far the evidence shows that they’re not close to doing capability Y. what exactly are X and Y here?
Afaict, this is (mostly?) true, which is partially why I have repeatedly communicated that these results are not as groundbreaking to AI progress as folks claim it to be.
And regarding the unit distance problem:
i asked claude, deepseek, chatGPT, and gemini about “top 50 most important unsolved math problems”, and they all ranked JC as being in the #10 to #20 rank. that feels pretty reasonable to justify it as legitimately “groundbreaking”
saying “well it’s not that impressive cause they just exhaustively found a finite example” feels like yet another goalpost-move, and when we get eg. the twin prime conjecture proved in generality (or disproved, because it can’t be proved/disproved with a finite example) then there will be a new cope for that.
The issue is that the goalposts had not been communicated on / understood / agreed on, not that they’ve been moved. As an example of a concept one might want to have, in order to agree on goalposts, is “algebraicness”: https://tsvibt.github.io/theory/pages/bl_24_07_25_09_52_56_652909.html
But this and other concepts are not understood and agreed on. A coarser concept like “difficult math problem / conjecture” doesn’t support the relevant inferences. We can’t infer from “solves a difficult math problem” to “has the beginnings of ASI”, because “solves a difficult math problem” describes a huge range of things, many of which do and many of which don’t have the beginnings of ASI.
it’s true that i’m arguing a bit towards the “generalized AI skeptic” and lumping many positions together and claiming hypocracy. it’s like the easiest mistake to make on the internet and i hate when i do it. sorry about that.
Part of the issue, though is that we realized that AIs had a jagged frontier, and more generally one of the takeaways is that capabilities are more fragmented/there’s less of a necessary unifying core of intelligence than expected.
It’s also worth noting that humans are also rather jagged in their abilities, showing strengths and weaknesses, and while general intelligence, often shortened to the g-factor/IQ is real, it’s also limited in it’s predictive power.
I will somewhat grant the argument that the jagged frontier affects AI more, but I can’t fully agree with claims that suggest that AI can’t do something big/dangerous enough to be an x-risk for example because of jaggedness.
Jaggedness is countering a simple inference from “big capability on dimension X and Y” ----[enthymeme: non-jaggedness]----> “big capabilities on everything including Z = destroy the world”. It’s not that “more jagged implies less dangerous”. It matters which capabilities you have.
The ‘rank’ doesn’t really matter, you are missing the point. What matters is which cognitive moves were required for the agent to arrive at an answer to those problems and what that allows us to predict about future progress.
Please focus on the specific underlying capabilities instead of assuming I am a “goalpost-mover.” These LLMs are in fact continuing to solve the type of problems I’ve come to expect they will be good at and have so far failed at the type I expect matters even more for AI timelines.
ok, so your model is that AI won’t be able to do capability X until they’re able to do capability Y, and so far the evidence shows that they’re not close to doing capability Y. what exactly are X and Y here?
It’s a bit difficult to explain quickly, but some of my thoughts on the matter are here.
And here: https://www.lesswrong.com/posts/jXjeYYPXipAtA2zmj/jacquesthibs-s-shortform?commentId=pCiAsJ7NLXrtBymw2