Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
Or if you like you could start the dialogue post thing, and then we can slow-cook it (like, a “correspondence game”)?
Yeah, that sounds good. I’ll figure out my opening statement and do that within probably a couple days.
approximately no one seemed willing to defend the confident short timelines view in a public debate
Now that you mention it, I’m feeling pretty interested in trying to devil’s-advocate short timelines / red-team my most bullish-on-LLMs views. I think I grok the case for them pretty well; or, at least, there are some short-timelines cases that I think hang together well enough for me to be worried about them. Would you be up to it? Same format as with Abram.[1]
(Though I can see it ending up with me speculating about possible ways to advance capabilities in ways I wouldn’t want to make public. Hopefully at worst we’ll need to edit out a few paragraphs, though.)
culture and adoption lags capabilities, A LOT
I always feel that there’s something off about this argument. Like, if a given tool provides massive benefits, it would stand to reason that it would be widely adopted quickly, because early adopters would start massively out-competing everyone else, thus pushing everyone else to figure out how to adopt those tools as well. No?
Slow diffusion makes sense if it’s bottlenecked by physical constraints, or by coordination across many actors, or if we’re talking about e. g. government institutions which are insulated from competitive pressures. But if we’re talking about market actors adopting tools that boost their individual productivity, I don’t think this mechanism makes sense.
the leading LLM companies have said that they spend very little effort on RL-for-math
Sources? All statements I’ve seen along these lines are of the form “this impressive benchmark-breaking model is not our specialized-for-math model”, which I interpret as “we did not train it differentially harder on math than is our baseline”, not as “we train very little on math as a baseline”. (Same as e. g. Mythos not being explicitly trained to be unusually good at hacking, which obviously doesn’t mean they didn’t RLVR it on hacking tasks.)
See e.g. see a discussion by Google here, confirming that the whole training loop was in natural language, at least as of last year.
I don’t think last-year methods are a good reference here. Last-year models weren’t good at outputting Lean proofs, now everyone and their mother are successfully prompting them into that. I would guess the AGI megacorps do use proper RLVR now, not RLAIF. (Like, I don’t think there are technical barriers anymore, so why wouldn’t they?)
There is a core to your argument that I do agree with, though. Compared to other domains, the math corpus offers a dramatically greater… breadth, I guess, of legible and useful tools and building blocks? Which indeed mostly comes from pretraining: it’s not as if RLVR chisels new math concepts into LLMs, and it’s the reason RL-on-math can take off the ground at all. (If we filtered out all math beyond elementary school from the training data and then tried to bootstrap IMO-level performance via RLVR, that would not go anywhere at all.)
But no, I think math having cheap ~oracle verifiers is an additional, separate reason for why they’re good at it.
> AI is capable of revolutionary advances in mathematics. Machine learning research is not different in kind.
> Internal models are significantly ahead of released ones
I think those are wrong. E. g., here roon says that models are still bad at DL research and that he doesn’t think Anthropic has internal models significantly ahead of the frontier, which presumably also reflects the situation at OpenAI. This may not be entirely accurate and complete information, but I don’t think he’s outright lying or is even intentionally misleading about this.
Effective long-horizon continual learning is likely the last remaining step to superintelligence. It could be a harder problem than a Millennium Prize problem; I consider that unlikely.
For the purposes of effective forecasting, I don’t recommend collapsing problem difficulty to a one-dimensional axis. (Is playing Go as well as AlphaZero harder or easier than resolving a Millennium Problem? I’d say it’s definitely harder in certain senses. And yet here we still are.)
Meh, that guy changes his mind ten times per day. Also, I can’t access the article, but apparently what he actually said was this:
“We can put guardrails, we can do this and that, but I think you have a lot of negative forces that are bringing it up that shouldn’t be bringing it up and they’re bringing up things that won’t happen.”
One can of course interpret it as a principled and firm accelerationist stance, but I’d say it means basically nothing.
Especially so early. All the recent developments haven’t had time to properly propagate, the landscape is going to shift dramatically in the coming weeks and months, and Trump does respond to stimuli and incentives.
It’s been four days and AGI labs haven’t announced resolving even one more Millennium Problem. There’s been an OpenAI comment to the NYT about “making substantial progress on” another one, some rumors, and then nothing.
Huge bear signal.
(I’m almost not joking: if they don’t crack open more world-class unresolved math problems soon, that would be a slight update downward regarding AI capabilities for me. I currently think they did resolve more but are sitting on them because blindly rushing them out may be PR-negative on the margins: going 1 → 2 on the Millennium Bench won’t cause all that much more hype while possibly pissing mathematicians off and alarming the public/governments further.)
Yeah, but for this to result in actual extinction:
AI systems need to be put in charge of all this machinery first.
Doing this requires every AI system in control of a sufficiently big subset of that machinery to competently coordinate towards the strategic outcome of “kill all humans”.
(1) will take time from the current state of the world, and (2) requires strategic competence of the sort that I think is currently missing from LLMs and which may not be quite RLVR-able into them on the current paradigm. If (1) happens but (2) isn’t quite there, this just results in a catastrophic global disaster that still leaves some humans alive and provokes a genuine Butlerian Jihad on the survivors’ part.
Of course, depending on how much crucial infrastructure is in AI control and how offence-dominated the balance is, the size of the “sufficiently big” subset and the needed strategic competence would shrink over time. But that’s the point: on the current paradigm, would take quite a few years (like, 5+?) to get there from our current position. Which was at least my point with regards to “this is not the endgame”.
Define “non-trivial”. Seems like a standard “if the task consists of N independent-ish steps each of which requires doing K subtasks competently, training the model to do K-type tasks competently will result in it being able to perform the entire N-step task”, which is IMO not that interesting of a generalization. The hard long-trajectory tasks are ones that (a) require figuring out genuinely novel subtask decompositions, and (b) require you to pay attention to your overall strategic position and adjust what you’re doing if you realize you’ve started to “drift off” your task. Playing games like Pokémon doesn’t satisfy either condition: the subtask decomposition is fixed and the game state itself provides a sort of “strategic harness” that prevents you from “going backwards” or “wandering off completely”.
I’m guessing it’s just transfer from RLVR environments, particularly computer use. For all we know they literally train on playing (other) games.
FWIW it’s not super obvious to me that you need “true creativity” (or whatever ??? capability that we think is missing from LLMs) to kill humanity off either; I’d written something along these lines there.
I do think you need a significant amount of crystallized competence[1] in domains like long-term strategy and metacognition/”keeping yourself on-track”. Some of this is likely RLVR-able in simulated self-play environments and long-running evals, but I tentatively expect that the fuzzier load-bearing subskills are not, and may in fact be getting fried by RLVR. In which case progress on this is not human-unbounded on the current paradigm, and thus likely won’t quite reach takeover levels on the default LLM-progress trajectory before something else makes the question moot (e. g., non-LLM paradigm taking off).
For example, your GPT-4o-but-with-a-hidden-agenda is implementable, but the hard part would be for the AI to actually coordinate all those favors it’s asking of people towards long-term outcomes in ways that don’t get detected and shut down by humans. I think this realistically tops out on the level of a major disaster along the lines of a massive AI cult, falling short of actual extinction because it gets detected and opposed with prejudice.
Maybe not, though. But AI risk has gone mainstream now, so I’d expect the response to be pretty drastic; more drastic than 2023!me thought.
Though not necessarily supergenius amounts of it, not given quantitative AI advantages in speed, numerosity, and coordination.
… I have some trouble parsing what you mean. As in, my use of “still” in statements like “DL still has terrible sample-efficiency and generalization”? “Still” is used in the sense of “despite that” here: “despite all capability advances, DL nevertheless has terrible sample-efficiency”, rather than “they have not managed to make DL sample-efficient at this point in time[, but they will eventually]”.
I don’t know if you’d consider the transformer architecture a deep insight, or something that a current Fable/Astra level model would be able to discover independently?
Depends on the starting point. Is it RNNs, or pre-deep-learning machine learning, or...?
I do think LLMs are ultimately capable of e. g. RNN → Transformer jumps, or say Vanilla Transformer → Kimi K3.
So what’s up with how AGI labs use mathematicians’ user data?
Here’s what I currently tentatively believe:
The current AI paradigm is in principle capable of autonomously (truly autonomously, without upcoming caveats) generating some novel math results of the sort that would be difficult for humans to produce. There is no reason to believe the basic premise of RL-on-math wouldn’t work.
OpenAI does not deliberately breach user privacy in ways they don’t disclose in order to steal mathematicians’ in-progress work and then present it as their own. In particular, they did not manually access Levent and Tristan’s NS-related sessions and did not prompt their new model with them.
OpenAI does use user data for training, in ways they largely disclose. That user data likely includes mathematicians’ work-in-progress results.
(3) may lead to (1) being overestimated: some of the latest autonomous AI results may be “seeded” by pre-publication results mathematicians inadvertently fed OpenAI. In all probability, OpenAI uses automated pipelines for processing and organizing user data into training datasets, which means no-one really knows what results they may or may not be training on (see OpenAI basically saying they have no idea if Levent and Tristan’s work ended up there).
I think (3) necessarily implies (4), and as far as I can tell, (3) seems correct? Like, OpenAI does use user data for training purposes unless you opt out, and mathematicians use it for their work; therefore, incautious mathematicians’ work-in-progress results end up in the training data for OpenAI’s next models; those models are ultimately capable of doing mathematical reasoning, and may be able to “complete” the work-in-progress results fed into them.
Taken together, this means OpenAI may effectively be scooping mathematical research at an industrial scale, in a way that results in a falsely high perception of their models’ mathematical capabilities, all done without anyone necessarily deliberately engineering this result with cynical intent or breaking any laws.
Indeed, what this brings to mind is the previous “overestimating AI math capabilities” pattern: AI models have a bird’s eye view on the entirety of the human mathematical enterprise to date, including obscure theorems and whatever unknown random results are present on the internet; that means problems that would be hard/multi-step for humans “look” very shallow for AIs; that means an AI’s ability to solve a given hard-for-humans problem doesn’t necessarily imply a corresponding level of even prosaic mathematical competence.
This is just a new iteration/escalation of it, where – now that LLMs are widely used to assist in math research – those “unknown random results” now also include pre-publication results.
… Alternatively, (3) is not happening, because AI capability researchers foresaw this problem and they’re filtering the training data for novel math results in order to then not train on them, specifically in order to not over-estimate their own models’ capabilities. I can see both worlds.
Edit: Of note, there is now a claim by a mathematician previously working on non-sofic groups that OpenAI’s result seems similar to his work-in-progress approach that he’d exposed to ChatGPT, and that after OpenAI posted their results, he’d written to them asking if his data had been used for training, but they gave an in-retrospect ambiguous answer. There are of course some default reasons to be suspicious regarding the veracity of this account; that said, non-sofic in particular did strike me as surprisingly “conceptually novel” compared to usual LLM math...
Huh. Dunno how much stock to put into this; the person does appear legit at least (though the second post is partly AI-written for some reason?).
So I think you’re correct about all of this except the important part: concluding that LLM progress won’t lead fairly directly to takeover. I worry these narrowly-correct points will be used for long-timeline hopium fuel
To be clear, I don’t really have long timelines either. But I think the error bars on them are pretty wide, inasmuch as they rely on novel breakthroughs instead of on straight-lines-on-graph extrapolations, and the two models differ significantly in latent mechanisms and the correct policies for them.
When I try to picture the current AI tech stack killing everyone, I still don’t really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack’s pinnacle.
I don’t think we’re in the crunch time.
All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there’s some sense in which what it learns are the “most shallow” capabilities consistent with a given data distribution. Not literal lookup tables, but it wants to stay as close to lookup tables as possible.
What can we say about its trajectory? AI progress is driven by parameter scaling, algorithmic improvements, data improvements, and RLVR. Of those:
Parameter scaling improves capabilities generally: larger models are in fact better at ~everything, including various “illegible” skills. However, this type of scaling can’t be pushed much farther much faster, and another general Opus 3 → Fable 5 jump is not going to birth a strategic supergenius.[1]
Algorithmic improvements are not really a big contributor to things, see this and this.
Data improvements basically means hiring skilled human labor to generate annotated data traces. This also doesn’t really scale much farther much faster, and there are no runaway loops possible this way: all capabilities are improved and encoded in a basically “manual” manner.
RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains, and may even degrade scarier general-purpose capabilities.
Now, there are a bunch of counter-points one may raise here:
“Aha, but DL research is itself an easily verifiable domain!”
Sure. DL research can produce a model that is really good at doing the sort of thing that DL researchers are doing right now, improving models in the ways they are currently being improved at superhuman speeds. But this results in the Singularity if and only if the no-RSI trajectory already points to it.
That is, “does in-paradigm RSI lead to the Singularity?” reduces to “does the current AI paradigm suffice for human-unbounded improvements in all domains?”.
And I argue the answer to the latter is “no”. Unbounded progress is only possible for easily verifiable tasks, and whereas scaling data and compute theoretically provides general improvements, the pace of those is hitched to humans.
“The current capabilities are already sufficient for transformative changes across the economy, once they diffuse, including speeding up compute buildout.”
Yes. @Vladimir_Nesov-style prosaic-RSI loop is baked-in. But it’s kinda really slow.
As in: Yes, it seems plausible that the current paradigm scales to somewhat-general researchers and engineers. But the specifics don’t seem to suggest that the resultant process will be much faster than humanity: it is theoretically general but not superhuman.
“Unbounded RLVR-derived progress is sufficient for the Singularity.” We’ll need to break this one down.
Roughly speaking, the progress avenues here can be split into DL research, performance in custom RL environments, hacking, and math.
“DL research” and “performance in custom RL environments” have been mostly discussed above. This leads to the “in-paradigm”/”prosaic” RSI.
Hacking: Its pinnacle is just not that scary. It may lead to a very bad global disaster, but it’s not literally world-ending, and the median scenario is probably not even that bad.
Math: There’s an extended discussion to be had there, which I’ll separate out.
On math, the question is: does the current paradigm/RLVR in fact suffice for general-purpose superhuman math research?
So far, all AI-generated math results appear to be “conceptually stale”. Yes, even the Navier-Stokes one.[2] They’re all counterexamples or other such constructions whose assembly is extremely laborious and tricky and hard to get right (no human did this, despite $1M on the line!), but which are ultimately unenlightening, done using known approaches out of known pieces in predictable ways.[3] So far, there really appear to be no “move 37″ equivalents there.
Now, one may argue that this is going to go away with just more RLVR. Perhaps there is no real qualitative difference despite what mathematicians protest, and those “conceptually novel” discoveries are just the same “prosaic” discoveries at a big-enough scale.
Naively reasoning top-down, I would be inclined to agree. Like, suppose you fixed some well-defined mathematical problem resolving which required doing conceptually novel work, put an LLM in an RL loop in front of it, and scaled the inference-time compute up the wazoo. Why wouldn’t it work? It should work.
… except, of course, that the compute scaling matters. In theory, a random proof generator, iterated enough times, can also prove anything. But it matters whether the algorithmic pathway by which you’re attempting to reach the target is the sort that can do it in polynomial time. Arguing this differently, if there were a qualitative ground-true difference between conceptually novel vs. stale results, it would be at least possible that LLMs are just not the type of entity that can do “conceptually novel” stuff, or which are at least stubbornly bad at it.
And the negative space is drawing my attention now. I don’t actually think that “conceptually novel” contributions have to be “large-scale”; novel insights can be “small” yet still feel novel. The fact that LLMs contributed a “conceptually stale” result as big as Navier-Stokes before they yielded even a small conceptual insight sure seems like it’s hinting at something.
I don’t think this is mathematicians coping. Indeed, what this seems to be is a special case of the general issue that people trying to do conceptually novel work with LLMs, including outside mathematics, run into,[4] which is that LLMs appear completely inept at it.
This leads to a branching:
If the above is correct, the crux is that LLM improvements in doing conceptually novel work are not human-unbounded. As in, they may still be able to improve at it if AGI labs hire enough scientists to distill research taste from, but the pace is going to be limited by data progress, and it’ll also top out at top-human level. This puts us into the prosaic-RSI regime.
On this branch, the pinnacle of math RLVR is probably still pretty boring. “Leanslop”, as people say. It can provide modest boosts to theorywork, but it’s not going to do anything really crazy. As Terence Tao keeps insisting, an engine for boringly resolving some conjectures y/n is just not all that fruitful.
If the above is incorrect, implications may go beyond math. If LLMs’ ability to do conceptually novel work in math can scale unboundedly, then LLMs’ ability to do conceptually novel work period may scale unboundedly. In which case prosaic-RSI regime may quickly ratchet up to the Singularity as LLMs become capable of inventing non-DL-paradigm AI improvements.
Note that “DL is bad at generalization” doesn’t actually help here much. As in, sure, perhaps LLMs’ math-specialized and DL-research specialized research tastes aren’t going to generalize to non-math and non-DL domains. But their object-level high-taste conceptually novel math/DL research is potentially sufficient to shift us outside the DL paradigm by itself, into AI paradigms which do generalize better.
… but, weighing all of the above, I think we should be updating away from the latter scenario. Again, all RL-on-math has done so far is conceptually stale, despite how far it was pushed, and vibe checks on LLMs keep being off.
(TBC, I don’t think “conceptually novel” vs. “conceptually stale” is what really cleaves reality at the joints here. But I do think there’s some sort of meaningful qualitative difference at play regarding the sorts of cognitive tasks LLMs are capable of vs. not.)
Summing up, it seems clear that the current AI paradigm is capable of:
Unboundedly superhuman hacking, potentially sufficient to cause a global disaster.
Unboundedly superhuman in-paradigm DL research, likely sufficient to set off prosaic RSI.
Unboundedly superhuman but “workmanlike” math research.
Unboundedly superhuman performance in perhaps some other important domains, like some bio stuff.
All of those have significant, trillion-dollar implications. The current AI tech stack is a major, revolutionary technology, potentially on the scale of the industrial revolution.
But there is a vast, vast gulf between trifles like these and the Singularity,[5] and I don’t see the current AI stack as being Singularity-capable. It may be, but there’s been approximately zero new evidence towards this since 2022.[6]
I think the meta-level error people are making is anchoring too much to surface-level signals like “oh, it’s pretty good at running DL experiments!” or “oh, it can find zero-days better than the entire rest of the human cybersecurity industry!” or “oh, it resolved one of the most important questions in math!”, instead of trying to mechanistically model what’s happening in as much detail as possible, and what the specific mixes of positive and negative results we observe tell us about the structure of intelligence and LLMs’ place in it. (To be fair, those surface-level signals are very flashy and distracting. I can’t say I’m entirely unmoved.)
I was debating whether to post this, because (a) maybe it’s best to let people convince themselves the sky is falling so that drastic regulations can be urgently pushed through because there may not be another good moment for this, and (b) it may be best not to do anything that might sway AGI labs’ conviction that they’re on the right track lest they go looking for conceptually novel AI insights themselves. But I’ll err on the side of candor.
So: on balance, I do not think this is the endgame.
Note that you have to measure by the worst areas of improvement along this jump, i. e., the ones improvements along which come purely from pretraining scaling and not from any other source of improvement.
Incidentally, I’ll be assuming here that everything happened the way OpenAI says: that they did not even accidentally train on Levent and Tristan’s data and that they really have a new model that produced this result and a bunch of other novel results in a basically autonomous manner.
I still have my doubts, but this looks more plausible now.
To be clear, the ability to do that still makes LLMs a powerful emerging technology, and superhuman at some important cognitive tasks. But “yet another powerful emerging technology” and “superhuman at this well-scoped cognitive task” is a far cry from “the Singularity”. Stockfish isn’t omnicide-capable.
E. g., me, @johnswentworth and @Steven Byrnes last I checked, this poor guy, and I guess even Anthropic researchers?
As in: the reference class of the emergence of homo sapiens, rather than “mere” industrial revolution.
Indeed, if anything, I’ve updated the other way. In the GPT-3 times, I was afraid of GPT-4. In the modern times, I don’t expect that much of GPT-7. GPT-5 to GPT-6 sure wasn’t all that much.
Not much, I think they’ll say this even if the proof is just “syntactically resampled”, as I suggest.
Levent’s “the proof looks more along the lines of another euler blowup proof we had” is what I put weight onto. Namely, it makes me update in favor of the derivation being independent after all, with no data contamination at play.
Like, he basically says the proof avenue they ultimately pursued was meaningfully different after all. I’m guessing he’s implying their abandoned approach was still available to OpenAI, and that training on it may have lead to it being regurgitated and then completed. But this is starting to sound pretty shaky.
If that’s something you can share: Does that model’s performance constitute the sort of surprising result that moves your (IIRC relatively bearish) AGI timelines?
Sure, but then why are people without such qualms, which I expect constitute a double-digit percentage of the population, not outcompeting them so massively that they’re forced to squash those qualms and adopt those tools anyway?