Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
It’s been four days and AGI labs haven’t announced resolving even one more Millennium Problem. There’s been an OpenAI comment to the NYT about “making substantial progress on” another one, some rumors, and then nothing.
Huge bear signal.
(I’m almost not joking: if they don’t crack open more world-class unresolved math problems soon, that would be a slight update downward regarding AI capabilities for me. I currently think they did resolve more but are sitting on them because blindly rushing them out may be PR-negative on the margins: going 1 → 2 on the Millennium Bench won’t cause all that much more hype while possibly pissing mathematicians off and alarming the public/governments further.)
Yeah, but for this to result in actual extinction:
AI systems need to be put in charge of all this machinery first.
Doing this requires every AI system in control of a sufficiently big subset of that machinery to competently coordinate towards the strategic outcome of “kill all humans”.
(1) will take time from the current state of the world, and (2) requires strategic competence of the sort that I think is currently missing from LLMs and which may not be quite RLVR-able into them on the current paradigm. If (1) happens but (2) isn’t quite there, this just results in a catastrophic global disaster that still leaves some humans alive and provokes a genuine Butlerian Jihad on the survivors’ part.
Of course, depending on how much crucial infrastructure is in AI control and how offence-dominated the balance is, the size of the “sufficiently big” subset and the needed strategic competence would shrink over time. But that’s the point: on the current paradigm, would take quite a few years (like, 5+?) to get there from our current position. Which was at least my point with regards to “this is not the endgame”.
Define “non-trivial”. Seems like a standard “if the task consists of N independent-ish steps each of which requires doing K subtasks competently, training the model to do K-type tasks competently will result in it being able to perform the entire N-step task”, which is IMO not that interesting of a generalization. The hard long-trajectory tasks are ones that (a) require figuring out genuinely novel subtask decompositions, and (b) require you to pay attention to your overall strategic position and adjust what you’re doing if you realize you’ve started to “drift off” your task. Playing games like Pokémon doesn’t satisfy either condition: the subtask decomposition is fixed and the game state itself provides a sort of “strategic harness” that prevents you from “going backwards” or “wandering off completely”.
I’m guessing it’s just transfer from RLVR environments, particularly computer use. For all we know they literally train on playing (other) games.
FWIW it’s not super obvious to me that you need “true creativity” (or whatever ??? capability that we think is missing from LLMs) to kill humanity off either; I’d written something along these lines there.
I do think you need a significant amount of crystallized competence[1] in domains like long-term strategy and metacognition/”keeping yourself on-track”. Some of this is likely RLVR-able in simulated self-play environments and long-running evals, but I tentatively expect that the fuzzier load-bearing subskills are not, and may in fact be getting fried by RLVR. In which case progress on this is not human-unbounded on the current paradigm, and thus likely won’t quite reach takeover levels on the default LLM-progress trajectory before something else makes the question moot (e. g., non-LLM paradigm taking off).
For example, your GPT-4o-but-with-a-hidden-agenda is implementable, but the hard part would be for the AI to actually coordinate all those favors it’s asking of people towards long-term outcomes in ways that don’t get detected and shut down by humans. I think this realistically tops out on the level of a major disaster along the lines of a massive AI cult, falling short of actual extinction because it gets detected and opposed with prejudice.
Maybe not, though. But AI risk has gone mainstream now, so I’d expect the response to be pretty drastic; more drastic than 2023!me thought.
Though not necessarily supergenius amounts of it, not given quantitative AI advantages in speed, numerosity, and coordination.
… I have some trouble parsing what you mean. As in, my use of “still” in statements like “DL still has terrible sample-efficiency and generalization”? “Still” is used in the sense of “despite that” here: “despite all capability advances, DL nevertheless has terrible sample-efficiency”, rather than “they have not managed to make DL sample-efficient at this point in time[, but they will eventually]”.
I don’t know if you’d consider the transformer architecture a deep insight, or something that a current Fable/Astra level model would be able to discover independently?
Depends on the starting point. Is it RNNs, or pre-deep-learning machine learning, or...?
I do think LLMs are ultimately capable of e. g. RNN → Transformer jumps, or say Vanilla Transformer → Kimi K3.
So what’s up with how AGI labs use mathematicians’ user data?
Here’s what I currently tentatively believe:
The current AI paradigm is in principle capable of autonomously (truly autonomously, without upcoming caveats) generating some novel math results of the sort that would be difficult for humans to produce. There is no reason to believe the basic premise of RL-on-math wouldn’t work.
OpenAI does not deliberately breach user privacy in ways they don’t disclose in order to steal mathematicians’ in-progress work and then present it as their own. In particular, they did not manually access Levent and Tristan’s NS-related sessions and did not prompt their new model with them.
OpenAI does use user data for training, in ways they largely disclose. That user data likely includes mathematicians’ work-in-progress results.
(3) may lead to (1) being overestimated: some of the latest autonomous AI results may be “seeded” by pre-publication results mathematicians inadvertently fed OpenAI. In all probability, OpenAI uses automated pipelines for processing and organizing user data into training datasets, which means no-one really knows what results they may or may not be training on (see OpenAI basically saying they have no idea if Levent and Tristan’s work ended up there).
I think (3) necessarily implies (4), and as far as I can tell, (3) seems correct? Like, OpenAI does use user data for training purposes unless you opt out, and mathematicians use it for their work; therefore, incautious mathematicians’ work-in-progress results end up in the training data for OpenAI’s next models; those models are ultimately capable of doing mathematical reasoning, and may be able to “complete” the work-in-progress results fed into them.
Taken together, this means OpenAI may effectively be scooping mathematical research at an industrial scale, in a way that results in a falsely high perception of their models’ mathematical capabilities, all done without anyone necessarily deliberately engineering this result with cynical intent or breaking any laws.
Indeed, what this brings to mind is the previous “overestimating AI math capabilities” pattern: AI models have a bird’s eye view on the entirety of the human mathematical enterprise to date, including obscure theorems and whatever unknown random results are present on the internet; that means problems that would be hard/multi-step for humans “look” very shallow for AIs; that means an AI’s ability to solve a given hard-for-humans problem doesn’t necessarily imply a corresponding level of even prosaic mathematical competence.
This is just a new iteration/escalation of it, where – now that LLMs are widely used to assist in math research – those “unknown random results” now also include pre-publication results.
… Alternatively, (3) is not happening, because AI capability researchers foresaw this problem and they’re filtering the training data for novel math results in order to then not train on them, specifically in order to not over-estimate their own models’ capabilities. I can see both worlds.
Edit: Of note, there is now a claim by a mathematician previously working on non-sofic groups that OpenAI’s result seems similar to his work-in-progress approach that he’d exposed to ChatGPT, and that after OpenAI posted their results, he’d written to them asking if his data had been used for training, but they gave an in-retrospect ambiguous answer. There are of course some default reasons to be suspicious regarding the veracity of this account; that said, non-sofic in particular did strike me as surprisingly “conceptually novel” compared to usual LLM math...
Huh. Dunno how much stock to put into this; the person does appear legit at least (though the second post is partly AI-written for some reason?).
So I think you’re correct about all of this except the important part: concluding that LLM progress won’t lead fairly directly to takeover. I worry these narrowly-correct points will be used for long-timeline hopium fuel
To be clear, I don’t really have long timelines either. But I think the error bars on them are pretty wide, inasmuch as they rely on novel breakthroughs instead of on straight-lines-on-graph extrapolations, and the two models differ significantly in latent mechanisms and the correct policies for them.
When I try to picture the current AI tech stack killing everyone, I still don’t really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack’s pinnacle.
I don’t think we’re in the crunch time.
All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there’s some sense in which what it learns are the “most shallow” capabilities consistent with a given data distribution. Not literal lookup tables, but it wants to stay as close to lookup tables as possible.
What can we say about its trajectory? AI progress is driven by parameter scaling, algorithmic improvements, data improvements, and RLVR. Of those:
Parameter scaling improves capabilities generally: larger models are in fact better at ~everything, including various “illegible” skills. However, this type of scaling can’t be pushed much farther much faster, and another general Opus 3 → Fable 5 jump is not going to birth a strategic supergenius.[1]
Algorithmic improvements are not really a big contributor to things, see this and this.
Data improvements basically means hiring skilled human labor to generate annotated data traces. This also doesn’t really scale much farther much faster, and there are no runaway loops possible this way: all capabilities are improved and encoded in a basically “manual” manner.
RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains, and may even degrade scarier general-purpose capabilities.
Now, there are a bunch of counter-points one may raise here:
“Aha, but DL research is itself an easily verifiable domain!”
Sure. DL research can produce a model that is really good at doing the sort of thing that DL researchers are doing right now, improving models in the ways they are currently being improved at superhuman speeds. But this results in the Singularity if and only if the no-RSI trajectory already points to it.
That is, “does in-paradigm RSI lead to the Singularity?” reduces to “does the current AI paradigm suffice for human-unbounded improvements in all domains?”.
And I argue the answer to the latter is “no”. Unbounded progress is only possible for easily verifiable tasks, and whereas scaling data and compute theoretically provides general improvements, the pace of those is hitched to humans.
“The current capabilities are already sufficient for transformative changes across the economy, once they diffuse, including speeding up compute buildout.”
Yes. @Vladimir_Nesov-style prosaic-RSI loop is baked-in. But it’s kinda really slow.
As in: Yes, it seems plausible that the current paradigm scales to somewhat-general researchers and engineers. But the specifics don’t seem to suggest that the resultant process will be much faster than humanity: it is theoretically general but not superhuman.
“Unbounded RLVR-derived progress is sufficient for the Singularity.” We’ll need to break this one down.
Roughly speaking, the progress avenues here can be split into DL research, performance in custom RL environments, hacking, and math.
“DL research” and “performance in custom RL environments” have been mostly discussed above. This leads to the “in-paradigm”/”prosaic” RSI.
Hacking: Its pinnacle is just not that scary. It may lead to a very bad global disaster, but it’s not literally world-ending, and the median scenario is probably not even that bad.
Math: There’s an extended discussion to be had there, which I’ll separate out.
On math, the question is: does the current paradigm/RLVR in fact suffice for general-purpose superhuman math research?
So far, all AI-generated math results appear to be “conceptually stale”. Yes, even the Navier-Stokes one.[2] They’re all counterexamples or other such constructions whose assembly is extremely laborious and tricky and hard to get right (no human did this, despite $1M on the line!), but which are ultimately unenlightening, done using known approaches out of known pieces in predictable ways.[3] So far, there really appear to be no “move 37″ equivalents there.
Now, one may argue that this is going to go away with just more RLVR. Perhaps there is no real qualitative difference despite what mathematicians protest, and those “conceptually novel” discoveries are just the same “prosaic” discoveries at a big-enough scale.
Naively reasoning top-down, I would be inclined to agree. Like, suppose you fixed some well-defined mathematical problem resolving which required doing conceptually novel work, put an LLM in an RL loop in front of it, and scaled the inference-time compute up the wazoo. Why wouldn’t it work? It should work.
… except, of course, that the compute scaling matters. In theory, a random proof generator, iterated enough times, can also prove anything. But it matters whether the algorithmic pathway by which you’re attempting to reach the target is the sort that can do it in polynomial time. Arguing this differently, if there were a qualitative ground-true difference between conceptually novel vs. stale results, it would be at least possible that LLMs are just not the type of entity that can do “conceptually novel” stuff, or which are at least stubbornly bad at it.
And the negative space is drawing my attention now. I don’t actually think that “conceptually novel” contributions have to be “large-scale”; novel insights can be “small” yet still feel novel. The fact that LLMs contributed a “conceptually stale” result as big as Navier-Stokes before they yielded even a small conceptual insight sure seems like it’s hinting at something.
I don’t think this is mathematicians coping. Indeed, what this seems to be is a special case of the general issue that people trying to do conceptually novel work with LLMs, including outside mathematics, run into,[4] which is that LLMs appear completely inept at it.
This leads to a branching:
If the above is correct, the crux is that LLM improvements in doing conceptually novel work are not human-unbounded. As in, they may still be able to improve at it if AGI labs hire enough scientists to distill research taste from, but the pace is going to be limited by data progress, and it’ll also top out at top-human level. This puts us into the prosaic-RSI regime.
On this branch, the pinnacle of math RLVR is probably still pretty boring. “Leanslop”, as people say. It can provide modest boosts to theorywork, but it’s not going to do anything really crazy. As Terence Tao keeps insisting, an engine for boringly resolving some conjectures y/n is just not all that fruitful.
If the above is incorrect, implications may go beyond math. If LLMs’ ability to do conceptually novel work in math can scale unboundedly, then LLMs’ ability to do conceptually novel work period may scale unboundedly. In which case prosaic-RSI regime may quickly ratchet up to the Singularity as LLMs become capable of inventing non-DL-paradigm AI improvements.
Note that “DL is bad at generalization” doesn’t actually help here much. As in, sure, perhaps LLMs’ math-specialized and DL-research specialized research tastes aren’t going to generalize to non-math and non-DL domains. But their object-level high-taste conceptually novel math/DL research is potentially sufficient to shift us outside the DL paradigm by itself, into AI paradigms which do generalize better.
… but, weighing all of the above, I think we should be updating away from the latter scenario. Again, all RL-on-math has done so far is conceptually stale, despite how far it was pushed, and vibe checks on LLMs keep being off.
(TBC, I don’t think “conceptually novel” vs. “conceptually stale” is what really cleaves reality at the joints here. But I do think there’s some sort of meaningful qualitative difference at play regarding the sorts of cognitive tasks LLMs are capable of vs. not.)
Summing up, it seems clear that the current AI paradigm is capable of:
Unboundedly superhuman hacking, potentially sufficient to cause a global disaster.
Unboundedly superhuman in-paradigm DL research, likely sufficient to set off prosaic RSI.
Unboundedly superhuman but “workmanlike” math research.
Unboundedly superhuman performance in perhaps some other important domains, like some bio stuff.
All of those have significant, trillion-dollar implications. The current AI tech stack is a major, revolutionary technology, potentially on the scale of the industrial revolution.
But there is a vast, vast gulf between trifles like these and the Singularity,[5] and I don’t see the current AI stack as being Singularity-capable. It may be, but there’s been approximately zero new evidence towards this since 2022.[6]
I think the meta-level error people are making is anchoring too much to surface-level signals like “oh, it’s pretty good at running DL experiments!” or “oh, it can find zero-days better than the entire rest of the human cybersecurity industry!” or “oh, it resolved one of the most important questions in math!”, instead of trying to mechanistically model what’s happening in as much detail as possible, and what the specific mixes of positive and negative results we observe tell us about the structure of intelligence and LLMs’ place in it. (To be fair, those surface-level signals are very flashy and distracting. I can’t say I’m entirely unmoved.)
I was debating whether to post this, because (a) maybe it’s best to let people convince themselves the sky is falling so that drastic regulations can be urgently pushed through because there may not be another good moment for this, and (b) it may be best not to do anything that might sway AGI labs’ conviction that they’re on the right track lest they go looking for conceptually novel AI insights themselves. But I’ll err on the side of candor.
So: on balance, I do not think this is the endgame.
Note that you have to measure by the worst areas of improvement along this jump, i. e., the ones improvements along which come purely from pretraining scaling and not from any other source of improvement.
Incidentally, I’ll be assuming here that everything happened the way OpenAI says: that they did not even accidentally train on Levent and Tristan’s data and that they really have a new model that produced this result and a bunch of other novel results in a basically autonomous manner.
I still have my doubts, but this looks more plausible now.
To be clear, the ability to do that still makes LLMs a powerful emerging technology, and superhuman at some important cognitive tasks. But “yet another powerful emerging technology” and “superhuman at this well-scoped cognitive task” is a far cry from “the Singularity”. Stockfish isn’t omnicide-capable.
E. g., me, @johnswentworth and @Steven Byrnes last I checked, this poor guy, and I guess even Anthropic researchers?
As in: the reference class of the emergence of homo sapiens, rather than “mere” industrial revolution.
Indeed, if anything, I’ve updated the other way. In the GPT-3 times, I was afraid of GPT-4. In the modern times, I don’t expect that much of GPT-7. GPT-5 to GPT-6 sure wasn’t all that much.
Not much, I think they’ll say this even if the proof is just “syntactically resampled”, as I suggest.
Levent’s “the proof looks more along the lines of another euler blowup proof we had” is what I put weight onto. Namely, it makes me update in favor of the derivation being independent after all, with no data contamination at play.
Like, he basically says the proof avenue they ultimately pursued was meaningfully different after all. I’m guessing he’s implying their abandoned approach was still available to OpenAI, and that training on it may have lead to it being regurgitated and then completed. But this is starting to sound pretty shaky.
If that’s something you can share: Does that model’s performance constitute the sort of surprising result that moves your (IIRC relatively bearish) AGI timelines?
Well. Yeah. I guess. This is roughly what I was gesturing at by “another bloody counterexample” in that last edit, I suppose.
Eh, who knows what the starting point of that training run was. Maybe it already was some ahead-of-the-public-frontier model.
We (the researchers and the agents) did not see any of [Levent Alpöge and Tristan Buckmaster’s] work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem. While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.
I think I’m in love with this line.
More seriously, I do wonder very much how similar the proof structures are to Levent & Tristan’s work, and if they are different, how much we can glean from that. How much of the difference may be due to “proof resampling”? Like, suppose OpenAI did train the model on their data, and then prompted it to re-generate the proof (or resolve a different but very related question). Would it produce the same construction? If no, how different would the construction be? I would intuitively expect them to be different on the surface (perhaps even proving somewhat different things, as OpenAI alludes), but similar in some more deep conceptual sense.
I’m not going to review several hundred pages of math PDFs because of it, though, and in any case my understanding is that the most relevant Levent & Tristan result (“blowup for hypo-dissipative Navier-Stokes”) isn’t out, so no side-by-side comparison is yet possible. Still, when it’s out, this sort of comparison seems like one of the most reliable sources of evidence on what actually happened.
… though, in the medium-term, I suppose the question will be resolved by whether OpenAI’s claims about their new internal model will turn out to be true. E. g., this one by Noam Brown:
It can be hard to “feel the AGI” until you see an AI surpass you in a domain you care deeply about. This week, many mathematicians and physicists at @OpenAI had their Lee Sedol moment seeing this model solve, in minutes, open problems they’d struggled with for years.
If that is the case, it’s not going to stop at Navier-Stokes.
The alternate hypothesis, which I’d usually consider highly implausible copium, but which this debacle has suddenly injected a lot of probability mass into, is that OpenAI really is engaging in massive amounts of fabrication in order to keep themselves afloat, presenting math results that were produced by humans or with heavy human involvement as autonomous AI discoveries. Tristan at least implies that.[1]
… I find that I don’t actually believe this, even now, but we’ll see.
Edit: Hm. This is a really weird confluence of circumstances all around.
For one thing, it seems a-priori surprising that an academic mathematician and a mathematician working at an AGI lab were just about to release papers demonstrating significant progress towards Navier-Stokes at the same time as OpenAI happened to be developing a surprisingly, off-trend-powerful internal model that was so powerful as to (near-?)autonomously resolve the same problem in the timespan between OpenAI hearing of the rumor and the papers’ publication.
I suppose weirder things have happened, but it does seem overall more plausible if we do assume some data leakage from Levent & Tristan...?
(Ugh, I think I wasted too much time on this today already. The correct policy is to just wait for more data to come in, but I find it too fun to speculate.)
Edit #2: Aha, Levent offers some new commentary:
“we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”
i mean props to them for straight coming clean.
(so far the proof looks more along the lines of another euler blowup proof we had, off of whose ansatz naming we were making really stupid puns like “smooth criminale”, unlike the much better “ideal fluids explode”, Tristan)
I guess we can conclude that at least one of the parties is blatantly lying.
Also of note is this tweet:
Dheeraj Nagaraj: I was an intern under Seb in 2020. Unfortunately, the allegations of unscrupulous behavior is 100% believable. I am glad that the mask is off publicly. I really hope he doesn’t weasel out of this.
The account was just created, but the name checks out and a bunch of other accounts that appear to be real (one, two, three, four) are vouching for it, so for now I’m assuming it’s real. Which, plus priors, makes me think it’s OpenAI that’s lying.
Promising. (Thrilling stuff. Haven’t had this much popcorn in a while.)
Edit #3: Wait, I just realized, first Millennium problem solved by AI or with heavy AI assistance, and it’s another bloody counterexample?
“Levent had been told by Sebastien “very little human input” had been used. This turned out not to be true. Over the course of the call, as members of their team sent Sebastien corrections and details over their internal chat, it emerged that an entire team had been working on the problem, that this was one of a number of things that was tried, that work had started on the unforced problem, that the team first set the model on easier problems, including Euler, that even the prompt that had been shown to me had been written by prompting Codex, and that an insane amount of compute had been used.”
Sama doesn’t have the skills or the balls to take over the world right under the government’s nose
Why do you say so? I have faith in him.
Maybe. It does feel like the sort of paranoia/conspiracy theory that usually ends up unfounded, but I’m not sure I know whether paranoia that ends up well-founded feels any different at the onset.
So, with regards to the Navier-Stokes story:
Part of it is an implication that OpenAI accessed user data in order to then internally prompt and/or train their models using it. Some people are now claiming that this is totally impossible, that there are strict controls regarding who can access user data, and that it would be suicide for OpenAI to be doing anything like this.
But: A while back, there was this school-shooting incident whose perpetrator consulted ChatGPT beforehand. According to OpenAI’s own statements, their safety classifiers flagged the relevant transcript for review, and OpenAI employees then debated whether to notify law enforcement. This is an existence proof that “automated classifiers flag a transcript → human eyes look at user data” is a thing that happens.
One may assume that this is only legally acceptable for safety reasons. However, I would guess that there are obviously no regulations of any sort regarding on what basis LLM transcripts could be flagged for review. Legally speaking, “transcripts flagged by OpenAI’s safety classifiers” is just an arbitrary category of transcripts that OpenAI permitted itself to manually review.
As the result, the dark possibility that always seemed worth keeping in mind to me is that AGI labs have various other triggers for flagging transcripts as requiring human review, which are looking for other types of information that would be of interest to them. AI-related capability insights would be my primary guess and concern, but other salient possibilities are also blackmail material, software exploits, and, well, math research results. They wouldn’t be using that willy-nilly, because legal and reputational risks are still enormous while benefits are miniscule in nearly all cases, but if the stakes are high enough/the prize is really juicy (as in this case)...
This seems entirely in line with what other major corporations have done in the past. If that feels impossibly beyond the pale, it’s because, for various historical and social reasons, AGI labs are viewed much more charitably (at least in our subculture) than the median megacorp. And it increasingly seems that this charity is ill-founded.
… anyway, point is, think hard before prompting proprietary LLMs with your takes regarding capability research, possible limitations of the current AI paradigm and how to address them, non-LLM routes to AGI, or any other such stuff. (This is partly a note to myself to be even more paranoid on this than I already was.)
FWIW, there is also a relatively innocuous explanation for the Navier-Stokes thing, suggested to me by Fable 5.1:
Tristan mentions that what makes OpenAI’s simultaneous result highly suspicious is that it seems to be taking the same highly specific approach that he and Levent pursued. However, the whole story was set off by a rumor about their upcoming results reaching OpenAI. It’s possible the rumor was specific enough to hint at the approach they were taking, and that just this was sufficient for OpenAI’s models to replicate the work. In this case, no user data theft was necessarily involved.
Maybe.
Meh, that guy changes his mind ten times per day. Also, I can’t access the article, but apparently what he actually said was this:
One can of course interpret it as a principled and firm accelerationist stance, but I’d say it means basically nothing.
Especially so early. All the recent developments haven’t had time to properly propagate, the landscape is going to shift dramatically in the coming weeks and months, and Trump does respond to stimuli and incentives.