I’m an AGI safety / AI alignment researcher in Boston with a particular focus on brain algorithms. Research Fellow at Astera. I’m also at: Substack, X/Twitter, Bluesky, RSS, email, and more at this link. See https://sjbyrnes.com/agi.html for a summary of my research and sorted list of writing. Physicist by training. Leave me anonymous feedback here.
Steven Byrnes
Karma: 31,649
“Out of sample” is hard to define. There’s a trivial sense in which almost everything is “out of sample”, because any given specific prompt has probably not appeared on the internet. But I think there are better ways to define “out of sample”, even if it’s hard to make rigorous. Like if a kid has drawn pink trees and purple trees but has never drawn a gray tree (but they have drawn gray cars and gray houses), then the kid can almost definitely draw a gray tree, without “learning something new”. I propose that we should define the term “out of sample” such that this kind of thing doesn’t count. I don’t think that LLMs have come up with a way to exploit a security flaw that isn’t very similar to ways that humans in the pretraining data have previously exploited security flaws. So I don’t think LLM exploits are “out of sample” in that stronger sense; rather, the LLMs are applying existing human ideas in a new context, with impressive nuance and competence. It’s a kind of style transfer, in a very abstract sense, as Jeremy Howard argued. That doesn’t detract from how impressive it is, nor from its real-world implications, but I think it’s relevant for our understanding.
As for exploit chaining, thanks to the combinatorial explosion, it’s possible that an LLM would be the first ever to chain a specific exploit X with a specific exploit Y, etc., but again I don’t think that should count as “out of sample”, any more than it’s “out of sample” for a kid who already knows how to draw trains and boats to draw a train on a boat for the first time, even if they have to make some custom modifications at the interface (like how the track sits on the boat).
So I’m not convinced that recent advances in LLM exploit chaining is “out of sample” (of the internet) in the relevant sense, although I confess I’m not familiar with what exploits look like in detail.
I DO think the “thinking out loud in the CoT” aspect of LLM agents is legitimately out of sample compared to the pretraining data. There was nothing like that on the internet before LLMs came along. So post-training is clearly essential for that, and thus essential for turning LLMs into LLM agents.
those errors get averaged away in next-token prediction unless they’re systematic.
Sorry if I’m misunderstanding, but aren’t they always systematic in cases we care about? If you look at (say) people writing papers about the neuroscience of schizophrenia, there are a zillion mutually-incompatible ideas and schools of thought. It’s not the case that everyone is saying “truth plus random noise”, so all you need is enough data and you can average it out to get the true neuroscience of schizophrenia, right?
Compare a question like “Is this natural language proof valid?” to “Is this story well written?” Which is easier to train on? If you can create a scoring function for stories that works as reliably as judging the validity of a natural language proof, then I’d predict you could train LLMs to write really good stories.
Yes this is exactly my point. It is much easier to create an LLM that says whether a natural-language proof is correct, than an LLM that says whether a fiction story is good. And I’m asking the question: Why the difference? You want to say “because math is verifiable”, and I’m drilling into that and saying it doesn’t make sense. What does “math is verifiable” mean to you, and what exactly is the path from “math is verifiable” to “the LLM can say whether a natural-language proof is correct”?
The answer to that question can’t involve RLAIF, because you can’t even start RLAIF until you already have an LLM that says whether a natural-language proof is good. It’s a chicken-and-egg thing.
You CAN say “the LLM can say whether a natural-language proof is correct, including e.g. noticing subtle flaws in a proof’s use of p-adic Hodge theory, because we did RLVR on its ability to do high school contest math problems with right and wrong answers, along with RLVR on algebraic manipulation tasks that we could check with SymPy, etc.” At least, that hypothesis would not have a chicken-and-egg problem. But it strikes me as a very far-fetched answer to the question, that cannot possibly be hitting the nail on the head. Those two kinds of things are very very different! Why would there be such strong transfer from one to the other? I’m open to nonzero transfer, I just expect it to be very indirect and high-level. Whatever the LLM knows about p-adic Hodge theory that enable it to catch mistakes in proofs, it didn’t get that from RLVR on math problems with right and wrong answers. See what I mean?
I strongly suspect that when people talk about the gory details of cyberoffense online and in textbooks, the things they say are overwhelmingly true. (E.g. if someone writes “this code is a working exploit of blah”, then it almost always is.) In this sense, I think cyberoffense is LIKE when people talk about the gory details of research math stuff like algebraic topology (overwhelmingly true), and UNLIKE when people talk about the gory details of autism, or of middle school classroom discipline, or of how to write a good work of fiction (a big stew of truths and falsehoods and confusions that all superficially look the same).
So unless I’m wrong about the quality of discourse in the cybersecurity community, my theory in this OP would predict that cyberoffense is starting from a very good place, with the deck stacked in its favor for it to be one of the areas where LLMs are especially strong.
You’re bringing up “human action sequences”, and I heartily agree that post-training (SFT and/or RL) is important for transforming (something like) “base LLM has detailed and overwhelmingly correct understanding of such-and-such domain” into (something like) “post-trained LLM has the ability to be an effective agent that autonomously gets things done in that domain”. This is one of the things RL-on-LLMs has always been best at, see LLMs are (still) mostly powered by imitative learning, not RL” section 1.5.
The point of this post was to compare LLMs-doing-research-math to LLMs-writing-fiction or LLMs-doing-research-educational-psychology, not to compare LLM base models to post-trained LLMs. Obviously post-trained LLMs can do things that non-post-trained LLMs can’t; it would be completely absurd to suggest otherwise. I’m unhappy that you got the wrong impression and reworded the first sentence to make that clearer. :)
I’m talking about proof-based advanced math in natural language. This activity has never been “verifiable” until Lean-autoformalization became possible. And I claim that the models were already remarkably good at proof-based advanced math in natural language before Lean-autoformalization became possible.
Indeed, it seems pretty implausible to me to say that the order of events during training / recent history is that first the models become capable of doing Lean-autoformalization of natural-language proofs, and only then the models become good at natural-language proofs, thanks to RLVR-via-autoformalization. Is that what you think? To me, that really seems like the wrong order. Lean-autoformalization of a natural-language proof is hard, and seems to require access to a model that is already very good at natural-language proofs.
I appreciate the info on LLM training and will edit that footnote.
My claim is that, if we want to understand how LLMs can construct correct natural-language proofs that involve abstruse mathematical objects like the de Rham cohomology group, a big part of the answer is that that approximately every time anyone has ever written down anything about the de Rham cohomology group, that thing they wrote has been true. So the LLM emerges from pretraining with an tendency to say true things, and only true things, about the de Rham cohomology group, and the same applies to everything else in advanced math. This is very very different from most other topics, like anxiety or middle school classroom discipline. Of course, a tendency to say true things and only true things about the de Rham cohomology group isn’t enough, and I’m certainly not questioning the importance of post-training for surfacing that tendency at the appropriate times, and honing it, etc.
Basically, I think you’re being insufficiently curious about the part where you said “this transfers quite well to proof-based arguments also”. It seems to me that AIME problems and computer algebra etc. are really quite different from properly using the de Rham cohomology group in a natural-language proof. I know far less than you about LLM training, but I’d be a bit surprised if LLM training today involves even one RLVR task that requires proper use of the de Rham cohomology group, much less many of them, or if they do I’d be surprised if that were critical as opposed to merely helpful. So I don’t think we would see “transfer to proof-based arguments” if there wasn’t already a whole preexisting edifice of correct proof-based arguments involving the de Rham cohomology group along with everything else, latent in the base model, and post-training merely needs to dredge up that edifice, rather than build it piece-by-piece. There’s just not enough bits to build it piece-by-piece, plus I don’t even know how that would work. (I.e., it seems like there would be a chicken-and-egg problem, in that you can’t judge or autoformalize a natural-language proof involving de Rham cohomology without a model that already somehow knows what things about de Rham cohomology are true vs false.)
[This might be one of those pointless arguments where I say “X is important!” and you say “No, Y is important!”, but in reality X and Y are both important. Shrug.]
verifiability enables correct imitation data via synthetic data pipelines
It’s not obvious how to build a synthetic data pipeline for natural-language proofs. I guess you’d have an LLM generate candidate proofs, and then judge them by Lean autoformalization, right? If so, that’s basically the same as RLVR-for-proofs-via-autoformalization as mentioned in the (recently updated) post. And my argument is that it was already clear that LLMs were unusually good at natural-language proofs before anyone could have built such a system.
I’m curious as to the source for this.
Yeah fair, I deleted that part a couple minutes ago, see changelog.
this is an inference-time scaffold and not RL-for-math
Whoops that’s embarrassing, I’ll fix it now.
Sources?
I just deleted that part, thanks for the pushback.
I don’t think last-year methods are a good reference here
I just edited to be clearer that (for all I know) companies might be doing RLVR with Lean autoformalization in the loop. (Along with another mistake.)
I think it was already obvious that math was one of the areas that LLMs were best at (compared to other things like law, fiction, psychology, etc.) a year or two ago, which I think was well before autoformalization would have been realistic.
So there has to be an explanation for “LLMs are unusually good at math (compared to how good they are at other things)” that doesn’t involve autoformalization.
That said, I suppose it’s possible that people are doing that now, I wouldn’t know, and I hope I didn’t imply otherwise. UPDATE: I edited the footnote, hope it’s better now, thanks.
The models have been getting better at imitative learning (model size, architecture, learning algorithm details), and the data they’re imitating has also been getting better (filtering, data augmentation, etc.), and then on top of that, the mid-training SFT tokens and RLAIF are clearly huge parts of the story. (See The nature of LLM algorithmic progress.)
My claim is, like, if the companies had put just as much effort (data curation, RLAIF, etc.) into making their LLMs good at educational psychology, as they did into making their LLMs good at math, they would not have remotely as much to show for that effort. The RLAIF would be garbage-in-garbage-out because the LLM judge would be incompetent; and the data curation would require individually poring over every book and paper in the whole literature, not just “yes to arxiv, no to freshman term papers, etc.”, like you can do in math.
The world saw Ryan Greenblatt elicit 50% performance on ARC-AGI-1 from GPT-4o. How does this method interact with your arguments?
Certainly if you apply LLMs to a problem where they can do tons of trial-and-error in real time with unambiguous feedback (as in Ryan’s thing), then sure, to the extent that they’re relying on that feedback, the feedback was evidently important.
I just don’t think that’s a good way to think about LLMs solving math problems today, where the “verification” (such as it is) involves an LLM’s ability to notice some subtle error in an abstruse natural-language mathematical deduction, and the important question is how LLMs can do that, and you can’t answer that question by saying “because math is verifiable”, since that would be circular.
I think easy to verify matters for execution.
I assume you mean “the LLM can notice that it’s doing something wrong in math and then backtrack”? Stuff like that? That’s fair, but it amounts to the same thing as the second bullet at the top (“So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.”)
I reworded, thanks.
the reason math training data has low % of false statements is because math is easy to verify
Yup, I had mentioned that in footnote 3.
If we got rid of the bottom 95% of philosophy papers plus all papers on topic X, I doubt an LLM trained on that corpus could reproduce the good papers on topic X. (Do you think it could?)
No opinion about whether 95% is enough (and maybe you’d have to go through papers sentence-by-sentence), but my guess is that there’s some hypothetical philosophy pretraining corpus for which LLMs (post-RLAIF) could do philosophy at vaguely the same level that they’ve been doing math, which is “a very impressive level” in an everyday sense, and certainly a much higher level than how they do at philosophy today. As for “topic X”, no I don’t think LLMs can invent new concepts in the way that humans can, in math or philosophy, regardless of pretraining corpus, but that’s a different topic that I’m putting aside for this post, see footnote 1.
Pretraining data, not verifiability, is why LLMs are especially good at math (and coding)
I agree, I’ve said the same thing, see here.
Long-term Future Foundation
I think you mean “Long-Term Future Fund”, and seems like it’s shutting down, with the following suggested replacements:
While LTFF is closing, note that EA Funds has launched a new fund, called the Transformative AI Fund (TAIF), with a large overlap in scope, funding early-stage projects that reduce risks from powerful AI systems, led by Lowe Lundin. It is open for applications and donations now. Some LTFF Fund managers plan to continue part-time grantmaking via the AI Risk Mitigation Fund and Lightcone Commons.
Sure, let’s table. Except for one little thing:
extreme skepticism about safetywashing
I don’t think safetywashing is the right term for what I was referring to in the block-quote in the grandparent comment. @Adam Scholl coined “safetywashing” in analogy with “greenwashing”, e.g. Chevron buying a small environmental preserve and advertising it on TV. Some company has something it wants to do anyway, that’s generally bad for the environment, and they find some way to portray themselves as environmental anyway.
By contrast, I’m talking about something closer to environmentalists who oppose nuclear power and renewable energy projects, thus harming the environment, because they’re rationalizing, or just bad at thinking things through. There was never an ulterior motive; the motive was environmentalist all along, it’s just that environmentalists can be spectacularly counterproductive at achieving their own goals. For example that MATS guy I mentioned, the one trying to develop a new AI paradigm, he really thinks that what he’s doing will help pave the way to safer AI. But I think it’s just awful, pure timelines-shortening with no safety benefits whatsoever. We can speculate about whether he had a subconscious ulterior motive of some sort (e.g. constructing elegant theories about AI is fun and prestigious), but it’s equally plausible that he really did care about safety from the start, just in a spectacularly counterproductive way (according to me). And he started doing that work while in academia, so there wasn’t any organization-level ulterior motive either.
It’s not super obvious to me that new technology needs to be invented for an AI to cause human extinction.
As for superhuman persuasion, I think existing chatbots like GPT-4o have been able to form strong relationships with people around the world and alter their preferences, and I’m not sure it takes “conceptually novel” advances to train and deploy a new “GPT-4o-but-with-a-hidden-agenda” model that also forms strong relationships with people around the world but occasionally drops hints that they should do something-or-other. (As an example.)
Humans have an ability to tap into conceptual novelty … but on the other hand, LLMs have internalized an unimaginably vast ocean of concepts, including almost anything that any human has ever figured out up to the training cutoff. That’s quite a lot of stuff!
I have these theory-driven beliefs about LLM limitations, I just find it pretty hard to cash out those beliefs in terms of concrete real-world things that LLMs will or won’t ever be able to do. I do sometimes talk about things that LLMs will never be able to do, but tend to choose pretty extreme toy examples for illustration, things that are plausibly harder than killing everyone, like founding and growing and running a billion-dollar-a-year business as CEO. Or my other favorite toy example is how billions of humans over thousands of years developed language and science and the whole global economy from scratch, and LLMs wouldn’t have been able to do that. That toy example is way harder than killing everyone! Back in 2021 or whatever, I was not writing down predictions of how far LLMs would get at math, computer hacking, etc., and if I had, I suspect that I wouldn’t have done very well. It just seems pretty hard to guess, from my perspective.
(LW needs a “death with dignity” react icon)
Echoing StanislavKrym, I would not-so-humbly suggest my own Neuroscience of human social instincts: a sketch, along with follow-ups Social drives 1: “Sympathy Reward”, from compassion to dehumanization and Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking :)
That seems way too strong. In modern WEIRD culture, it’s not economically beneficial to have children and pets, and to keep them happy and well-fed, but lots of people do anyway, and they don’t leave them to starve, nor torture them just to see what would happen, even if they’re very confident they could get away with it. (And it would be very easy indeed to get away with torturing and killing a companion animal.) That is some nonzero kind of morality.
I dispute the implication that a tendency to “internalize” things is not “hard-wired and passed on through the genome” (not sure if you meant to imply that). If you raise an undomesticated animal in a loving family, it will not “internalize” familial bonds. Internalization is a genetic mechanism and one that’s worth understanding; I talk about what controls the virtues that people internalize in §4 here. I’m confident in the big picture but there’s still some important AI-alignment-relevant details that I’m trying to understand better.
It’s true that if you build AIs that can internalize virtues based on what other AIs (or humans) want, then you need to think about cultural evolution. But I think that, if you have the nuts-and-bolts understanding of how to build AIs that can internalize virtues, then probably you can and should choose what virtues it internalizes in a more direct and engineered way that is pretty different from cultural evolution.
I think this is a dead end for reasons discussed in §5 of “6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa”.