That’s a different argument than the one Eliezer is making. Eliezer is saying that the ASI model you train is going to be great at everything no matter what. You’re arguing that it’s difficult/not incentivized to train an AI on alignment research, and so the model companies are in practice going to fail at that.
I disagree that we can’t train an AI to do alignment research. It’s true that we can’t have models align an AGI inside an RL environment and give it a reward, but you can definitely train a model to e.g. perform interpretability research, catch instances of reward hacking, and predict what other models are going to do during deployment before you run them. All of the above are things that the labs train their models to do today.
Training the models to be good at software engineering is bad in the sense that it’s capabilities progress, but there’s a third category of things like “can take over the world” or “can manipulate humans into arbitrary actions” that are useful for takeover that models aren’t learning. “Maybe we can train a model to only help us with X, that is extremely hard but not sufficient to take over the world”, is the vague hope that I imagined Eliezer was originally responding to.
That’s a different argument than the one Eliezer is making. Eliezer is saying that the ASI model you train is going to be great at everything no matter what.
No, Eliezer is not making this argument. Clearly you can train an AI to be really good at some random narrow task (chess), and then have it not get good at other stuff. The problem is that if you hope to get certain amounts of useful work out of the AI (like doing alignment research, or helping you do a pivotal act), then then the AI will also be good at all the stuff you don’t want it to be good at. Eliezer obviously doesn’t believe that there are no skills you can train an AI to be good at without making it be good at other tasks, this obviously fails in a ton of edge-cases and doesn’t survive even the most basic of sanity-checks.
you can definitely train a model to e.g. perform interpretability research, catch instances of reward hacking, and predict what other models are going to do. All of the above are things that the labs train their models to do today.
The models are currently not remotely as good at those tasks as they are at software engineering or other things that follow very neatly from the training distribution. Models are really not good at any kind of conceptual research, and I don’t think we have much traction at making it better that is not “just make the model better at everything”. I am confident to call in @ryan_greenblatt on whether this is true about current model or near-future models, if you trust his judgement here.
catch instances of reward hacking, and predict what other models are going to do
The former, maybe, the latter no? Models are not very good at conceptual work, and the latter would require performing complicated macro analysis doing lots of conceptual work that current models really suck at.
“Functionally predict what X language model is going to do” is almost the quintessentially short horizon task that is easy to setup inside an environment. I do not know what proportion of spend Anthropic has allocated toward interpretabilityish tasks like this compared to SWE, or how effective they are at “conceptual research”, but you can spend money on that task and get a model to do a better job, that is just a fact. Likewise for other subtasks that matter for ensuring models are aligned, like reviewing transcripts and spotting misbehavior. Just like how, before the model companies have completely automated the full loop of what a software engineer does, they can do RL on verifiable subsections of the task like “implement this feature and don’t make any bugs.”
No, Eliezer is not making this argument
You’re right, I bastardized it in that comment in my haste, but I did not do so in the post. The AGI Ruin post says:
8. The best and easiest-found-by-optimization algorithms for solving problems we want an AI to solve, readily generalize to problems we’d rather the AI not solve; you can’t build a system that only has the capability to drive red cars and not blue cars, because all red-car-driving algorithms generalize to the capability to drive blue cars.
“Problems we want to solve” includes (but is not limited to) those subtasks. Maybe the model companies don’t have the right incentive structure to take advantage of this, but this sentence is “wrong, at least in the manner that’s relevant for us.” I can modify the post to make it clear that I’m not necessarily saying the model companies are going to use this as much as necessary.
but you can spend money on that task and get a model to do a better job, that is just a fact
You can’t spend money on that task and get a model that is better at that job, without also making it better at all the other jobs. Like, we really actually do not have a way to only make models narrowly better in some domain. You can go a bit beyond the frontier on the margin with specialized RL environments and some specialized RLHF, but you can’t go substantially beyond the frontier (this is basically what the bitter lesson is about).
The best way we have of driving model progress forward in a domain is to push model progress forward in all domains, which is centrally what Eliezer is talking about. When you train models to be good software engineers, they also become good lawyers.
The one big differential that does exist is that capability elicitation on tasks (and to some degree actual underlying performance) where you can easily verify solutions is much easier than capability elicitation on tasks where you can’t. This could hypothetically help us, but mostly hurts us, because for alignment research purposes we care more about tasks where verifying solutions is difficult, and you can probably get to RSI using tasks where solutions are relatively easy to verify.
You can go a bit beyond the frontier on the margin with specialized RL environments and some specialized RLHF, but you can’t go substantially beyond the frontier (this is basically what the bitter lesson is about).
Seems like we just like disagree on the object-level question here, and also what the Bitter Lesson implies for this situation. My current impression is that the labs got most of their generalization during pretraining, and that the primary gains since 2024 have been due to specialized RL on tasks that doesn’t generalize well, and that the massive diversity of the environments that the labs go out of their way to procure reflects this. If what you say was true, why wouldn’t the labs just train mostly on Math and then expect the models to generalize their gains to Law and SWE? It’s a lot easier to make synthetic Math datasets and they wouldn’t have to spend billions building these arenas.
I’m not an expert though; this might be better resolved if someone at or near the labs just gave us their opinion on what % of the gains from training on SWE RL environments goes to software engineering and what % actually uplifts other tasks; I’m sure they’ve measured it.
If it didn’t, why wouldn’t the labs just train on Math and then expect the models to generalize to Law and SWE?
My best guess at the actual thing that happens when you do this is that you stop making progress on getting better at math! You need a wide diversity of RL environments if you want to drive progress forward on any task. If you only have narrow RL environments you get overfitting.
Happy to have someone with more frontier lab experience chime-in, though unfortunately people are usually pretty tight-lipped about this stuff. We could ask @gwern for his take?
Kinda hard to adjudicate this without numbers, but vibes-wise I agree more with lc. I updated slightly towards longer timelines on the release of o1 / o3 due to how little the RL seemed to be generalizing. It wasn’t particularly outside my expectations, but I thought there was some chance that the RL would Just Generalize the same way that early instruction following Just Generalized, and that does not seem to be the case.
I strongly expect that if you want to make progress on math at the current margin, you would want more math environments, not other environments. And e.g. I think Claude’s somewhat worse performance on math is because Anthropic didn’t prioritize it the way GDM and OAI did.
Similarly I expect that models are getting good at software engineering because (a) companies are very actively training for it and (b) it’s unusually easy to train for (lots of online data, somewhat verifiable rewards). I don’t think either of these are true for the kind of alignment research you (Habryka) are imagining.
I strongly expect that if you want to make progress on math at the current margin, you would want more math environments, not other environments. And e.g. I think Claude’s somewhat worse performance on math is because Anthropic didn’t prioritize it the way GDM and OAI did.
To be clear, this is also my belief!
I am not saying we have pushed capabilities on our training distributions so far that literally the best way to train them is to train them on other unrelated tasks. But also, if you just went totally hard on math, you would run into overfitting issues and would get better performance if you diversify the training distribution.
Performance variation within a generation is dependent on training distribution, performance between model generations tends to follow broad capability benefits across many tasks (with a systematic bias towards stuff that is easier to generate reward for, ever since we switched towards lots of RL training).
“Functionally predict what X language model is going to do” is almost the quintessentially short horizon task that is easy to setup inside an environment.
I thought you meant “predict what the next model generation is going to do”, which seems like it would be useful, but is clearly not this. What does it even mean to predict the next token of a next token predictor? What would the purpose of it be? Do you mean something in-between?
That’s a different argument than the one Eliezer is making. Eliezer is saying that the ASI model you train is going to be great at everything no matter what. You’re arguing that it’s difficult/not incentivized to train an AI on alignment research, and so the model companies are in practice going to fail at that.I disagree that we can’t train an AI to do alignment research. It’s true that we can’t have models align an AGI inside an RL environment and give it a reward, but you can definitely train a model to e.g. perform interpretability research, catch instances of reward hacking, and predict what other models are going to do during deployment before you run them. All of the above are things that the labs train their models to do today.
Training the models to be good at software engineering is bad in the sense that it’s capabilities progress, but there’s a third category of things like “can take over the world” or “can manipulate humans into arbitrary actions” that are useful for takeover that models aren’t learning. “Maybe we can train a model to only help us with X, that is extremely hard but not sufficient to take over the world”, is the vague hope that I imagined Eliezer was originally responding to.
No, Eliezer is not making this argument. Clearly you can train an AI to be really good at some random narrow task (chess), and then have it not get good at other stuff. The problem is that if you hope to get certain amounts of useful work out of the AI (like doing alignment research, or helping you do a pivotal act), then then the AI will also be good at all the stuff you don’t want it to be good at. Eliezer obviously doesn’t believe that there are no skills you can train an AI to be good at without making it be good at other tasks, this obviously fails in a ton of edge-cases and doesn’t survive even the most basic of sanity-checks.
The models are currently not remotely as good at those tasks as they are at software engineering or other things that follow very neatly from the training distribution. Models are really not good at any kind of conceptual research, and I don’t think we have much traction at making it better that is not “just make the model better at everything”. I am confident to call in @ryan_greenblatt on whether this is true about current model or near-future models, if you trust his judgement here.
The former, maybe, the latter no? Models are not very good at conceptual work, and the latter would require performing complicated macro analysis doing lots of conceptual work that current models really suck at.
“Functionally predict what X language model is going to do” is almost the quintessentially short horizon task that is easy to setup inside an environment. I do not know what proportion of spend Anthropic has allocated toward interpretabilityish tasks like this compared to SWE, or how effective they are at “conceptual research”, but you can spend money on that task and get a model to do a better job, that is just a fact. Likewise for other subtasks that matter for ensuring models are aligned, like reviewing transcripts and spotting misbehavior. Just like how, before the model companies have completely automated the full loop of what a software engineer does, they can do RL on verifiable subsections of the task like “implement this feature and don’t make any bugs.”
You’re right, I bastardized it in that comment in my haste, but I did not do so in the post. The AGI Ruin post says:
“Problems we want to solve” includes (but is not limited to) those subtasks. Maybe the model companies don’t have the right incentive structure to take advantage of this, but this sentence is “wrong, at least in the manner that’s relevant for us.” I can modify the post to make it clear that I’m not necessarily saying the model companies are going to use this as much as necessary.
You can’t spend money on that task and get a model that is better at that job, without also making it better at all the other jobs. Like, we really actually do not have a way to only make models narrowly better in some domain. You can go a bit beyond the frontier on the margin with specialized RL environments and some specialized RLHF, but you can’t go substantially beyond the frontier (this is basically what the bitter lesson is about).
The best way we have of driving model progress forward in a domain is to push model progress forward in all domains, which is centrally what Eliezer is talking about. When you train models to be good software engineers, they also become good lawyers.
The one big differential that does exist is that capability elicitation on tasks (and to some degree actual underlying performance) where you can easily verify solutions is much easier than capability elicitation on tasks where you can’t. This could hypothetically help us, but mostly hurts us, because for alignment research purposes we care more about tasks where verifying solutions is difficult, and you can probably get to RSI using tasks where solutions are relatively easy to verify.
Seems like we just like disagree on the object-level question here, and also what the Bitter Lesson implies for this situation. My current impression is that the labs got most of their generalization during pretraining, and that the primary gains since 2024 have been due to specialized RL on tasks that doesn’t generalize well, and that the massive diversity of the environments that the labs go out of their way to procure reflects this. If what you say was true, why wouldn’t the labs just train mostly on Math and then expect the models to generalize their gains to Law and SWE? It’s a lot easier to make synthetic Math datasets and they wouldn’t have to spend billions building these arenas.
I’m not an expert though; this might be better resolved if someone at or near the labs just gave us their opinion on what % of the gains from training on SWE RL environments goes to software engineering and what % actually uplifts other tasks; I’m sure they’ve measured it.
My best guess at the actual thing that happens when you do this is that you stop making progress on getting better at math! You need a wide diversity of RL environments if you want to drive progress forward on any task. If you only have narrow RL environments you get overfitting.
Happy to have someone with more frontier lab experience chime-in, though unfortunately people are usually pretty tight-lipped about this stuff. We could ask @gwern for his take?
Kinda hard to adjudicate this without numbers, but vibes-wise I agree more with lc. I updated slightly towards longer timelines on the release of o1 / o3 due to how little the RL seemed to be generalizing. It wasn’t particularly outside my expectations, but I thought there was some chance that the RL would Just Generalize the same way that early instruction following Just Generalized, and that does not seem to be the case.
I strongly expect that if you want to make progress on math at the current margin, you would want more math environments, not other environments. And e.g. I think Claude’s somewhat worse performance on math is because Anthropic didn’t prioritize it the way GDM and OAI did.
Similarly I expect that models are getting good at software engineering because (a) companies are very actively training for it and (b) it’s unusually easy to train for (lots of online data, somewhat verifiable rewards). I don’t think either of these are true for the kind of alignment research you (Habryka) are imagining.
To be clear, this is also my belief!
I am not saying we have pushed capabilities on our training distributions so far that literally the best way to train them is to train them on other unrelated tasks. But also, if you just went totally hard on math, you would run into overfitting issues and would get better performance if you diversify the training distribution.
Performance variation within a generation is dependent on training distribution, performance between model generations tends to follow broad capability benefits across many tasks (with a systematic bias towards stuff that is easier to generate reward for, ever since we switched towards lots of RL training).
Not sure whether that changes your answer.
No, I did expect you had the same belief on the math thing. (Otherwise I wouldn’t have said “kinda hard to adjudicate” I’d have said “lc is right”.)
It just seemed like something that you might not have been fully incorporating into this discussion even though you believed it.
I thought you meant “predict what the next model generation is going to do”, which seems like it would be useful, but is clearly not this. What does it even mean to predict the next token of a next token predictor? What would the purpose of it be? Do you mean something in-between?