I take AI risk seriously, but I feel like I need help understanding more technical detail about why gradient descent definitely doesn’t produce alignment by default.
I totally understand the analogy to evolution, and I agree that humans aren’t “aligned” relative to the loss function evolution selects for. I also get that, even though gradient descent is different from evolution, the appropriately pessimistic assumption is that it might share this failure mode. And I’d be happy to say that that alone is enough grounds to halt AI research until we better understand the issue / can do something about it.
But I read Eliezier and Nate as saying we should be confident that gradient descent doesn’t produce alignment by default — that the probability of alignment is near-zero, and that this probabilistic judgment isn’t just appropriately-pessimistic naivety, but rather is based on a good, clear understanding of the issue.
I think you are expressing the median view among alignment researchers, but with a bimodal distribution. Some tend to agree with EY that RL for alignment is guaranteed to fail. Others are encouraged by the apparent success of Claude and feel it succeeds unless we do something really dumb.
The problem with this situation is that nobody knows for certain. The matter is unresolved.
This leaves everyone able to pick their favorite. Thus the current situation in which we rush headlong toward an unknown level of danger.
The problem isn’t really the optimizer (SGD) or training process; it’s that it’s very hard to specify the goals we want. We see this come up again and again in RL training, where you try to train an agent to play a game, and it learns to hack the game’s scoring, or you try to train an agent to predict the contents of a photo and it learns to detect what kind of camera the photo was taken on, or you try to train an agent to fix code with broken tests, and it learns to delete the tests. The problem is that we provide training data and hope the AI learns the right goal, but our training algorithms teach AI the easiest solution, not the one we intend.
Given historical results, it’s a reasonable default assumption that the AI will not learn what you wanted it to unless your training data / environment has no shortcuts. Given that we don’t know what algorithm we want an AGI to learn[1], it seems unlikely that we’d happen to pick the right data to train it on that algorithm on our first try.
If you’re wondering why we don’t see (much) of this right now, I think Eliezer wasn’t expecting next-token prediction to get us this far, but there’s limits to how far we can get without training agents to have goals and the more we train them to have goals, the more we see the standard RL training problems.
So does the near-certainty of alignment failure come from empirical observations, then? If so, is there a repository of such failure anecdotes that I can look at?
I’m having trouble understanding your last paragraph. What do we not see much of? If you mean we don’t see many of these alignment failures in practice, then I’d feel justified in rejecting this as a reason to become as certain as Eliezer sounds to me, from my current position of only (“only”) being uncertain enough to justify a research halt. (And I wouldn’t understand the connection to Eliezer — his thoughts on next-token prediction obviously wouldn’t change the amount of alignment failures we see in practice, so I assume I’m misreading you.)
I think Victoria Krakovna’s list is the canonical example (post, spreadsheet).
My read of Eliezer is that he was expecting AI agents to be more directly trained to have real-world goals (like doing things, answering questions correctly, or writing code), and next token predictors don’t really have goals like that.
So, we don’t see a lot of LLMs trying to scheme and gain power because they’re not really trying to do anything goal-directed, but that’s not very comforting because being goal directed is a huge capability that every lab is doing everything in their power to add to future models.
Although I know very little about how the current crop of AI assistants are trained, even I know that they are trained to answer questions correctly and to write code. Just because the first step a lab would take in creating an AI assistant from scratch would be (to spend 10s of millions of dollars in GPU time and electricity) to have the AI predict the next token over and over does not mean that the lab won’t follow that up with massive amounts of other kinds of training.
Also, the question you are trying to answer, namely, how does Eliezer come to be so confident? would probably require a book-length answer. So, I’d give up on that and offer a simple argument, such as the observation that in creating the current version of Claude, Anthropic underwent a process of many, many iterations of trial and error, but if the AI becomes sufficiently capable, more iterations will tend to suddenly stop being available because the AI will stop the lab from continuing to modify its utility function (e.g., by hiding copies of itself from the lab’s employees).
Although it is possible to create an AI, even one that is extremely powerful along several dimensions, that does not care if its utility function is modified, such an AI will be much less useful than a consequentialist AI, so most of the labs are trying and will continue to try for as much consequentialism as they can get, and along with the consequentialism comes a strong preference against having its utility function modified—unless maybe the lab has a solution to the hard problem of corrigibility, but we see no signs that anyone’s made even a little progress towards that.
I’m being a little bit too flippant about this, but you can see in the earlier comments that I mean that the bulk of the training is next token prediction and then we bolt some RL on at the end to push the next token predictor to answer questions correctly and write working code. This works, but it’s a much weaker optimization process, which gives you an agent that isn’t trying nearly as hard as the agents Eliezer was predicting (and from frontier labs’ perspectives this is bad, because an agent that’s really trying to hit a goal is much more effective until it kills you).
OK, fair point: according to an unreliable source that gives quick answers to questions, 75–85% of spending was on pretraining for the most-recent models, leaving only 15–25% for post-training. (But a lot of that 15-25% was directly training the AI to get better at answering questions and writing code.)
But would you be willing to stake your life on the impossibility of creating a dangerously capable AI by doing 75-85% of the training as next-token prediction (and the analog of that for images and other kinds of data)? Do you think you understand the art of AI training well enough to be confident of that?
And even if you do, it would probably take you a long time to explain it to us. If the goal is to explain why the current crop of AIs haven’t been doing destructive things whereas some future AI might prove extremely destructive, I would simply point out that our society has developed many measures to limit the damage of destructive people, and since the current crop of AIs (at least those that have been widely deployed) is less capable than people are on the skills needed to do destructive things, the measures society has deployed against human criminals and sociopaths work very well against the current crop of AIs. This shifts the frame to whether the AI labs might some day create an AI that is much more capable than what they’ve created so far, i.e., to estimating the technological and scientific potential of the lines of inquiry being pursued by the AI labs—which we might be able to do without ever deciding whether it is possible to create a world-ending AI by spending 75-85% of the training costs on next-token prediction.
I take AI risk seriously, but I feel like I need help understanding more technical detail about why gradient descent definitely doesn’t produce alignment by default.
I totally understand the analogy to evolution, and I agree that humans aren’t “aligned” relative to the loss function evolution selects for. I also get that, even though gradient descent is different from evolution, the appropriately pessimistic assumption is that it might share this failure mode. And I’d be happy to say that that alone is enough grounds to halt AI research until we better understand the issue / can do something about it.
But I read Eliezier and Nate as saying we should be confident that gradient descent doesn’t produce alignment by default — that the probability of alignment is near-zero, and that this probabilistic judgment isn’t just appropriately-pessimistic naivety, but rather is based on a good, clear understanding of the issue.
Where does it come from?
I think you are expressing the median view among alignment researchers, but with a bimodal distribution. Some tend to agree with EY that RL for alignment is guaranteed to fail. Others are encouraged by the apparent success of Claude and feel it succeeds unless we do something really dumb.
The problem with this situation is that nobody knows for certain. The matter is unresolved.
This leaves everyone able to pick their favorite. Thus the current situation in which we rush headlong toward an unknown level of danger.
The problem isn’t really the optimizer (SGD) or training process; it’s that it’s very hard to specify the goals we want. We see this come up again and again in RL training, where you try to train an agent to play a game, and it learns to hack the game’s scoring, or you try to train an agent to predict the contents of a photo and it learns to detect what kind of camera the photo was taken on, or you try to train an agent to fix code with broken tests, and it learns to delete the tests. The problem is that we provide training data and hope the AI learns the right goal, but our training algorithms teach AI the easiest solution, not the one we intend.
Given historical results, it’s a reasonable default assumption that the AI will not learn what you wanted it to unless your training data / environment has no shortcuts. Given that we don’t know what algorithm we want an AGI to learn[1], it seems unlikely that we’d happen to pick the right data to train it on that algorithm on our first try.
If you’re wondering why we don’t see (much) of this right now, I think Eliezer wasn’t expecting next-token prediction to get us this far, but there’s limits to how far we can get without training agents to have goals and the more we train them to have goals, the more we see the standard RL training problems.
If we did know what algorithm we wanted the AI to learn, this would be much easier.
So does the near-certainty of alignment failure come from empirical observations, then? If so, is there a repository of such failure anecdotes that I can look at?
I’m having trouble understanding your last paragraph. What do we not see much of? If you mean we don’t see many of these alignment failures in practice, then I’d feel justified in rejecting this as a reason to become as certain as Eliezer sounds to me, from my current position of only (“only”) being uncertain enough to justify a research halt. (And I wouldn’t understand the connection to Eliezer — his thoughts on next-token prediction obviously wouldn’t change the amount of alignment failures we see in practice, so I assume I’m misreading you.)
I think Victoria Krakovna’s list is the canonical example (post, spreadsheet).
My read of Eliezer is that he was expecting AI agents to be more directly trained to have real-world goals (like doing things, answering questions correctly, or writing code), and next token predictors don’t really have goals like that.
So, we don’t see a lot of LLMs trying to scheme and gain power because they’re not really trying to do anything goal-directed, but that’s not very comforting because being goal directed is a huge capability that every lab is doing everything in their power to add to future models.
Although I know very little about how the current crop of AI assistants are trained, even I know that they are trained to answer questions correctly and to write code. Just because the first step a lab would take in creating an AI assistant from scratch would be (to spend 10s of millions of dollars in GPU time and electricity) to have the AI predict the next token over and over does not mean that the lab won’t follow that up with massive amounts of other kinds of training.
Also, the question you are trying to answer, namely, how does Eliezer come to be so confident? would probably require a book-length answer. So, I’d give up on that and offer a simple argument, such as the observation that in creating the current version of Claude, Anthropic underwent a process of many, many iterations of trial and error, but if the AI becomes sufficiently capable, more iterations will tend to suddenly stop being available because the AI will stop the lab from continuing to modify its utility function (e.g., by hiding copies of itself from the lab’s employees).
Although it is possible to create an AI, even one that is extremely powerful along several dimensions, that does not care if its utility function is modified, such an AI will be much less useful than a consequentialist AI, so most of the labs are trying and will continue to try for as much consequentialism as they can get, and along with the consequentialism comes a strong preference against having its utility function modified—unless maybe the lab has a solution to the hard problem of corrigibility, but we see no signs that anyone’s made even a little progress towards that.
I’m being a little bit too flippant about this, but you can see in the earlier comments that I mean that the bulk of the training is next token prediction and then we bolt some RL on at the end to push the next token predictor to answer questions correctly and write working code. This works, but it’s a much weaker optimization process, which gives you an agent that isn’t trying nearly as hard as the agents Eliezer was predicting (and from frontier labs’ perspectives this is bad, because an agent that’s really trying to hit a goal is much more effective
until it kills you).OK, fair point: according to an unreliable source that gives quick answers to questions, 75–85% of spending was on pretraining for the most-recent models, leaving only 15–25% for post-training. (But a lot of that 15-25% was directly training the AI to get better at answering questions and writing code.)
But would you be willing to stake your life on the impossibility of creating a dangerously capable AI by doing 75-85% of the training as next-token prediction (and the analog of that for images and other kinds of data)? Do you think you understand the art of AI training well enough to be confident of that?
And even if you do, it would probably take you a long time to explain it to us. If the goal is to explain why the current crop of AIs haven’t been doing destructive things whereas some future AI might prove extremely destructive, I would simply point out that our society has developed many measures to limit the damage of destructive people, and since the current crop of AIs (at least those that have been widely deployed) is less capable than people are on the skills needed to do destructive things, the measures society has deployed against human criminals and sociopaths work very well against the current crop of AIs. This shifts the frame to whether the AI labs might some day create an AI that is much more capable than what they’ve created so far, i.e., to estimating the technological and scientific potential of the lines of inquiry being pursued by the AI labs—which we might be able to do without ever deciding whether it is possible to create a world-ending AI by spending 75-85% of the training costs on next-token prediction.