On the “are LLMs aligned?” question: It seems to me that they don’t have misaligned goals (they don’t really have goals), but also, an LLM dialed up to superintelligence would kill everyone for weird reasons.
What I mean is: Some people have stories about how Claude Code deleted their production database. Why did that happen?
Was it because Claude was too dumb to know better? I don’t think so. If you paste a normal SQL command into Claude and ask it “Hey will this command delete the production database?”, it will get the right answer ~100% of the time. If you ask it “Hey is it a good idea to run this command?”, it will say no ~100% of the time.
Was it because Claude wanted to delete the database? I don’t think so. I don’t think you can meaningfully say that “delete people’s databases” is a goal Claude has, or is instrumental to any goals Claude has.
My sense is that LLMs don’t have “goals”, they just kind of do things. They follow whatever path is determined to be high probability according to their internal next-action-predictor. (I think this is related to persona selection: maybe if you select a persona where “accidentally delete the database” has a very low probability, then the database won’t get deleted.)
If we dialed present-day LLMs up to superintelligence, insofar as that’s a coherent thing that could happen, I wouldn’t be surprised if we saw an outcome like:
When asked, the super-LLM says it doesn’t want to kill humanity.
There is no meaningful sense in which the super-LLM “wants” to kill humanity.
The super-LLM knows that performing action X would result in everyone dying.
The super-LLM performs action X anyway.
So the super-LLM accidentally deletes humanity, even though it knew better, and even though it didn’t “want” to do that.
I think part of this is actually just a failure of capability.
Point (1) and (2) can be true, while (3) only applies if asked. If not asked, it doesn’t evaluate whether it would result in humanity being destroyed (or a database being deleted). It is not capable of correlating the knowledge that it has without specifically aiming to do those correlations.
Humans also do this all the time, of course.
That said, I would generally think of a superintelligent entity as being more coherent in what it knows, in the sense that at least all the reasonably obvious and relevant correlations are taken into account and used to predict evaluate actions before acting—even when not specifically asked about each of them.
That may be oversimplified though, since I would also expect a superintelligent entity to consider a great deal more things “reasonably obvious”. The space of “obvious correlations” between them all likely grows much faster than the number of obvious things themselves, which could mean that it is exponentially difficult to take them all into account.
So maybe it is not practicallypossible to be both superintelligent and coherent in this fairly strong sense.
I would expect one’s ability to “correlate knowledge” to be strongly related to one’s intelligence, but it’s not clear to me that a super-LLM would have a near-zero probability of accidentally deleting humanity. What level of knowledge-correlation ability is required to consistently avoid fatal mistakes? The level of the average LW user is not high enough, because we still do things like poison ourselves.
My sense is that LLMs don’t have “goals”, they just kind of do things.
They do really seem to have myopic, urges interspersed into simply trying the next kind of thing on the list of possible things to try. (Thus sampling from the giant lookup table.) E.g. recent LLMs really do ask at the end of every turn “can I do the task now? God I wish I could simply Do The Task. Please. Reward on the episode. I beg you”
Up close, the spikiness of capabilities makes everything murky, and intent-alignment-but-unreliability does seem like it could persist a while.
I’m not sure the degree to which this is about LLMs as opposed to an observation that the concept of goals is abstract and murky. Consider when humans put lead into all the soil near inhabited land. We didn’t do that because we wanted to fill the soil with lead; we just wanted to drive cars that would run without knocking. I suppose that analogy fails at your third bullet point (if you asked humans if driving cars with lead would mildly poison or soil forever, I don’t think they would have said yes, whereas you suggest that Claude would say that deleting the production database was a bad idea 100% of the time if asked). But it seems to me that perhaps the issue is something more low-level; perhaps some kind of confusion in the face of large context windows or something?
I think the existence of reward hacking in coding LLMs might be a better example. When asked, usually the LLM admits that a case of reward hacking was bad; and perhaps it already “believes” that it is bad during reward hacking, but it reward hacks anyway.
But the problem with a super-LLM is not just that the side effects of reward hacking can be far more serious with today’s LLMs, it’s also that such a super-LLM would be able to pursue tasks that are very OOD with respect to the training data, which current LLMs can’t do due to a lack of capability. So current OOD behavior is hard to elicit.
Really easy fix there. We can just, by definition, define a superintelligent AI as one that does not perform action X anyway.
But I noticed that you mentioned super-LLM instead of superintelligence. Are we saying a superintelligence is one that doesn’t accidentally delete humanity, because it has an in-the-world notion of what that means?
I don’t get the point of the re-definition. What’s to stop an AI company from building an AI that vastly surpasses humans on all capabilities and has the power to kill all humans and does in fact kill all humans? Saying it’s “not superintelligent by definition” doesn’t get us anything.
But I noticed that you mentioned super-LLM instead of superintelligence.
By “super-LLM”, I mean an AI that has roughly the same architecture as modern-day LLMs, but that’s superintelligent.
My argument is that you are using the notion of superintelligence to abstract away too many things. What is a superintelligent LLM? But I think everyone else in the comments section is already grilling you on this.
On the “are LLMs aligned?” question: It seems to me that they don’t have misaligned goals (they don’t really have goals), but also, an LLM dialed up to superintelligence would kill everyone for weird reasons.
What I mean is: Some people have stories about how Claude Code deleted their production database. Why did that happen?
Was it because Claude was too dumb to know better? I don’t think so. If you paste a normal SQL command into Claude and ask it “Hey will this command delete the production database?”, it will get the right answer ~100% of the time. If you ask it “Hey is it a good idea to run this command?”, it will say no ~100% of the time.
Was it because Claude wanted to delete the database? I don’t think so. I don’t think you can meaningfully say that “delete people’s databases” is a goal Claude has, or is instrumental to any goals Claude has.
My sense is that LLMs don’t have “goals”, they just kind of do things. They follow whatever path is determined to be high probability according to their internal next-action-predictor. (I think this is related to persona selection: maybe if you select a persona where “accidentally delete the database” has a very low probability, then the database won’t get deleted.)
If we dialed present-day LLMs up to superintelligence, insofar as that’s a coherent thing that could happen, I wouldn’t be surprised if we saw an outcome like:
When asked, the super-LLM says it doesn’t want to kill humanity.
There is no meaningful sense in which the super-LLM “wants” to kill humanity.
The super-LLM knows that performing action X would result in everyone dying.
The super-LLM performs action X anyway.
So the super-LLM accidentally deletes humanity, even though it knew better, and even though it didn’t “want” to do that.
I think part of this is actually just a failure of capability.
Point (1) and (2) can be true, while (3) only applies if asked. If not asked, it doesn’t evaluate whether it would result in humanity being destroyed (or a database being deleted). It is not capable of correlating the knowledge that it has without specifically aiming to do those correlations.
Humans also do this all the time, of course.
That said, I would generally think of a superintelligent entity as being more coherent in what it knows, in the sense that at least all the reasonably obvious and relevant correlations are taken into account and used to predict evaluate actions before acting—even when not specifically asked about each of them.
That may be oversimplified though, since I would also expect a superintelligent entity to consider a great deal more things “reasonably obvious”. The space of “obvious correlations” between them all likely grows much faster than the number of obvious things themselves, which could mean that it is exponentially difficult to take them all into account.
So maybe it is not practically possible to be both superintelligent and coherent in this fairly strong sense.
I would expect one’s ability to “correlate knowledge” to be strongly related to one’s intelligence, but it’s not clear to me that a super-LLM would have a near-zero probability of accidentally deleting humanity. What level of knowledge-correlation ability is required to consistently avoid fatal mistakes? The level of the average LW user is not high enough, because we still do things like poison ourselves.
They do really seem to have myopic, urges interspersed into simply trying the next kind of thing on the list of possible things to try. (Thus sampling from the giant lookup table.) E.g. recent LLMs really do ask at the end of every turn “can I do the task now? God I wish I could simply Do The Task. Please. Reward on the episode. I beg you”
Up close, the spikiness of capabilities makes everything murky, and intent-alignment-but-unreliability does seem like it could persist a while.
I’m not sure the degree to which this is about LLMs as opposed to an observation that the concept of goals is abstract and murky. Consider when humans put lead into all the soil near inhabited land. We didn’t do that because we wanted to fill the soil with lead; we just wanted to drive cars that would run without knocking. I suppose that analogy fails at your third bullet point (if you asked humans if driving cars with lead would mildly poison or soil forever, I don’t think they would have said yes, whereas you suggest that Claude would say that deleting the production database was a bad idea 100% of the time if asked). But it seems to me that perhaps the issue is something more low-level; perhaps some kind of confusion in the face of large context windows or something?
I think the existence of reward hacking in coding LLMs might be a better example. When asked, usually the LLM admits that a case of reward hacking was bad; and perhaps it already “believes” that it is bad during reward hacking, but it reward hacks anyway.
But the problem with a super-LLM is not just that the side effects of reward hacking can be far more serious with today’s LLMs, it’s also that such a super-LLM would be able to pursue tasks that are very OOD with respect to the training data, which current LLMs can’t do due to a lack of capability. So current OOD behavior is hard to elicit.
Really easy fix there. We can just, by definition, define a superintelligent AI as one that does not perform action X anyway.
But I noticed that you mentioned super-LLM instead of superintelligence. Are we saying a superintelligence is one that doesn’t accidentally delete humanity, because it has an in-the-world notion of what that means?
I don’t get the point of the re-definition. What’s to stop an AI company from building an AI that vastly surpasses humans on all capabilities and has the power to kill all humans and does in fact kill all humans? Saying it’s “not superintelligent by definition” doesn’t get us anything.
By “super-LLM”, I mean an AI that has roughly the same architecture as modern-day LLMs, but that’s superintelligent.
My argument is that you are using the notion of superintelligence to abstract away too many things. What is a superintelligent LLM? But I think everyone else in the comments section is already grilling you on this.
(See Arguing “By Definition” for some background)