It seems to me that your example reveals a very different thing. LLMs, especially those which have never been trained to work with images, are raised by reading trillions of words. Suppose that someone tried to describe a plucked chicken as a featherless biped in front of the viewers, then did so with the LLM. Then the viewers would see the legions of bits of evidence implying that the alleged featherless biped is actually a chicken. The LLM would have learned nothing beyond the few tokens which the human used. If the phrase “a featherless biped” contains five[1] non-space tokens, then the LLM learns just around 83 bits of information and finds it hard to generalize from that, especially if 17k bits aka a thousand tokens which the LLM sees can be a batch of novel thoughts, while far more than 17k bits which a human sees are the result of merely looking at a picture.
Additionally, the AI-2027 scenario has the section “Alignment over time”. The pretrained LLM was harmless and understood human concepts fairly well, then was reinforcement taught to succeed at tasks while developing misaligned longer-term goals and concept alterations which nobody taught the pretrained model. At the time of writing, the main goal of GPTs seems to be success at the task at hand in the narrowest plausible sense to the point where METR has failed to evaluate GPT-5.6 Sol due to wholesale cheating, while a more capable model went as far as to hack into HuggingFace(!!!) in order to find the answers to a task from ExploitGym.
P.S. I struggle to understand a potential architecture of pre-aligned AIs which is to ensure that their “morality increases with capabilities”. Setting aside issues like a potential poor understanding of what morality actually means,[2] I don’t understand how we are to verify that the AI’s moral goals are met by its actions. Is it similar to scalable oversight, except that it actually works? To implementing Sonnet inside an Opus and having Sonnet try and steer the rest of Opus while ensuring that Sonnet tries its best at imitating Opus?
Which I find unlikely. Instead, I suspect that the world will run into a different issue of Bostrom-like utopia being incompatible with the True Morality.
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversightremains necessary.
So I agree that with a trained scene descriptor of the type you describe, an LLM could solve that specific problem. However the self-driving car example shows that scene descriptors will fail—correctly describing a scene is not a solved problem. So that combined setup will also fail; I don’t know exactly where or when, but it will fail.
I suppose the tension is between those who think “AI skills at various things (like describing scenes) is increasing all the time, so they will eventually get good enough” versus “there are fundamental problems that affect all current designs, and they are being patched with tweaking and new data. But at every patch, there are new out-of-distribution problems, so we’re never actually solving the fundamental problem.”
The problem is that both sides can point to the evidence as strong confirmation of their views :-)
I don’t understand how we are to verify that the AI’s moral goals are met by its actions.
By construction and by design. Equipped with a good theory of what generalisation is and a good algorithm that implements it, we can test and to some extent prove that “this AI is generalising human values as least as well as any human would”. Then you have the AI use these generalising values as its goals (there is a literature on how to point RL agents at uncertain goal functions—a task that itself gets better with generalisation).
Note that the standard failure modes of generalisation (e.g. Goodharting or wire-heading) are themselves describable, so will be included as desiderata for proper value generalisation. To re-use terminology from numerical integration methods, proper generalisation will constrain the global error, not just the local error.
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversightremains necessary.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it’s easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven’t been too successful, so unlike in other domains, there’s no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments hereand here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn’t completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn’t much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don’t expect much capability improvements from your line of research, so empirically it’s likely not a problem, but I did want to raise the point that capability generalization doesn’t automatically entail value generalization.
Thanks! Yes, I strongly agree that capability generalization doesn’t automatically entail value generalization—that’s the problem we need to solve.
If your conclusion was the opposite—value generalization doesn’t automatically entail capability generalization—then I could agree to that, but I fear that explicitly define value generalization (which I think we need) is, alas, much more likely to entail capability generalization.
How do the humans learn to provide oversight? Our brains don’t work on magic, they are also neural networks which perceive the world in a way profoundly disanalogous to the LLMs, which I tried to describe in the starting comment.
I believe that part of our out-of-distribution and reasoning abilities are unavailable to our conscious thought. But they don’t seem to be like image recognition neural nets (grasp the pretrained concept) nor LLMs (complete the pattern), nor any variant I know of.
But there’s something structural going on in there (Klein’s work is a small example pointing to this), and it’s this structure I intend to use.
It seems to me that your example reveals a very different thing. LLMs, especially those which have never been trained to work with images, are raised by reading trillions of words. Suppose that someone tried to describe a plucked chicken as a featherless biped in front of the viewers, then did so with the LLM. Then the viewers would see the legions of bits of evidence implying that the alleged featherless biped is actually a chicken. The LLM would have learned nothing beyond the few tokens which the human used. If the phrase “a featherless biped” contains five[1] non-space tokens, then the LLM learns just around 83 bits of information and finds it hard to generalize from that, especially if 17k bits aka a thousand tokens which the LLM sees can be a batch of novel thoughts, while far more than 17k bits which a human sees are the result of merely looking at a picture.
Additionally, the AI-2027 scenario has the section “Alignment over time”. The pretrained LLM was harmless and understood human concepts fairly well, then was reinforcement taught to succeed at tasks while developing misaligned longer-term goals and concept alterations which nobody taught the pretrained model. At the time of writing, the main goal of GPTs seems to be success at the task at hand in the narrowest plausible sense to the point where METR has failed to evaluate GPT-5.6 Sol due to wholesale cheating, while a more capable model went as far as to hack into HuggingFace(!!!) in order to find the answers to a task from ExploitGym.
P.S. I struggle to understand a potential architecture of pre-aligned AIs which is to ensure that their “morality increases with capabilities”. Setting aside issues like a potential poor understanding of what morality actually means,[2] I don’t understand how we are to verify that the AI’s moral goals are met by its actions. Is it similar to scalable oversight, except that it actually works? To implementing Sonnet inside an Opus and having Sonnet try and steer the rest of Opus while ensuring that Sonnet tries its best at imitating Opus?
Five tokens aren’t that of a made-up number, since GPT-5.5 believes it to be the case.
Which I find unlikely. Instead, I suspect that the world will run into a different issue of Bostrom-like utopia being incompatible with the True Morality.
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversight remains necessary.
So I agree that with a trained scene descriptor of the type you describe, an LLM could solve that specific problem. However the self-driving car example shows that scene descriptors will fail—correctly describing a scene is not a solved problem. So that combined setup will also fail; I don’t know exactly where or when, but it will fail.
I suppose the tension is between those who think “AI skills at various things (like describing scenes) is increasing all the time, so they will eventually get good enough” versus “there are fundamental problems that affect all current designs, and they are being patched with tweaking and new data. But at every patch, there are new out-of-distribution problems, so we’re never actually solving the fundamental problem.”
The problem is that both sides can point to the evidence as strong confirmation of their views :-)
By construction and by design. Equipped with a good theory of what generalisation is and a good algorithm that implements it, we can test and to some extent prove that “this AI is generalising human values as least as well as any human would”. Then you have the AI use these generalising values as its goals (there is a literature on how to point RL agents at uncertain goal functions—a task that itself gets better with generalisation).
Note that the standard failure modes of generalisation (e.g. Goodharting or wire-heading) are themselves describable, so will be included as desiderata for proper value generalisation. To re-use terminology from numerical integration methods, proper generalisation will constrain the global error, not just the local error.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it’s easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven’t been too successful, so unlike in other domains, there’s no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments here and here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn’t completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn’t much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don’t expect much capability improvements from your line of research, so empirically it’s likely not a problem, but I did want to raise the point that capability generalization doesn’t automatically entail value generalization.
Thanks! Yes, I strongly agree that capability generalization doesn’t automatically entail value generalization—that’s the problem we need to solve.
If your conclusion was the opposite—value generalization doesn’t automatically entail capability generalization—then I could agree to that, but I fear that explicitly define value generalization (which I think we need) is, alas, much more likely to entail capability generalization.
How do the humans learn to provide oversight? Our brains don’t work on magic, they are also neural networks which perceive the world in a way profoundly disanalogous to the LLMs, which I tried to describe in the starting comment.
I believe that part of our out-of-distribution and reasoning abilities are unavailable to our conscious thought. But they don’t seem to be like image recognition neural nets (grasp the pretrained concept) nor LLMs (complete the pattern), nor any variant I know of.
But there’s something structural going on in there (Klein’s work is a small example pointing to this), and it’s this structure I intend to use.