The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversightremains necessary.
So I agree that with a trained scene descriptor of the type you describe, an LLM could solve that specific problem. However the self-driving car example shows that scene descriptors will fail—correctly describing a scene is not a solved problem. So that combined setup will also fail; I don’t know exactly where or when, but it will fail.
I suppose the tension is between those who think “AI skills at various things (like describing scenes) is increasing all the time, so they will eventually get good enough” versus “there are fundamental problems that affect all current designs, and they are being patched with tweaking and new data. But at every patch, there are new out-of-distribution problems, so we’re never actually solving the fundamental problem.”
The problem is that both sides can point to the evidence as strong confirmation of their views :-)
I don’t understand how we are to verify that the AI’s moral goals are met by its actions.
By construction and by design. Equipped with a good theory of what generalisation is and a good algorithm that implements it, we can test and to some extent prove that “this AI is generalising human values as least as well as any human would”. Then you have the AI use these generalising values as its goals (there is a literature on how to point RL agents at uncertain goal functions—a task that itself gets better with generalisation).
Note that the standard failure modes of generalisation (e.g. Goodharting or wire-heading) are themselves describable, so will be included as desiderata for proper value generalisation. To re-use terminology from numerical integration methods, proper generalisation will constrain the global error, not just the local error.
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversightremains necessary.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it’s easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven’t been too successful, so unlike in other domains, there’s no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments hereand here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn’t completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn’t much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don’t expect much capability improvements from your line of research, so empirically it’s likely not a problem, but I did want to raise the point that capability generalization doesn’t automatically entail value generalization.
Thanks! Yes, I strongly agree that capability generalization doesn’t automatically entail value generalization—that’s the problem we need to solve.
If your conclusion was the opposite—value generalization doesn’t automatically entail capability generalization—then I could agree to that, but I fear that explicitly define value generalization (which I think we need) is, alas, much more likely to entail capability generalization.
How do the humans learn to provide oversight? Our brains don’t work on magic, they are also neural networks which perceive the world in a way profoundly disanalogous to the LLMs, which I tried to describe in the starting comment.
I believe that part of our out-of-distribution and reasoning abilities are unavailable to our conscious thought. But they don’t seem to be like image recognition neural nets (grasp the pretrained concept) nor LLMs (complete the pattern), nor any variant I know of.
But there’s something structural going on in there (Klein’s work is a small example pointing to this), and it’s this structure I intend to use.
The problem with LLMs and other modern designs is that once you turn a specific problem into an objective (or a benchmark) it’s generally easy to modify the AI to solve it.
Fully autonomous self-driving cars don’t exist. The biggest reason for this is that the AIs in them can’t generalise consistently to novel situations, and so human oversight remains necessary.
So I agree that with a trained scene descriptor of the type you describe, an LLM could solve that specific problem. However the self-driving car example shows that scene descriptors will fail—correctly describing a scene is not a solved problem. So that combined setup will also fail; I don’t know exactly where or when, but it will fail.
I suppose the tension is between those who think “AI skills at various things (like describing scenes) is increasing all the time, so they will eventually get good enough” versus “there are fundamental problems that affect all current designs, and they are being patched with tweaking and new data. But at every patch, there are new out-of-distribution problems, so we’re never actually solving the fundamental problem.”
The problem is that both sides can point to the evidence as strong confirmation of their views :-)
By construction and by design. Equipped with a good theory of what generalisation is and a good algorithm that implements it, we can test and to some extent prove that “this AI is generalising human values as least as well as any human would”. Then you have the AI use these generalising values as its goals (there is a literature on how to point RL agents at uncertain goal functions—a task that itself gets better with generalisation).
Note that the standard failure modes of generalisation (e.g. Goodharting or wire-heading) are themselves describable, so will be included as desiderata for proper value generalisation. To re-use terminology from numerical integration methods, proper generalisation will constrain the global error, not just the local error.
I tend to agree with this, but I will note that home robotics tasks are much better tests of generalization than most benchmarks, and the short answer to why this is comes down to OOD pressure from humans is essentially always there, plus it’s easier to restrain how much training data LLMs having access to robotics have in the physical world, plus generalization is a current bottleneck, and unlike other fields, robotics has tried to make much less general robots useful, and they haven’t been too successful, so unlike in other domains, there’s no overhang/inefficient market to exploit here.
I talk about this more at length in 2 comments here and here, and my main takeaway is pre-trained LLMs are both worse than humans at generalization, and also people tend to overestimate how much worse they are, and this means the LLM paradigm isn’t completely off the mark in the way past AI paradigms were (because they would have essentially no ability to do any of this before today.)
Notably though, this is only capabilities generalization, and more generally capability generalization and value generalization really are different, even if they are strongly correlated now.
This isn’t much of a problem if you expect your value generalization approach does work very well and as a side effect induces capability generalization, and truthfully I don’t expect much capability improvements from your line of research, so empirically it’s likely not a problem, but I did want to raise the point that capability generalization doesn’t automatically entail value generalization.
Thanks! Yes, I strongly agree that capability generalization doesn’t automatically entail value generalization—that’s the problem we need to solve.
If your conclusion was the opposite—value generalization doesn’t automatically entail capability generalization—then I could agree to that, but I fear that explicitly define value generalization (which I think we need) is, alas, much more likely to entail capability generalization.
How do the humans learn to provide oversight? Our brains don’t work on magic, they are also neural networks which perceive the world in a way profoundly disanalogous to the LLMs, which I tried to describe in the starting comment.
I believe that part of our out-of-distribution and reasoning abilities are unavailable to our conscious thought. But they don’t seem to be like image recognition neural nets (grasp the pretrained concept) nor LLMs (complete the pattern), nor any variant I know of.
But there’s something structural going on in there (Klein’s work is a small example pointing to this), and it’s this structure I intend to use.