Former AI safety research engineer, now PhD student in philosophy of ML at Cambridge. I’m originally from New Zealand but have lived in the UK for 6 years, where I did my undergrad and masters degrees (in Computer Science, Philosophy, and Machine Learning). Blog: thinkingcomplete.blogspot.com
Richard_Ngo
Karma: 4,471
I really like this post, it feels like it draws attention to an important lack of clarity.
One thing I’d suggest changing: when introducing new terminology, I think it’s much better to use terms that are already widely comprehensible if possible, than terms based on specific references which you’d need to explain to people who are unfamiliar in each case.
So I’d suggest renaming ‘ass-number’ to wild guess and ‘foxy aggregation’ to multiple models or similar.
I’m not sure what you mean by “actual computation rather than the algorithm as a whole”. I thought that I was talking about the knowledge of the trained model which actually does the “computation” of which move to play, and you were talking about the knowledge of the algorithm as a whole (i.e. the trained model plus the optimising bot).
Rhymes with carp.
In the scenario governed by data, the part that counts as self-improvement is where the AI puts itself through a process of optimisation by stochastic gradient descent with respect to that data.
You don’t need that much hardware for data to be a bottleneck. For example, I think that there are plenty of economically valuable tasks that are easier to learn than StarCraft. But we get StarCraft AIs instead because games are the only task where we can generate arbitrarily large amounts of data.
Yudkowsky mainly wrote about recursive self-improvement from a perspective in which algorithms were the most important factors in AI progress—e.g. the brain in a box in a basement which redesigns its way to superintelligence.
Sometimes when explaining the argument, though, he switched to a perspective in which compute was the main consideration—e.g. when he talked about getting “a hyperexponential explosion out of Moore’s Law once the researchers are running on computers”.
What does recursive self-improvement look like when you think that data might be the limiting factor? It seems to me that it looks a lot like iterated amplification: using less intelligent AIs to provide a training signal for more intelligent AIs.
I don’t consider this a good reason to worry about IA, though: in a world where data is the main limiting factor, recursive approaches to generating it still seem much safer than alternatives.
Yeah, this seems like a reasonable argument. It feels like it really relies on this notion of “pretty smart” though, which is hard to pin down. There’s a case for including all of the following in that category:
Dolphins
Crows
Parrots
Elephants
Whales
Seals
Tigers
Some dogs
Octopuses
Some of the Pleistocene megafauna
And yet I’d guess that none of these were/are on track to reach human-level intelligence. Agree/disagree?
My argument is consistent with the time from dolphin- to human-level intelligence being short in our species, because for anthropic reasons we find ourselves with all the necessary features (dexterous fingers, sociality, vocal chords, etc).
The claim I’m making is more like: for every 1 species that reaches human-level intelligence, there will be N species that get pretty smart, then get stuck, where N is fairly large. (And this would still be true if neurons were, say, 10x smaller and 10x more energy efficient.)
Now there are anthropic issues with evaluating this argument by pegging “pretty smart” to whatever level the second-most-intelligent species happens to be at. But if we keep running evolution forward, I can imagine elephants, whales, corvids, octopuses, big cats, and maybe a few others reaching dolphin-level intelligence. But I have a hard time picturing any of them developing cultural evolution.
Mesa-optimizers are in the search space and would achieve high scores in the training set, so why wouldn’t we expect to see them?
I like this as a statement of the core concern (modulo some worries about the concept of mesa-optimisation, which I’ll save for another time).
With respect to formalization, I did say up front that less-formal work, and empirical work, is still valuable.
I missed this disclaimer, sorry. So that assuages some of my concerns about balancing types of work. I’m still not sure what intuitions or arguments underlie your optimism about formal work, though. I assume that this would be fairly time-consuming to spell out in detail—but given that the core point of this post is to encourage such work, it seems worth at least gesturing towards those intuitions, so that it’s easier to tell where any disagreement lies.
I have fairly mixed feelings about this post. On one hand, I agree that it’s easy to mistakenly address some plausibility arguments without grasping the full case for why misaligned mesa-optimisers might arise. On the other hand, there has to be some compelling (or at least plausible) case for why they’ll arise, otherwise the argument that ‘we can’t yet rule them out, so we should prioritise trying to rule them out’ is privileging the hypothesis.
Secondly, it seems like you’re heavily prioritising formal tools and methods for studying mesa-optimisation. But there are plenty of things that formal tools have not yet successfully analysed. For example, if I wanted to write a constitution for a new country, then formal methods would not be very useful; nor if I wanted to predict a given human’s behaviour, or understand psychology more generally. So what’s the positive case for studying mesa-optimisation in big neural networks using formal tools?
In particular, I’d say that the less we currently know about mesa-optimisation, the more we should focus on qualitative rather than quantitative understanding, since the latter needs to build on the former. And since we currently do know very little about mesa-optimisation, this seems like an important consideration.
The trained AlphaZero model knows lots of things about Go, in a comparable way to how a dog knows lots of things about running.
But the algorithm that gives rise to that model can know arbitrarily few things. (After all, the laws of physics gave rise to us, but they know nothing at all.)
I’d say that this is too simple and programmatic to be usefully described as a mental model. The amount of structure encoded in the computer program you describe is very small, compared with the amount of structure encoded in the neural networks themselves. (I agree that you can have arbitrarily simple models of very simple phenomena, but those aren’t the types of models I’m interested in here. I care about models which have some level of flexibility and generality, otherwise you can come up with dumb counterexamples like rocks “knowing” the laws of physics.)
As another analogy: would you say that the quicksort algorithm “knows” how to sort lists? I wouldn’t, because you can instead just say that the quicksort algorithm sorts lists, which conveys more information (because it avoids anthropomorphic implications). Similarly, the program you describe builds networks that are good at Go, and does so by making use of the rules of Go, but can’t do the sort of additional processing with respect to those rules which would make me want to talk about its knowledge of Go.
I don’t think there is a fundamental difference in kind between trees, bacteria, humans, and hypothetical future AIs
There’s at least one important difference: some of these are intelligent, and some of these aren’t.
It does seem plausible that the category boundary you’re describing is an interesting one. But when you indicate in your comment below that you see the “AI hypothesis” and the “life hypothesis” as very similar, then that mainly seems to indicate that you’re using a highly nonstandard definition of AI, which I expect will lead to confusion.
It feels like this post pulls a sleight of hand. You suggest that it’s hard to solve the control problem because of the randomness of the starting conditions. But this is exactly the reason why it’s also difficult to construct an AI with a stable implementation. If you can do the latter, then you can probably also create a much simpler system which creates the smiley face.
Similarly, in the real world, there’s a lot of randomness which makes it hard to carry out tasks. But there are a huge number of strategies for achieving things in the world which don’t require instantiating an intelligent controller. For example, trees and bacteria started out small but have now radically reshaped the earth. Do they count as having “perception, cognition, and action that are recognizably AI-like”?
The human knows the rules and the win condition. The optimisation algorithm doesn’t, for the same reason that evolution doesn’t “know” what dying is: neither are the types of entities to which you should ascribe knowledge.
it’s not obvious to me that this is a realistic target
Perhaps I should instead have said: it’d be good to explain to people why this might be a useful/realistic target. Because if you need propositions that cover all the instincts, then it seems like you’re basically asking for people to revive GOFAI.
(I’m being unusually critical of your post because it seems that a number of safety research agendas lately have become very reliant on highly optimistic expectations about progress on interpretability, so I want to make sure that people are forced to defend that assumption rather than starting an information cascade.)
As an additional reason for the importance of tabooing “know”, note that I disagree with all three of your claims about what the model “knows” in this comment and its parent.
(The definition of “know” I’m using is something like “knowing X means possessing a mental model which corresponds fairly well to reality, from which X can be fairly easily extracted”.)
I think at this point you’ve pushed the word “know” to a point where it’s not very well-defined; I’d encourage you to try to restate the original post while tabooing that word.
This seems particularly valuable because there are some versions of “know” for which the goal of knowing everything a complex model knows seems wildly unmanageable (for example, trying to convert a human athlete’s ingrained instincts into a set of propositions). So before people start trying to do what you suggested, it’d be good to explain why it’s actually a realistic target.
I used to define “agent” as “both a searcher and a controller”
Oh, I really like this definition. Even if it’s too restrictive, it seems like it gets at something important.
I’m not sure what you meant by “more compressed”.
Sorry, that was quite opaque. I guess what I mean is that evolution is an optimiser but isn’t an agent, and in part this has to do with how it’s a very distributed process with no clear boundary around it. Whereas when you have the same problem being solved in a single human brain, then that compression makes it easier to point to the human as being an agent separate from its environment.
The rest of this comment is me thinking out loud in a somewhat incoherent way; no pressure to read/respond.
It seems like calling something a “searcher” describes only a very simple interface: at the end of the search, there needs to be some representation of the output which it has found. But that output may be very complex.
Whereas calling something a “controller” describes a much more complex interface between it and its environment: you need to be able to point not just to outcomes, but also to observations and actions. But each of those actions is usually fairly simple for a pure controller; if it’s complex, then you need search to find which action to take at each step.
Now, it seems useful to sometimes call evolution a controller. For example, suppose you’re trying to wipe out a virus, but it keeps mutating. Then there’s a straightforward sense in which evolution is “steering” the world towards states where the virus still exists, in the short term. You could also say that it’s steering the world towards states where all organisms have high fitness in the long term, but organisms are so complex that it’s easier to treat them as selected outcomes, and abstract away from the many “actions” by evolution which led to this point.
In other words, evolution searches using a process of iterative control. Whereas humans control using a process of iterative search.
(As a side note, I’m now thinking that “search” isn’t quite the right word, because there are other ways to do selection than search. For example, if I construct a mathematical proof (or a poem) by writing it one line at a time, letting my intuition guide me, then it doesn’t really seem accurate to say that I’m searching over the space of proofs/poems. Similarly, a chain of reasoning may not branch much, but still end up finding a highly specific conclusion. Yet “selection” also doesn’t really seem like the right word either, because it’s at odds with normal usage, which involves choosing from a preexisting set of options—e.g. you wouldn’t say that a poet is “selecting” a poem. How about “design” as an alternative? Which allows us to be agnostic about how the design occurred—whether it be via a control process like evolution, or a process of search, or a process of reasoning.)
My default would be “raahp”, which doesn’t have any of the problems you mentioned.
I’m wondering if the disagreement over the centrality of this example is downstream from a disagreement about how easy the “alignment check-ins” that Critch talks about are. If they are the sort of thing that can be done successfully in a couple of days by a single team of humans, then I share Critch’s intuition that the system in question starts off only slightly misaligned. By contrast, if they require a significant proportion of the human time and effort that was put into originally training the system, then I am much more sympathetic to the idea that what’s being described is a central example of misalignment.
My (unsubstantiated) guess is that Paul pictures alignment check-ins becoming much harder (i.e. closer to the latter case mentioned above) as capabilities increase? Whereas maybe Critch thinks that they remain fairly easy in terms of number of humans and time taken, but that over time even this becomes economically uncompetitive.