I’m Steve Byrnes, a professional physicist in the Boston area. I have a summary of my AGI safety research interests at: https://sjbyrnes.com/agi.html
steve2152
Karma: 1,715
FYI, I now have a whole post elaborating on “inner alignment”: mesa-optimizers vs steered optimizers
Mesa-Optimizers vs “Steered Optimizers”
Thanks! Yes, I am also definitely worried about that.
I 100% agree that the default result, in the absence of careful effort, would be value lock-in at some point in time, when the neocortex part grows clever enough to undermine the subcortex part, and then you better hope that the locked-in values are what you want!
On the optimistic side:
-
There’s no law that says the subcortex part has to be super dumb and simple; we can have less-powerful AIs steering more powerful AIs, helped by intrusive interpretability tools, running faster and in multiple instances, etc. (as has been discussed in other contexts of course);
-
We can try to instill a motivation system from the start that doesn’t want to undermine the subcortex part—in particular, corrigible motivation. This is basically reliant on the “corrigibility is a broad basin of attraction” argument being correct, I think.
On the pessimistic side, I’m not at all confident that either of those things would work. For (2) in particular, I remain concerned about ontological crises (or other types of goal instability upon learning and reflection) undermining corrigibility after an indeterminate amount of time. (This remains my go-to example of a possibly-unsolvable safety problem, or at least I have no idea how to solve it.)
So yeah, maybe we would be doomed in this scenario (or at least doomed to roll the dice). Or maybe we just need to keep working on it :-)
-
Great list!!
On the neuroscience side, I’ve been trying to dive into “if we build AGI using similar algorithms as the human brain, how can we make it safe and beneficial?” Further reading. That’s more studying algorithms than “studying humans”, probably.
I guess these are all more on the strategy side, but...
-
Out of possible futures in which we’ve invented cheap superintelligent AGIs, do a survey of which one most people on earth would actually want to live in. How does it interact with different personalities and different value systems? Further reading.
-
If everyone on Earth had a superintelligent AGI helper, what would they do with it, and what would be the societal consequences? What if each person can buy an AGI whose capabilities are proportional to the amount of money they spend on its hardware?
-
How can we avoid the failure mode (assuming it is in fact a failure mode) where we solve the technical problem of making AGIs that are docile and subservient, but then there’s a political movement of people identifying with those AGIs and lobbying to make them more selfish and independent, presumably citing the analogy of slavery? What sort of AGI, and AGI-human interaction framework, would make this more or less likely to happen? Further reading.
-
Hmm, maybe we’re talking past each other. Let’s say I have something like AlphaZero, where 50,000 bytes of machine code trains an AlphaZero-type chess-playing agent, whose core is a 1-billion-parameter ConvNet. The ConvNet takes 1 billion bytes to specify. Meanwhile, the reward-calculator p, which calculates whether checkmate has occurred, is 100 bytes of machine code.
Would you say that the complexity of the trained chess-playing agent is 100 bytes or 50,000 bytes or 1 billion bytes?
I guess you’re going to say 50,000, because you’re imagining a Turing machine that spends a year doing the self-play to calculate the billion-parameter ConvNet and then immediately the same Turning machine starts running that ConvNet it just calculated. From the perspective of Kolmogorov complexity, it doesn’t matter that it spends a year calculating the ConvNet, as long as it does so eventually.
By the same token, you can always turn a search-y agent into an equivalent discriminitive-y agent, given infinite processing time and storage, by training the latter on a googol queries of the former. If you’re thinking about Kolmogorov complexity, then you don’t care about a googol queries, as long as it works eventually.
Therefore, my first comment is not really relevant to what you’re thinking about. Sorry. I was not thinking about algorithms-that-write-arbitrary-code-and-then-immediately-run-it, I was thinking about the complexity of the algorithms that are actually in operation as the agent acts in the world.
If my hand touches fire and thus immediately moves backwards by reflex, would this be an example of a discriminative policy, because an input signal directly causes an action without being processed in the brain?
Yes. But the lack-of-processing-in-the-brain is not the important part. A typical ConvNet image classifier does involve many steps of processing, but is still discriminative, not search-y, because it does not work by trying out different generative models and picks the one that best explains the data. You can build a search-y image classifier that does exactly that, but most people these days don’t.
If we’re talking about algorithmic complexity, there’s a really important distinction, I think. In the space of actions, we have:
SEARCH: Search over an action space for an action that best achieves a goal
DISCRIMINATIVE: There is a function that goes directly from sensory inputs to the appropriate action (possibly with a recurrent state etc.)
Likewise, in the space of passive observations (e.g. classifying images), we have:
SEARCH: Search over a space of generative models for the model that best reproduces the observations (a.k.a. “analysis by synthesis”)
DISCRIMINATIVE: There is a function that goes directly from sensory inputs to understanding / classification.
The search methods are generally:
Algorithmically simpler
More sample-efficient
Slower at run-time
(Incidentally, I think the neocortex does the “search” option in both these cases (and that they aren’t really two separate cases in the brain). Other parts of the brain do the “discriminative” option in certain cases.)
I’m a bit confused about how my comment here relates to your post. If p is the goal (“win at chess”), the simplest search-based agent is just about exactly as complicated as p (“do a minimax search to win at chess”). But RL(p), at least with the usual definition of “RL”, will learn a very complicated computation that contains lots of information about which particular configurations of pieces are advantageous or not, e.g. the ResNet at the core of AlphaZero.
Are you imagining that the policy π is searching or discriminative? If the latter, why are you saying that π is just as simple as p? (Or are you saying that?) “Win at chess” seems a lot simpler than “do the calculations described by the following million-parameter ResNet”, right?
Gary Marcus vs Cortical Uniformity
Sorta related: my comment here
On the Russell / Pinker debate, I thought Pinker had an interesting rhetorical sleight-of-hand that I hadn’t heard before...
When people on the “AGI safety is important” side explain their position, there’s kinda a pedagogical dialog:
A: Superintelligent AGI will be awesome, what could go wrong? B: Well it could outclass all of humanity and steer the future in a bad direction. A: OK then we won’t give it an aggressive goal. B: Even with an innocuous-sounding goal like “maximize paperclips” it would still kill everyone… A: OK, then we’ll give it a good goal like “maximize human happiness”. B: Then it would forcibly drug everyone. A: OK, then we’ll give it a more complicated goal like … B: That one doesn’t work either because …
...And then Pinker reads this back-and-forth dialog, removes a couple pieces of it from their context, and says “The existential risk scenario that people are concerned about is the paperclip scenario and/or the drugging scenario! They really think those exact things are going to happen!” Then that’s the strawman that he can easily rebut.
Pinker had other bad arguments too, I just thought that was a particularly sneaky one.
Well, sure, you could take bigger gradient-descent steps for some errors than others. I’m not aware of people doing that, but again, I haven’t checked. I don’t know how well that would work (if at all).
The thing you’re talking about here sounds to me like “a means to an end” rather than “an end in itself”, right? If writing “Karma 100000: …” creates the high-karma-ish answer we wanted, does it matter that we didn’t use rewards to get there? I mean, if you want algorithmic differences between Transformers and brains, there are loads of them, I could go on and on! To me, the interesting question raised by this post is: to what extent can they do similar things, even if they’re doing it in very different ways? :-)
Meanwhile, Jeff Hawkins says “Every part of the neocortex is running the same algorithm”, and it’s looking like maybe brains aren’t doing that complicated a set of things.
This is nitpicking, but your post goes back and forth between the “underlying algorithm” level and the “learned model” level. Jeff Hawkins is talking about the underlying algorithm level when he says that it is (more or less) the same in every part of the neocortex. But almost all the things you mention in “My algorithm as I understand it” are habits of thought that you’ve learned over the years. (By the same token, we should distinguish between “Transformer + SGD” and “whatever calculations are being done by the particular weight settings in the trained Transformer model”.)
I don’t expect there to be much simplicity or universality at the “learned model” level … I expect that people use lots of different habits of thought.
Has anyone done anything like “Train a neural net on Reddit, where it’s somehow separately rewarded for predicting the next word, and also for predicting how much karma a cluster of words will get, and somehow propagating that back into the language generation?”)
I imagine the easiest thing would be to pre-pend the karma to each post, fine-tune the model, then you can generate high-karma posts by just prompting with “Karma 1000: …”. I’m not aware of anyone having done this specific thing but didn’t check. I vaguely recall something like that for AlphaStar, where they started by imitation learning with the player’s skill flagged, and then could adjust the flag to make their system play better or worse.
What’s happening in System 2 thought?
If you haven’t already, see Kaj’s Against System 1 and System 2. I agree with everything he wrote; the way I would describe it is: Our brains house a zoo of compositional generative models, and system 2 is a cool thing where generative models can self-assemble into an ad-hoc crappy serial computer. For example, you can learn a Generative Model X that first summons a different Generative Model Y, and then summons either Generative Model Z₁ or Z₂ conditional on some feature of Generative Model Y. (Something like that … I guess I should write this up better someday.) Anyway, this is a pretty neat trick. Can a trained Transformer NN do anything like that? I think there’s some vague sense in which a 6-layer Transformer can do similar things as a series of 6 serial human thoughts maybe?? I don’t know. There’s definitely a ton of differences too.
...chunking...
My vague sense about foresight (rolling out multiple steps before deciding what to do) is that it’s helpful for sample-efficiency but not required in the limit of infinite training data. Some examples: in RL, both TD learning and tree search eventually converge to the same optimal answer; AlphaGo without a tree search is good but not as good as AlphaGo with a tree search.
Perhaps not coincidentally, language models are pretty sample inefficient compared to people...
In my everyday life, I feel like my thoughts very often involve a sequence of two or three chunks, like “I will reach into my bag and then pull out my wallet”, and somewhat less often is it a longer sequence than that, but i dunno.
Maybe “AlphaStar can’t properly block or not block narrow passages using buildings” is an example where it’s held back by lack of foresight.
You’re asking about pure predictive (a.k.a. self-supervised) learning. As far as I know, it’s an open question what the safety issues are for that (if any), even in a very concrete case like “this particular Transformer architecture trained on this particular dataset using SGD”. I spent a bit of time last summer thinking about it, but didn’t get very far. See my post self-supervised learning and manipulative predictions for one particular possible failure mode that I wasn’t able to either confirm or rule out. (I should go back to it at some point.) See also my post self-supervised learning and AGI safety for everything else I know on the topic. And of course I must mention Abram’s delightful Parable of Predict-o-matic if you haven’t already seen it; again, this is high-level speculation that might or might not apply to any particular concrete system (“this particular Transformer architecture trained by SGD”). Lots of open questions!
An additional set of potential problems comes from your suggestion to put it in a robot body and actually execute the commands. Can it even walk? Of course it can figure out walking by letting it try with a reward signal, but now we’re not talking about pure predictive learning anymore. Hmm, after thinking about it, I guess I’m cautiously optimistic that, in the limit of infinite training data from infinitely many robots learning to walk, a large enough Transformer doing predictive learning could learn to read its own sense data and walk without any reward signal. But then how do you get it to do useful things? My suggestion here was to put a metadata flag into inputs where a robot is being super-helpful, and then when you have the robot start acting in the real world, turn that flag on. Now we’re bringing in supervised learning, I guess.
In the event that the robot was actually capable of doing anything at all, I would be very concerned that you press go and then the system wanders farther and farther out of distribution and does weird, dangerous things that have a high impact on the world.
As for concrete advice for the GPT-7 team: I would suggest at least throwing out the robot body and making a text / image prediction system in a box, and then put a human in the loop looking at the screen before going out and doing stuff. This can still be very powerful and economically useful, and it’s a step in the right direction: it eliminates the problem of the system just going off and doing something weird and high-impact in the world because it wandered out of distribution. It doesn’t eliminate all problems, because the system might still become manipulative. As I mentioned in the 1st paragraph, I don’t know whether that’s a real problem or not, more research is needed. It’s possible that we’re all just doomed in your scenario. :-)
Ooh, sounds interesting, thanks for the tip!
Hofstadter also has a thing maybe like that when he talks about “analogical reasoning”.
Thanks!
I’m curious what you think is going on here that seems relevant to inner alignment.
Hmm, I guess I didn’t go into detail on that. Here’s what I’m thinking.
For starters, what is inner alignment anyway? Maybe I’m abusing the term, but I think of two somewhat different scenarios.
-
In a general RL setting, one might say that outer alignment is alignment between what we want and the reward function, and inner alignment is alignment between the reward function and “what the system is trying to do”. (This one is closest to how I was implicitly using the term in this post.)
-
In the “risks from learned optimization” paper, it’s a bit different: the whole system (perhaps an RL agent and its reward function, or perhaps something else entirely) is conceptually bundled together into a single entity, and you do a black-box search for the most effective “entity”. In this case, outer alignment is alignment between what we want and the search criterion, and inner alignment is alignment between the search criterion and “what the system is trying to do”. (This is not really what I had in mind in this post, although it’s possible that this sort of inner alignment could also come up, if we design the system by doing an outer search, analogous to evolution.)
Note that neither of these kinds of “inner alignment” really comes up in existing mainstream ML systems. In the former (RL) case, if you think of an RL agent like AlphaStar, I’d say there isn’t a coherent notion of “what the system is trying to do”, at least in the sense that AlphaStar does not do foresighted planning towards a goal. Or take AlphaGo, which does have foresighted planning because of the tree search; but here we program the tree search by hand ourselves, so there’s no risk that the foresighted planning is working towards any goal except the one that we coded ourselves, I think.
So, “RL systems that do foresighted planning towards explicit goals which it invents itself” are not much of a thing these days (as far as I know), but they presumably will be a thing in the future (among other things, this is essential for flexibly breaking down goals into sub-goals). And the neocortex is in this category. So yeah, it seems reasonable to me to extend the term “inner alignment” to this case too.
So anyway, the neocortex creates explicit goals for itself, like “I want to get out of debt”, and uses foresight / planning to try to bring them about. (Of course it creates multiple contradictory goals, and also has plenty of non-goal-seeking behaviors, but foresighted goal-seeking is one of the things people sometimes do! And of course transient goals can turn into all-consuming goals in self-modifying AGIs) The neocortical goals have something to do with subcortical reward signals, but it’s obviously not a deterministic process, and therefore there’s an opportunity for inner alignment problems.
...noting there doesn’t seem to be much divergence between their objective functions...
This is getting into a different question I think… OK, so, If we build an AGI along the lines of a neocortex-like system plus a subcortex-like system that provides reward signals and other guidance, will it reliably do the things we designed it to do? My default is usually pessimism, but I guess I shouldn’t go too far. I think some of the things that this system is designed to do, it seems to do very reliably. Like, almost everyone learns language. This requires, I believe, a cooperation between the neocortex and a subcortical system that flags human speech sounds as important. And it works almost every time! A more important question is, can we design the system such that the neocortex will wind up reliably seeking pre-specified goals? Here, I just don’t know. I don’t think humans and animals provide strong evidence either way, or at least it’s not obvious to me...
-
Sorta related is my appendix to this article.
I do think signal propagation time is probably a big contributor. I think activating a generative model in the GNW entails activating a particular set of interconnected neurons scattered around the GNW parts of the neocortex, which in turn requires those neurons to talk with each other. You can think of a probabilistic graphical model … you change the value of some node and then run the message-passing algorithm a bit, and the network settles into a new configuration. Something like that, I think...
The GNW seems like it can only broadcast “simple” or “small” things. … Something like a hypothesis in the PP paradigm seems like too big and complex a thing to be “sent” on the GNW.
My picture is: the GNW can broadcast everything that you have ever consciously thought about. This is an awfully big space. And it is a space of generative models (a.k.a. hypotheses).
The GNW is at the top of the (loose) hierarchy: the GNW sends predictions down to the lower-level regions of the neocortex, which in turn send prediction errors back to the GNW.
If, say, some particular upcoming input is expected to be important, then GNW can learn to build a generative model where the prediction of that input has a high confidence attached to it. That will bias the subsequent behavior of the system towards whatever models tend to be driven by the prediction error coming from that input. Thus we build serial computers out of our parallel minds.
Given the serial, discrete nature of the GNW, it follows that consciousness is fundamentally a discrete and choppy thing, not a smooth continuous stream.
Here’s my take. Think of the neocortex as having a zoo of generative models with methods for building them and sorting through them. The models are compositional—compatible models can snap together like legos. Thus I can imagine a rubber wine glass, because the rubber generative models bottom out in a bunch of predictions of boolean variables, the wine glass generative models bottom out in a bunch of predictions of different boolean variables (and/or consistent predictions of the same boolean variables), and therefore I can union the predictions of the two sets of models.
Your GNW has an active generative model built out of lots of component models. I would say that the “tennis-match-flow” case entails little sub-sub-components asynchronously updating themselves as new information comes in—the tennis ball was over there, and now it’s over here. By contrast the more typically “choppy” way of thinking involves frequently throwing out the whole manifold of generative models all at once, and activating a wholly new set of interlocking generative models. The latter (unlike the former) involves an attentional blink, because it takes some time for all the new neural codes to become active and synchronized, and in between you’re in an incoherent, unstable state with mutually-contradictory generative models fighting it out.
Perhaps the attentional blink literature is a bit complicated because, with practice or intention, you can build a single GNW generative model that predicts both of two sequential inputs.
Q.1: Is the global neuronal workspace a bottleneck for motor control?
I vote strong no, or else I’m misunderstanding what you’re talking about. Let’s say you’re standing up, and your body tips back microscopically, and you slightly tension your ankles to compensate and stay balanced. Are you proposing that this ankle-tension command has to go through the GNW? I’m quite confident that it doesn’t. Stuff like that doesn’t necessarily even reach the neocortex at all, let alone the GNW. In this post I mentioned the example of Ian Waterman, who could not connect motor control output signals to feedback signals except by routing through the GNW. He had to be consciously thinking about how his body was moving constantly; if he got distracted he would collapse.
Well, we ought to be able to either figure out how to use this kind of system safely, or prove it’s impossible. Either would be valuable. :-)
I don’t think it’s obviously impossible though. In particular, with the right motivation, it won’t be motivated to undermine the steering signals. And also, the subcortex can be a slightly-less powerful AI, assisted by intrusive interpretability tools, multiple copies running faster, etc.
Yeah, I struggle with that too. Maybe an alternative (or at least starting point) would be to try to solve the challenge of building a question-answering oracle that has no motivation to lie or manipulate or escape its box, etc. I think that is a goal I can fully understand, although maybe I just haven’t thought about it carefully enough to find the edge cases. :-)