Great introduction. As someone with a background in computational neuroscience, I’m really looking forward to what you have to say on all this.
By the way, you seem to use a very broad umbrella for covering “brain-like-AGI” approaches. Would you classify something like a predictive coding network as more brain-like or more prosaic? What about a neural Turing machine? In other words, do you have a distinct boundary in mind for separating the red box from the blue box, or is your classification more fuzzy (or does it not matter)?
If I’m not mistaken, the things you brought up are at too low a level to be highly relevant for safety, in my opinion. I guess this series will mostly be at Marr’s “computational level” whereas you’re asking “algorithm-level” questions, more or less. I’ll be talking a lot about things vaguely like “what is the loss function” and much less about things vaguely like “how is the loss function minimized”.
For example, I think you can train a neural Turing machine with supervised learning, or you can train a neural Turing machine with RL. That distinction would be very relevant for safety, along with the question of exactly how these supervisory signals (error signals in SL / rewards in RL) are being calculated. By contrast, the question of whether it’s a neural Turing machine versus a predictive coding network or whatever is less relevant for safety, in my opinion. It’s not totally irrelevant, and those kinds of things will come up from time to time in future posts, but it’s mostly irrelevant.
Y’know, there’s a thing that I call “The gory details of the neocortical algorithm”. This thing is the very last thing that I hope we figure out about the brain, as I think those details are extremely helpful for building AGI but only slightly helpful for AGI safety. Needless to say, “the “gory details of the neocortical algorithm” are the exact thing that practically everyone in computational neuroscience and ML spends all their time trying to figure out! :-P I’m in no position to slow that tidal wave, but at least I’ll set a good example by talking about it approximately as little as I can get away with, while meanwhile enthusiastically promoting work on more safety-helping topics within comp neuro. I haven’t always done that in the past, but better late than never. :-)
Hey Steve, I might be wrong here but I don’t think Jon’s question was specifically about what architectures you’d be talking about. I think he was asking more specifically about how to classify something as Brain-like-AGI for the purposes of your upcoming series.
The way I read your answer makes it sound like the safety considerations you’ll be discussing depend more on whether the NTM is trained via SL or RL rather than whether it neatly contains all your (soon to be elucidated) Brain-like-AGI properties.
Though that might actually have been what you meant so I probably should have asked for clarification before I presumptively answered Jon for you.
the safety considerations you’ll be discussing depend more on whether the NTM is trained via SL or RL rather than whether it neatly contains all your (soon to be elucidated) Brain-like-AGI properties.
I’m confused; this statement makes it sound like “whether it’s trained via SL or RL” is NOT a possible candidate for a “brain-like-AGI property”. Why can’t it be? Or maybe I’m reading too much into your wording.
Oops, strangely enough I just wasn’t thinking about that possibility. It’s obvious now, but I assumed that SL vs RL would be a minor consideration, despite the many words you’ve already written on reward.
If I’m reading your question right I think the answer is:
I’m going to make a bunch of claims about the algorithms underlying human intelligence, and then talk about safely using algorithms with those properties. If our future AGI algorithms have those properties, then this series will be useful, and I would be inclined to call such an algorithm “brain-like”.
i.e. The distinction depends on whether or not a given architecture has some properties Steve will mention later. Which, given Steve’s work, are probably the key properties of “A learned population of Compositional Generative Models + A largely hardcoded Steering Subsystem”.
OK that’s fair, I didn’t really answer Jon’s question.
So: What makes an AI algorithm brain-like for present purposes? Probably the biggest single thing is whether it’s an actor-critic model-based RL algorithm. If yes, there’s a good chance that some of the things I talk about in this series will be at least somewhat applicable to it. If no, probably not.
But I’m not too sure and wouldn’t want to make any promises. “Actor-critic model-based RL” is AFAIK a big diverse group of a thousand different models that work in a thousand different ways. Probably all of them have some kinds of safety-relevant differences from what I’m gonna talk about. But it’s hard for me to say anything in general.
Great introduction. As someone with a background in computational neuroscience, I’m really looking forward to what you have to say on all this.
By the way, you seem to use a very broad umbrella for covering “brain-like-AGI” approaches. Would you classify something like a predictive coding network as more brain-like or more prosaic? What about a neural Turing machine? In other words, do you have a distinct boundary in mind for separating the red box from the blue box, or is your classification more fuzzy (or does it not matter)?
Thanks!
If I’m not mistaken, the things you brought up are at too low a level to be highly relevant for safety, in my opinion. I guess this series will mostly be at Marr’s “computational level” whereas you’re asking “algorithm-level” questions, more or less. I’ll be talking a lot about things vaguely like “what is the loss function” and much less about things vaguely like “how is the loss function minimized”.
For example, I think you can train a neural Turing machine with supervised learning, or you can train a neural Turing machine with RL. That distinction would be very relevant for safety, along with the question of exactly how these supervisory signals (error signals in SL / rewards in RL) are being calculated. By contrast, the question of whether it’s a neural Turing machine versus a predictive coding network or whatever is less relevant for safety, in my opinion. It’s not totally irrelevant, and those kinds of things will come up from time to time in future posts, but it’s mostly irrelevant.
Y’know, there’s a thing that I call “The gory details of the neocortical algorithm”. This thing is the very last thing that I hope we figure out about the brain, as I think those details are extremely helpful for building AGI but only slightly helpful for AGI safety. Needless to say, “the “gory details of the neocortical algorithm” are the exact thing that practically everyone in computational neuroscience and ML spends all their time trying to figure out! :-P I’m in no position to slow that tidal wave, but at least I’ll set a good example by talking about it approximately as little as I can get away with, while meanwhile enthusiastically promoting work on more safety-helping topics within comp neuro. I haven’t always done that in the past, but better late than never. :-)
Hey Steve, I might be wrong here but I don’t think Jon’s question was specifically about what architectures you’d be talking about. I think he was asking more specifically about how to classify something as Brain-like-AGI for the purposes of your upcoming series.
The way I read your answer makes it sound like the safety considerations you’ll be discussing depend more on whether the NTM is trained via SL or RL rather than whether it neatly contains all your (soon to be elucidated) Brain-like-AGI properties.
Though that might actually have been what you meant so I probably should have asked for clarification before I presumptively answered Jon for you.
I’m confused; this statement makes it sound like “whether it’s trained via SL or RL” is NOT a possible candidate for a “brain-like-AGI property”. Why can’t it be? Or maybe I’m reading too much into your wording.
Oops, strangely enough I just wasn’t thinking about that possibility. It’s obvious now, but I assumed that SL vs RL would be a minor consideration, despite the many words you’ve already written on reward.
If I’m reading your question right I think the answer is:
i.e. The distinction depends on whether or not a given architecture has some properties Steve will mention later. Which, given Steve’s work, are probably the key properties of “A learned population of Compositional Generative Models + A largely hardcoded Steering Subsystem”.
OK that’s fair, I didn’t really answer Jon’s question.
So: What makes an AI algorithm brain-like for present purposes? Probably the biggest single thing is whether it’s an actor-critic model-based RL algorithm. If yes, there’s a good chance that some of the things I talk about in this series will be at least somewhat applicable to it. If no, probably not.
But I’m not too sure and wouldn’t want to make any promises. “Actor-critic model-based RL” is AFAIK a big diverse group of a thousand different models that work in a thousand different ways. Probably all of them have some kinds of safety-relevant differences from what I’m gonna talk about. But it’s hard for me to say anything in general.