Working on AGI safety via a deep-dive into brain algorithms, see https://sjbyrnes.com/agi.html
Steven Byrnes
Karma: 3,573
There’s a nice brain-like vision model here, and it even parses optical illusions in the same way people do. As far as I understand it, if there’s a sudden change of, um, color, or whatever it is for migraine aura, it has to be (1) an edge of a thing, (2) an edge of an occluding thing, (3) a change of texture within a single surface (e.g. wallpaper). When you block a head with your hand, your visual system obviously and correctly parses it as (2). But here there’s no occluder model that fits all the visual input data—maybe because some of the neurons that would offer evidence of an occluding shape are messed up and not sending those signals. So (2) doesn’t fit the data. And there’s no single-surface theory that fits all the visual input data either, so (3) gets thrown out too. So eventually the visual system settles on (1) as the best (least bad) parsing of the scene.
I dunno, something like that, I guess.
I slightly edited that section header to make it clearer what the parenthetical “(matrix multiplications, ReLUs, etc.)” is referring to. Thanks!
I agree that it’s hard to make highly-confident categorical statements about all current and future DNN-ish architectures.
I don’t think the human planning algorithm is very much like MCTS, although you can learn to do MCTS (just like you can learn to mentally run any other algorithm—people can learn strategies about what thoughts to think, just like they can strategies about what actions to execute). I think the built-in capability is that compositional-generative-model-based processing I was talking about in this post.
Like, if I tell you “I have a banana blanket”, you have a constraint (namely, I just said that I have a banana blanket) and you spend a couple seconds searching through generative models until you find one that is maximally consistent with both that constraint and also all your prior beliefs about the world. You’re probably imagining me with a blanket that has pictures of bananas on it, or less likely with a blanket made of banana peels, or maybe you figure I’m just being silly.
So by the same token, imagine you want to squeeze a book into a mostly-full bag. You have a constraint (the book winds up in the bag), and you spend a couple seconds searching through generative models until you find one that’s maximally consistent with both that constraint and also all your prior beliefs and demands about the world. You imagine a plausible way to slide the book in without ripping the bag or squishing the other content, and flesh that out into a very specific action plan, and then you pick the book up and do it.
When we need a multi-step plan, too much to search for in one go, we start needing to also rely on other built-in capabilities like chunking stuff together into single units, analogical reasoning (which is really just a special case of compositional-generative-model-based processing), and RL (as mentioned above, RL plays a role in learning to use metacognitive problem-solving strategies). Maybe other things too.
I don’t think causality per se is a built-in feature, but I think it comes out pretty quickly from the innate ability to learn (and chunk) time-sequences, and then incorporate those learned sequences into the compositional-generative-model-based processing framework. Like, “I swing my foot and then kick the ball and then the ball is flying away” is a memorized temporal sequence, but it’s also awfully close to a causal belief that “kicking the ball causes it to fly away”. (...at least in conjunction with a second memorized temporal sequence where I don’t kick the ball and it just stays put.) (See also counterfactuals.)
I’m less confident about any of this than I sound :)
Oh OK I think I misunderstood you.
So the context was: I think there’s an open question about the extent to which the algorithms underlying human intelligence in particular, and/or AGI more generally, can be built from operations similar to matrix multiplication (and a couple other operations). I’m kinda saying “no, it probably can’t” while the scaling-is-all-you-need DNN enthusiasts are kinda saying “yes, it probably can”.
Then your response is that humans can’t multiply matrices in their heads. Correct? But I don’t think that’s relevant to this question. Like, we don’t have low-level access to our own brains. If you ask GPT-3 (through its API) to simulate a self-attention layer, it wouldn’t do particularly well, right? So I don’t think it’s any evidence either way.
“Surpassed” seems strange to me; I’ll bet that the first AGI system will have a very GPT-like module, that will be critical to its performance, that will nevertheless not be “the whole story.” Like, by analogy to AlphaGo, the interesting thing was the structure they built around the convnets, but I don’t think it would have worked nearly as well without the convnets.
I dunno, certainly that’s possible, but also sometimes new algorithms outright replace old algorithms. Like GPT-3 doesn’t have any LSTM modules in it, let alone HHMM modules, or syntax tree modules, or GOFAI production rule modules. :-P
I really like the report, although maybe I’m not a neutral judge, since I was already inclined to agree with pretty much everything you wrote. :-P
My own little AGI doom scenario is very much in the same mold, just more specific on the technical side. And much less careful and thorough all around. :)
For benefits of generality (4.3.2.1), an argument I find compelling is that if you’re trying to invent a new invention or design a new system, you need a cross-domain system-level understanding of what you’re trying to do and how. Like at my last job, it was not at all unusual for me to find myself sketching out the algorithms on a project and sketching out the link budget and scrutinizing laser spec sheets and scrutinizing FPGA spec sheets and nailing down end-user requirements, etc. etc. Not because I’m individually the best person at each of those tasks—or even very good!—but because sometimes a laser-related problem is best solved by switching to a different algorithm, or an FPGA-related problem is best solved by recognizing that the real end-user requirements are not quite what we thought, etc. etc. And that kind of design work is awfully hard unless a giant heap of relevant information and knowledge is all together in a single brain / world-model.
Yeah thanks!
I guess was thinking that kids who don’t get bad cases at the time are unlikely to have long-term effects. I think polio is like that. In particular, I assume that only the bad COVID cases get into the nervous system, where I’m especially concerned. So that’s how I got a lower number. But I dunno either :-)
Sure but that would make OP’s point weaker not stronger, right?
my version is here (bulleted list at the end) :-)
My kids really don’t mind wearing masks. They really just don’t care, they don’t even think about it. Sometimes we’ll get home and they’ll just forget to take their masks off! Like, for a really long time! They just got used to wearing masks when going out, pretty quickly into the pandemic. Young kids are adaptable. :)
I’m not really sure what your question is getting at. There’s no sense in directly comparing my need for a mask to my kids’ need for a mask. It’s not like we only own one mask and need to fight over it…
For what it’s worth, it wasn’t my decision, but I am very happy that everyone in their school has to wear masks indoors. The benefit of reducing in-school COVID spread seems to me to overwhelmingly outweigh the (trivial) costs of making kids and teachers wear masks. I think that the prevailing COVID rates in the community would need to be very low indeed—maybe 10× or 100× lower than today—before I would endorse having kids in school stop wearing masks, at least until there’s a vaccine available for kids.
Hmm, I think for me the dominant cost of masks is that they’re mildly annoying. That’s a much bigger cost for me than the monetary price or the time spent laundering them or whatever.
I endorse not wearing masks when they provide zero or infinitesimal benefit. Like, where I live, there’s a rule that people walking alone outside need to wear a mask. That’s a really dumb and annoying rule.
I expect to be doing more stuff without masks, and more stuff period, when I’m fully vaccinated, and so are my friends, and when the prevailing COVID rates in the community are much lower than they are now. Can’t wait, and I think it won’t be much longer, in my community anyway. :)
Thanks for the nice post! Here’s why I disagree :)
Technological deployment lag
Normal technologies require (1) people who know how to use the technology, and (2) people who decide to use the technology. If we’re thinking about a “real-deal AGI” that can do pretty much every aspect of a human job but better and cheaper, then (1) isn’t an issue because the AGI can jump into existing human roles. It would be less like “technology deployment” and more like a highly-educated exquisitely-skilled immigrant arriving into a labor market. Such a person would have no trouble getting a job, in any of a million different roles, in weeks not decades. For (2), the same “real-deal AGI” would be able to start companies of its own accord, build factories, market products and services, make money, invest it in starting more companies, etc. etc. So it doesn’t need anyone to “decide to use the technology” or to invest in the technology.
Regulation will slow things down
I think my main disagreement comes from my thinking of AGI development as being “mostly writing and testing code inside R&D departments”, rather than “mostly deploying code to the public and learning from that experience”. I agree that it’s feasible and likely for the latter activity to get slowed down by regulation, but the former seems much harder to regulate for both political reasons and technical reasons.
The political is: It’s easy to get politicians riled up about the algorithms that Facebook is actually using to influence people, and much harder to get politicians riled up about whatever algorithms Facebook is tinkering with (but not actually deploying) in some office building somewhere. I think there would only be political will once we start getting “lab escape accidents” with out-of-control AGIs self-replicating around the internet, or whatever, at which point it may well be too late already.
The technical is: A lot of this development will involve things like open-source frameworks to easily parallelize software, and easier-to-use faster open-source implementations of new algorithms, academic groups publishing papers, and so on. I don’t see any precedent or feasible path for the regulation of these kinds of activities, even if there were the political will.
Not that we shouldn’t develop political and technical methods to regulate that kind of thing—it seems like worth trying to figure out—just that it seems extremely hard to do and unlikely to happen.
Overestimating the generality of AI technology
My own inside-view story (see here for example) is that human intelligence is based around a legible learning algorithm, and that researchers in neuroscience and AI are making good progress in working out exactly how that learning algorithm works, especially in the past 5 years. I’m not going to try to sell you on that story here, but fwiw it’s a short-ish timelines story that doesn’t directly rely on the belief that currently-popular deep learning models are very general, or even necessarily on the right track.
What are the complications? Death? Weeks in the hospital? Lifelong complications?
I suspect that if there was even 1-in-100,000 chance of that kind of consequence from regularly wearing masks, I would have heard about it by now. But if you have a reference to actual incidents (not just speculation that it’s possible, but actual people who had these kinds of very very serious problems), I’d be interested to see that.
I want to consider possible impacts of my decisions that are either (1) common, (2) rare but catastrophic. MIS-C is not super catastrophic, but it’s fatal if you’re not promptly hospitalized, and occasionally fatal even if you are, if I understand correctly. So it enters into consideration, despite being rare. And even so I wound up declaring that MIS-C risk is too low to be decision relevant. I have a hard time imagining that wearing a mask will lead to consequences anywhere remotely as serious as MIS-C. So it wouldn’t enter into my consideration unless it was very common, like >1%.
Sorry, I meant “negative consequences of mask-wearing are rare”, not “wearing moist and soiled masks is rare”. I’ve worn moist and soiled masks from time to time, and nothing bad has happened to me so far, except perhaps looking a bit unprofessional :-)
And I meant “rare compared to 100%”. Like, if even 1% of mask-wearers got a throat bacterial infection, that would be millions of throat bacterial infections in my country, 50,000 in my state, hundreds in my town, and probably at least one or two among my friends and family and acquaintances. So if that’s actually a thing that’s happening at a 1% rate, I think I would have heard something about it by now. Unless those infections were really not a big deal, such that they don’t rise to the level of even being worth mentioning to your friends.
(How many people do you personally know who have gotten a bacterial throat infection from mask wearing? How bad was it? Were they hospitalized? How many days of work did they miss?)
So I figure that bacterial throat infections from mask wearing is either <<1% likely to happen, or it’s really not a big deal when it does happen, or (most likely) both.
Those don’t seem to be worth worrying about, so far as I can tell. It seems like they are rare (or maybe nonexistent) (insofar as I haven’t heard of any such issues through friends or family or the news), and they also sound like not that big a deal even if they do happen. You can tell me if I’m missing something.
You’re welcome but I hope you’re not taking my word for anything. Note the warning at the top :-)
Thanks for the great questions!!
Why does it need to be its own separate module?
Maybe you’re ahead of me, but it took me until long after this post—just a couple weeks ago—to realize that you can take a neural circuit set up for RL, and jury-rig it to do supervised learning instead.
I think this is a big part of the story behind what the vmPFC is doing. And, in a certain sense, the amygdala too. More on this in a forthcoming post.
couldn’t the neocortex also be running lookup table like auto or hetero-associative learning … Why is the cerebellum faster at producing outputs than the neocortex?
I think of the neocortex as doing analysis-by-synthesis—it searches through a space of generative models for one that matches the input data. There’s a lot of recurrency—the signals bounce around until it settles into an equilibrium. For example in this model, there’s a forward pass from the input data, and that activates some generative models that seem plausible. But it may be multiple models that are mutually inconsistent. For example, in the vision system, “Yarn” and “Yam” are sufficiently close that a feedforward pass would activate both possibilities simultaneously. Then there’s this message-passing algorithm where the different possibilities compete to explain the data, and it settles on one particular compositional generative model.
So this seems like a generally pretty slow inference algorithm. But the slowness is worth it, because the neocortex winds up understanding the input data, i.e. fitting it into its structured model of the world, and hence it can now flexibly query it, make predictions, etc.
I think the cerebellum is much simpler than that, and closer to an actual lookup table, and hence presumably much faster.
The cerebellum is also closer in proximity to the spinal cord, which reduces communication delays when reading proprioceptive nerves and commanding muscles.
I have seen evidence this is the case but also that the context actually comes through the climbing fibers and training (shoulda) signal through the mossy/parallel fibers. Eg here for eyeblink operant conditioning https://www.cs.cmu.edu/afs/cs/academic/class/15883-f17/readings/hesslow-2013.pdf
That paper says “it has been the dominant working assumption in the field that the CS is transmitted to the cerebellar cortex via the mossy fibres (mf) and parallel fibres (pf) whereas information about the US is provided by climbing fibres (cf) originating in the inferior olive”, which is what I said (US=shoulda, CS=context). Where are you disagreeing here? Or do they contradict that later in the paper?
How does the neocortex carry the “shoulda” signal?
The neocortex is a fancy algorithm that understands the world and takes intelligent actions. It’s not perfect, but it’s pretty great! So whatever the neocortex does, that’s kinda a “ground truth” for the question of “What is the right thing to do right now?”—at least, it’s a ground truth from the perspective of the much stupider cerebellum. So my proposal is that whatever the neocortex is outputting, that’s a “shoulda” signal that the cerebellum wants to imitate.
This suggests that the neocortex can learn the cerebellar mapping and short-circuit to use it? Why does it need to go through the cerebellum to do this? Rather than via the motor cortex and efferent connections back to the muscles?
I’m not sure I understand your question. The neocortex outputs don’t need to go through the cerebellum. People can be born without a cerebellum entirely, they turn out OK. But since the cerebellum is like a super-fast memoizer / lookup table, I think the neocortex can work better by passing signals through the cerebellum.
Anyway, that was all just casual speculation, I don’t know how the motor cortex, midbrain, cerebellum, and outgoing nerves are wired together. (I’m very interested to learn, just haven’t gotten around to it.)
Hmm, this page suggests that there are both motor-pathways-through-the-cerebellum and motor-pathways-bypassing-the-cerebellum. But that’s not a reliable source—that page seems to be riddled with errors. So I dunno.
Young kids catching COVID: how much to worry?
I’d say “more than offset”. Increases chip makers’ economies of scale and justifies higher R&D outlays...
Why does this approach only need to be implemented in neo-cortex like AGIs?
Oh, I wasn’t saying that. I wouldn’t know either way, I haven’t thought about it. RL is a very big and heterogeneous field. I only know little bits and pieces of it. It’s a lot easier to make a specific proposal that applies to a specific architecture—the architecture that I happen to be familiar with—than to try to make a more general proposal. So that’s all I did.
What do you mean by “factored series of value functions”? If you’re thinking of my other post, maybe that’s possible, although not what I had in mind, because humans can feel conflicted, but my other post is talking about a mechanism that does not exist in the human brain.
Confusion here is the agent updating a belief while conflict is the agent deciding to take an action?
Yeah, that’s what I was going for.
I think of human mental model updates as being overwhelmingly “adding more things” rather than “editing existing things”. Like you see a funny video of a fish flopping around, and then a few days later you say “hey, look at the cat, she’s flopping around just like that fish video”. I’m not sure I’m disagreeing with you here, but your language kinda implies rare dramatic changes, I guess like someone changing religion and having an ontological crisis. That’s certainly an important case but much less common.