phd student in comp neuroscience @ mpi brain research frankfurt. https://twitter.com/janhkirchner and https://universalprior.substack.com/
Jan
Karma: 977
Cool experiment! I could imagine that the tokenizer handicaps GPT’s performance here (reversing the characters leads to completely different tokens). With a character-level tokenizer GPT should/might be able to handle that task better!
Neuroscience and Natural Abstractions
Similarities in structure and function abound in biology; individual neurons that activate exclusively to particular oriented stimuli exist in animals from drosophila (Strother et al. 2017) via pigeons (Li et al. 2007) and turtles (Ammermueller et al. 1995) to macaques (De Valois et al. 1982). The universality of major functional response classes in biology suggests that the neural systems underlying information processing in biology might be highly stereotyped (Van Hooser, 2007, Scholl et al. 2013). In line with this hypothesis, a wide range of neural phenomena emerge as optimal solutions to their respective functional requirements (Poggio 1981, Wolf 2003, Todorov 2004, Gardner 2019). Intriguingly, recent studies on artificial neural networks that approach human-level performance reveal surprising similarity between emerging representations in both artificial and biological brains (Kriegeskorte 2015, Yamins et al. 2016, Zhuang et al. 2020).
Despite the commonalities across different animal species, there is also substantial variability (Van Hooser, 2007). One prominent example of a functional neural structure that is present in some, but absent in other, animals is the orientation pinwheel in the primary visual cortex (Meng et al. 2012), synaptic clustering with respect to orientation selectivity (Kirchner et al. 2021), or the distinct three-layered cortex in reptiles (Tosches et al. 2018). These examples demonstrate that while general organization principles might be universal, the details of how exactly and where in the brain the principles manifest is highly dependent on anatomical factors (Keil et al. 2012, Kirchner et al. 2021), genetic lineage (Tosches et al. 2018), and ecological factors (Roeth et al. 2021). Thus, the universality hypothesis as applied to biological systems does not imply perfect replication of a given feature across all instances of the system. Rather, it suggests that there are broad principles or abstractions that underlie the function of cognitive systems, which are conserved across different species and contexts.
Here’s an operationalization. Suppose someday we write computer code that can do the exact same useful computational things that the neocortex (etc.) does, for the exact same reason. My question is: Might that code look like a learning-from-scratch algorithm?
Hmm, I see. If this is the crux, then I’ll put all the remaining nitpicking at the end of my comment and just say: I think I’m on board with your argument. Yes, it seems conceivable to me that a learning-from-scratch program ends up in a (functionally) very similar state to the brain. The trajectory of how the program ends up there over training probably looks different (and might take a bit longer if it doesn’t use the shortcuts that the brain got from evolution), but I don’t think the stuff that evolution put in the cortex is strictly necessary.
A caveat: I’m not sure how much weight the similarity between the program and the brain can support before it breaks down. I’d strongly suspect that certain aspects of the cortex are not logically implied by the statistics of the environment, but rather represent idiosyncratic quirks that were adapted at some point during evolution. Those idiosyncratic quirks won’t be in the learning-from-scratch program. But perhaps (probably?) they are also not relevant in the big scheme of things.
I’m inclined to put different layer thicknesses (including agranularity) in the category of “non-uniform hyperparameters”.
Fair! Most people in computational neuroscience are also very happy to ignore those differences, and so far nothing terribly bad happened.
If you buy the “locally-random pattern separation” story (Section 2.5.4), that would make it impossible for evolution to initialize the adjustable parameters in a non-locally-random way.
You point out yourself that some areas (f.e. the motor cortex) are granular, so that argument doesn’t work there. But ignoring that, and conceding the cerebellum and the drosophila mushroom body to you (not my area of expertise), I’m pretty doubtful about postulating “locally-random pattern separation” in the cortex. I’m interpreting your thesis to cash out as “Given a handful of granule cells from layer 4, the connectivity with pyramidal neurons in layer 2⁄3 is (initially) effectively random, and therefore layer 2⁄3 neurons need to learn (from scratch) how to interpret the signal from layer 4″. Is that an okay summary?
Because then I think this fails at three points:
One characteristic feature of the cortex is the presence of cortical maps. They exist in basically all sensory and motor cortices, and they have a very regular structure that is present in animal species separated by as much as 64 million years of evolution. These maps imply that if you pick a handful of granule cells from layer 4 that are located nearby, their functional properties will be somewhat similar! Therefore, even if connectivity between L4 and L2/3 is locally random it doesn’t really matter since the input is somewhat similar in any case. Evolution could “use” that fact to pre-structure the circuit in L2/3.
Connectivity between L4 and L2/3 is not random. Projections from layer 4 are specific to different portions of the postsynaptic dendrite, and nearby synapses on mature and developing dendrites tend to share similar activation patterns. Perhaps you want to argue that this non-randomness only emerges through learning and the initial configuration is random? That’s a possibility, but …
… when you record activity from neurons in the cortex of an animal that had zero visual experience prior to the experiment (lid-suture), they are still orientation-selective! And so is the topographic arrangement of retinal inputs and the segregation of eye-specific inputs. At the point of eye-opening, the animals are already pretty much able to navigate their environment.
Obviously, there are still a lot of things that need to be refined and set up during later development, but defects in these early stages of network initialization are pretty bad (a lot of neurodevelopmental disorders manifest as “wiring defects” that start in early development).
I’m very confused by this. I have coded up a ConvNet with random initialization. It was computationally tractable; in fact, it ran on my laptop!
Okay, my claim there came out a lot stronger than I wanted and I concede a lot of what you say. Learning from scratch is probably not computationally intractable in the technical sense. I guess what I wanted to argue is that it appears practically infeasible to learn everything from scratch. (There is a lot of “everything” and not a lot of time to learn it. Any headstart might be strictly necessary and not just a nice-to-have).
(As a side point: your choice of a convnet as the example is interesting. People came up with convnets because fully-connected, randomly initialized networks were not great at image classification and we needed some inductive bias in the form of a locality constraint to learn in a reasonable time. That’s the point I wanted to make.)
I guess maybe what you’re claiming is: we can’t have all three of {learning from scratch, general intelligence, computational tractability}.
Interesting, I haven’t thought about it like this before. I do think it could be possible to have all three—but then it’s not the brain anymore. As far as I can tell, evolutionary pressures make complete learning from scratch infeasible.
I’m pretty confused here.
Yeah, the feeling’s mutual 😅 But the discussion is also very rewarding for me, thank you for engaging!
I am in favor of learning-from-scratch, and I am also in favor of specific designed inductive biases, and I don’t think those two things are in opposition to each other.
A couple of thoughts:
Yes, I agree that the inductive bias (/genetically hardcoded information) can live in different components: the learning rule, the network architecture, or the initialization of the weights. So learning-from-scratch is logically compatible with inductive biases—we can just put all the inductive bias into the learning rule and the architecture and none in the weights.
But from the architecture and the learning rule, the hardcoded info can enter into the weights very rapidly (f.e. first step of the learning rule: set all the weights to the values appropriate for an adult brain. Or, more realistically, a ConvNet architecture can be learned from a DNN by setting a lot of connections to zero). Therefore I don’t see what it could buy you to assume the weights to be free of inductive bias.
There might also be a case that in the actual biological brain the weights are not initialized randomly. See f.e. this work on clonally related neurons.
Something that is not appreciated a lot outside of neuroscience: “Learning” in the brain is as much a structural process as it is a “changing weights” process. This is particularly true throughout development but also into adulthood—activity-dependent learning rules do not only adjust the weights of connections, but they can also prune bad connections and add new connections. The brain simultaneously produces activity, which induces plasticity, which changes the circuit, which produces slightly different activity in turn.
The point is, this kind of explanation does not talk about subplates and synapses, it talks about principles of algorithms and computations.
That sounds a lot more like cognitive science than neuroscience! This is completely fine (I did my undergrad in CogSci), but it requires a different set of arguments from the ones you are providing in your post, I think. If you want to make a CogSci case for learning from scratch, then your argument has to be a lot more constructive (i.e. literally walk us through the steps of how your proposed system can learn all/a lot of what humans can learn). Either you take a look at what is there in the brain (subplate, synapses, …), describe how these things interact, and (correctly) infer that it’s sufficient to produce a mind (this is the neuroscience strategy); Or you propose an abstract system, demonstrate that it can do the same thing as the mind, and then demonstrate that the components of the abstract system can be identified with the biological brain (this is the CogSci strategy). I think you’re skipping step two of the CogSci strategy.
Whatever that explanation is, it’s a thing that we can turn into a design spec for our own algorithms, which, powered by the same engineering principles, will do the same computations, with the same results.
I’m on board with that. I anticipate that the design spec will contain (the equivalent of) a ton of hardcoded genetic stuff also for the “learning subsystem”/cortex. From a CogSci perspective, I’m willing to assume that this genetic stuff could be in the learning rule and the architecture, not in the initial weights. From a neuroscience perspective, I’m not convinced that’s the case.
is that true even if there haven’t been any retinal waves?
Blocking retinal waves messes up the cortex pretty substantially (same as if the animal were born without eyes). There is the beta-2 knockout mouse, which has retinal waves but with weaker spatiotemporal correlations. As a consequence beta-2 mice fail to track moving gratings and have disrupted receptive fields.
Theory #4 appears very natural to me, especially in the light of papers like Chen et al 2006 or Cuntz et al 2012. And another supporting intuition from developmental neuroscience is that development is a huge mess and that figuring out where to put a long-range connection is really involved. And there can be a bunch of circuit remodeling on a local scale, once you established a long-range connection, there is little hope of substantially rewiring it.
In case you want to dive deeper into this (and you don’t want to read all those papers), I’d be happy to chat more about this :)
Hi! :) Thanks for the comment! Yes, that’s on purpose, the idea is that a lot of the shorthand in molecular neuroscience are very hard to digest. So since the exact letters don’t matter I intentionally garbled them with a Glitch Text Generator. But perhaps that isn’t very clear without explanation, I’ll add something.
This word Ǫ̵͎͊
G̦̉̇O-GlcNAc”l͉͇̝̽͆̚i̷͔̓̏͌c̷̱̙̍̂͜k̷̠͍͌l̷̢̍͗̃n̷̖͇̏̆å̴̤c̵̲̼̫͑̎̆ f.e. a garbled version of O-GLicklnac, which in term is the phonetic version of “
Hi, thanks for the response! I apologize, the “Left as an exercise” line was mine, and written kind of tongue-in-cheek. The rough sketch of the proposition we had in the initial draft did not spell out sufficiently clearly what it was I want to demonstrate here and was also (as you point out correctly) wrong in the way it was stated. That wasted people’s time and I feel pretty bad about it. Mea culpa.
I think/hope the current version of the statement is more complete and less wrong. (Although I also wouldn’t be shocked if there are mistakes in there). Regarding your points:
The limit now shows up on both sides of the equation (as it should)! The dependence on on the RHS does actually kind of drop away at some point, but I’m not showing that here. I’d previously just sloppily substituted “chose as a large number” and then rewrite the proposition in the way indicated at the end of the Note for Proposition 2. That’s the way these large deviation principles are typically used.
Yeah, that should have been an rather than a . Sorry, sloppy.
True. Thinking more about it now, perhaps framing the proposition in terms of “bridges” was a confusing choice; if I revisit this post again (in a month or so 🤦♂️) I will work on cleaning that up.
Hi Jennifer!
Awesome, thank you for the thoughtful comment! The links are super interesting, reminds me of some of the research in empirical aesthetics I read forever ago.
On the topic of circular preferences: It turns out that the type of reward model I am training here handles non-transitive preferences in a “sensible” fashion. In particular, if you’re “non-circular on average” (i.e. you only make accidental “mistakes” in your rating) then the model averages that out. And if you consitently have a loopy utility function, then the reward model will map all the elements of the loop onto the same reward value.
Finally: Yes, totally, feel free to send me the guest ID either here of via DM!
This work by Michael Aird and Justin Shovelain might also be relevant: “Using vector fields to visualise preferences and make them consistent”
And I have a post where I demonstrate that reward modeling can extract utility functions from non-transitive preference orderings: “Inferring utility functions from locally non-transitive preferences”
(Extremely cool project ideas btw)
There’s an important caveat here:
The visual stimuli are presented 8 degrees over the visual field for 100ms followed by a 100ms grey mask as in a standard rapid serial visual presentation (RSVP) task.
I’d be willing to bet that if you give the macaque more than 100ms they’ll get it right—That’s at least how it is for humans!
(Not trying to shift the goalpost, it’s a cool result! Just pointing at the next step.)
Great point! And thanks for the references :)
I’ll change your background to Computational Cognitive Science in the table! (unless you object or think a different field is even more appropriate)
Thank you for the comment and the questions! :)
This is not clear from how we wrote the paper but we actually do the clustering in the full 768-dimensional space! If you look closely as the clustering plot you can see that the clusters are slightly overlapping—that would be impossible with k-means in 2D, since in that setting membership is determined by distance from the 2D centroid.
Oh true, I completely overlooked that! (if I keep collecting mistakes like this I’ll soon have enough for a “My mistakes” page)
Interesting, I added a note to the text highlighting this! I was not aware of that part of the story at all. That makes it more of a Moloch-example than a “mistaking adversarial for random”-example.
Hey Charlie!
Good comment, gave me a feeling “oh, ups, why didn’t I?” for a while. I think having the Elo-like algorithm as a baseline to compare to would have been a good thing to have in any case. But there is something that the NN can do that the Elo-like algorithm can’t; generalization. Every “new” element (or even an interpolation of older elements) will get the “initial score” (like 1500 in chess) in Elo, while the NN can exploit similarities between the new element and older elements.
Yes please, would be excited to see that!
I enjoyed the footnote a lot :) And the entire story, of course. Thanks for writing!
Thanks for letting me know! Should be fixed now.
That… makes a lot of sense. Yep, that’s probably the answer! Thank you :)
Hey Steve! Thanks for writing this, it was an interesting and useful read! After our discussion in the LW comments, I wanted to get a better understanding of your thinking and this sequence is doing the job. Now I feel I can better engage in a technical discussion.
I can sympathize well with your struggle in section 2.6. A lot of the “big picture” neuroscience is in the stage where it’s not even wrong. That being said, I don’t think you’ll find a lot of neuroscientists who nod along with your line of argument without raising objections here and there (neuroscientists love their trivia). They might be missing the point, but I think that still makes your theory (by definition) controversial. (I think the term “scientific consensus” should be used carefully and very selectively).
In that spirit, there are a few points that I could push back on:
Cortical uniformity (and by extension canonical microcircuits) are extremely useful concepts for thinking about the brain. But they are not literally 100% accurate. There are a lot of differences between different regions of the cortex, not only in thickness but also in the developmental process (here or here). I don’t think anyone except for Jeff Hawkin believes in literal cortical uniformity.
In section 2.5.4.1 you are being a bit dismissive of biologically-”realistic” implementations of backpropagation. I used to be pretty skeptical too, but some of the recent studies are beginning to make a lot of sense. This one (a collaboration of Deepmind and some of the established neuroscience bigshots) is really quite elegant and offers some great insights on how interneurons and dendritic branches might interact.
A more theoretical counter: If evolution could initialize certain parts of the cortex so that they are faster “up and running” why wouldn’t it? (Just so that we can better understand it? How nice!) From the perspective of evolution, it makes a lot of sense to initialize the cortex with an idea of what an oriented edge is because oriented edges have always been around since the inception of the eye.
Or, in terms of computation theory, learning from scratch is computationally intractable. Strong, informative priors over hypothesis space might just be necessary to learn anything worthwhile at all.
But perhaps I’m missing the point with that nitpicking. I think the broader conceptual question I have is: What does “randomly initialized” even mean in the brain? At what point is the brain initialized? When the neural tube forms? When interneurons begin to migrate to the cortex? When the first synapses are established? When the subplate is gone? When the pruning of excess synapses and the apoptosis of cells is over? When the animal/human is born? When all the senses begin to transmit input? After college graduation?
Perhaps this is the point that the “old-timer” also wanted to make. It doesn’t really make sense to separate the “initialization” from the “refinement”. They happen at the same time, and whether you put a certain thing into one category or the other is up to individual taste.
All of this being said, I’m very curious to read the next parts of this sequence! :) Perhaps my points don’t even affect your core argument about AI Safety.