Jon Garcia(Jonathan William Garcia)

Karma: 363

I have a PhD in Computational Neuroscience from UCSD (Bachelor’s was in Biomedical Engineering with Math and Computer Science minors). Ever since junior high, I’ve been trying to figure out how to engineer artificial minds, and I’ve been coding up artificial neural networks ever since I first learned to program. Obviously, all my early designs were almost completely wrong/unworkable/poorly defined, but I think my experiences did prime my brain with inductive biases that are well suited for working on AGI.

Although I now work as a data scientist in R&D at a large medical device company, I continue to spend my free time studying the latest developments in AI/ML/DL/RL and neuroscience and trying to come up with models for how to bring it all together into systems that could actually be implemented. Unfortnately, I don’t seem to have much time to develop my ideas into publishable models, but I would love to have the opportunity to share ideas with those who do.

Of course, I’m also very interested in AI Alignment (hence the account here). My ideas on that front mostly fall into the “learn (invertible) generative models of human needs/goals and hook those up to the AI’s own reward signal” camp. I think methods of achieving alignment that depend on restricting the AI’s intelligence or behavior are about as destined to failure in the long term as Prohibition or the War on Drugs in the USA. We need a better theory of what reward signals are for in general (probably something to do with maximizing (minimizing) the attainable (dis)utility with respect to the survival needs of a system) before we can hope to model human values usefully. This could even extend to modeling the “values” of the ecological/socioeconomic/political supersystems in which humans are embedded or of the biological subsystems that are embedded within humans, both of which would be crucial for creating a better future.

Jon Garcia 29 Sep 2022 16:07 UTC
1 point
0 ∶ 0
in reply to: Lee Sharkey’s comment on: Interpreting Neural Networks through the Polytope Lens
Here is a paper that addresses using activation functions that bound the so-called “open space”:
Improved Adversarial Robustness by Reducing Open Space Risk via Tent Activations
According to the paper:
We hypothesize that adversarial attacks exploit the open space risk of classic monotonic activation functions. This paper introduces the tent activation function with bounded open space risk and shows that tents make deep learning models more robust to adversarial attacks. We demonstrate on the MNIST dataset that a classifier with tents yields an average accuracy of 91.8% against six white-box adversarial attacks, which is more than 15 percentage points above the state of the art.
Basically, causing all unbounded polytopes to have a zero-affine-transformation at extreme values improves adversarial robustness.
By the way, although the tent activation function prevents monotonic growth in the direction perpendicular to the decision hyperplane, I haven’t heard of any activation function that prevents the neuron from being active when the input goes too far out of distribution in a direction parallel to the hyperplane. It might be interesting to explore that angle.

Jon Garcia 28 Sep 2022 4:58 UTC
7 points
1 ∶ 0
on: 7 traps that (we think) new alignment researchers often fall into
1. Also, coming up with your own ideas first can help you better understand what you find in the literature. I’ve found that students learn more readily when they come to a subject with questions already in mind, having tried to figure things out on their own and realized where they had gaps in their mental framework, rather than just receiving a firehose of new information with no context.
2. Perhaps try pursuing a number of proxy goals for short, pre-defined periods, while tracking whether each proxy goal is likely to be instrumental for reaching the terminal goal. Assessing the instrumentality of each proxy should be easier once you’ve started to get a sense of where each project can lead, and abandoning those that are clearly not going to be fruitful should be easier if you don’t plan on going all-in from the start.
3. Don’t be afraid to ask stupid questions. We often tend to refrain from asking questions that we predict would cause those more experienced to perceive us as idiots. Ignore those predictions. Even when the answer is obvious to everyone else, it will help the writer practice clarifying their ideas from a new perspective, which could even help them understand their own work better. And sometimes everyone else is just afraid to look like idiots, too.
4. Try steel-manning the best argument you can come up with against an authority’s position. Ideas that can withstand the harshest scrutiny are those worth keeping. Ideas that can be destroyed by the truth should be. Help the intellectual community filter the chaff from the wheat.
5. Good hypotheses always entail predictive models. If you can’t program it, you don’t really understand it.
6. I can’t think of anything else to add to this one.
7. Also, don’t wait until you’ve learned linear algebra, multivariable calculus, probability theory, and machine learning before starting to tackle the alignment problem. It’s easier to learn these things once you already know where they will be useful to you. Plus, we may not have enough time to wait on mathematicians to come up with provable guarantees of AI safety.

Jon Garcia 27 Sep 2022 14:52 UTC
0 points
0 ∶ 0
in reply to: latinostats’s comment on: ACHIEVABLE AGI?
My comment did not assume the physical. Entropy is not a physical concept but a statistical one. It can be used to measure the complexity of any system, physical or not.

Any real system, whether physical, spiritual, or anything else, will necessarily be computable. Even the mind of God would have to be technically computable, although you could argue that it would require more computational power than could possibly fit in the physical universe.

Again, there is no magic. Nothing is intrinsically mysterious. Mystery is a symptom of a finite, ignorant mind. Whatever realms might exist beyond this physical universe are necessarily logically coherent and explainable in principle. And that applies to any subsystem of any reality, whether consciousness, life, or anything else.

Jon Garcia 27 Sep 2022 5:27 UTC
2 points
1 ∶ 0
on: ACHIEVABLE AGI?
There is no evidence that the structure and function of the human brain (along with its physical embodiment) is insufficient to explain all aspects of human intelligence and consciousness. Even if it was discovered that the physical brain fell short of accounting for all cognitive phenomena, and even if no other known physical system or process could account for it, and even if an immaterial soul was actually discovered somehow, it still wouldn’t imply that human intelligence could not be emulated.

There is no magic. All real phenomena are causally computable. An immaterial soul would be no exception. Even a non-physical conscious system would necessarily possess inner computational mechanisms that are themselves not conscious (e.g., “spiritual synapses”).

The only cop-out would be if the immaterial soul was performing necessary computations so vast that they could not be performed on a physical computer the size of Earth within a human lifespan (therefore necessitating a non-physical computational substrate that is not bound by physical constraints on memory density and processing speed). There is no evidence of this.

You have too high a view of human-level intelligence and consciousness. We are in fact intellectual bottom-feeders in the vast ocean of possible mind-space. And that’s even accounting for physical limits on computational power.

Intelligibility is not evidence for intelligent design. It is evidence for a low-entropy system, which has a far higher prior probability than a supernatural Intelligentsia.

Jon Garcia 25 Sep 2022 13:38 UTC
4 points
0 ∶ 0
on: Interpreting Neural Networks through the Polytope Lens
This is a really useful way of looking at deep neural networks. An extension to non-ReLU activation functions would be interesting, although for most, I feel that the only difference would be to make the boundaries between polytopes smooth (e.g., with Swish or GELU) or to make it so affine transformations only occur near the boundaries (e.g., with sigmoid, tanh, or tent).

Notice that outside of the inner sphere of polytopes, there will inevitably be an infinite region of unbounded polytopes, where the “leftover” affine transformations continue on to more extreme values forever. This may be a source of adversarial examples, where the network reaches very high confidence in predicting the wrong class with even seemingly small perturbations. I imagine that a lot of adversarial training involves tamping down these unbounded polytopes. Maybe constraining all neurons to output 0 when the input goes too far out of distribution would solve this.

Jon Garcia 23 Sep 2022 4:10 UTC
5 points
0 ∶ 0
on: You Are Not Measuring What You Think You Are Measuring

First Law of Experiment Design: you are not measuring what you think you are measuring.

In a way, this is just a corollary to the good, old-fashioned “correlation does not imply causation” principle. I guess the important difference is that this is a warning directed at experimenters who tend to assume that they know better.

Second Law of Experiment Design: if you measure enough different stuff, you might figure out what you’re actually measuring.

Before COVID, my manager’s manager would semi-regularly put on a multi-day training course where he would teach everyone in his organization about the process of “Design of Experiments” (DOE) (https://en.wikipedia.org/wiki/Design_of_experiments). I work with other data scientists, statisticians, biochemist, and engineers, often performing experiments with medical devices, so this is relevabt to all of us.

The idea behind a DOE is to create a mutifactorial experimental design that tests the effects of many factors at once. The first step is to brainstorm all possible factors that may influence the experimental results (often using a fishbone diagram [https://en.m.wikipedia.org/wiki/Ishikawa_diagram] to focus on factors arising from Materials, Method, Measurement, Machine, Man, or “Mother Nature”). Each factor is then labelled as either control (C, factors fixed throughout all parts of the experiment), noise (N, random factors that cannot/won’t be controlled for but may cause some variation in the results), or experimental (X, factors to be caried systematically to test for their effects).

Next, high and low settings are chosen for each X factor, and all possible combinations of settings are arranged in a hypercube. Instead of experimenting on one factor at a time with enough repetitions to build up statistical significance, you can perform just a few repetitions at each corner of the hypercube. You can still use the results to look at single-factor contributions to the experimental effect by aggregating data from all corners of the low side and high side of the hypercube along one dimension. But you can also look at the impact of factor interactions and the nonlinearities that result, which would have acted as hidden confounders in more traditional single-factor experiments.

I liked how this method gave a systematic way of thinking about multifactorial experimental effects. Such experiments tend to uncover a lot more information about a system than you would see otherwise. To actually tease out the underlying causal mechanisms at work, though, would require deeper statistics and modeling than we ever got into in that course.

Jon Garcia 2 Sep 2022 22:12 UTC
1 point
0 ∶ 0
in reply to: Thomas Kwa’s comment on: Laziness in AI
It seems that Richard is pointing more toward a means of minimizing how much effort an AI puts toward satisfying its preferences rather than how impactful it allows its goals to be, although the two are very tightly linked (more like minimizing its behavior’s impact on its own energy reserves than on other agents or the environment).
One approach to laziness might be to predict the amount of physical joules it would take to reach each candidate goal that it considers. Goals that would be more satisfying according to its value metric but that would require too much more energy to achieve could be passed over for less satisfying goals that require less energy. As an example, an AI that values seeing smiles on human faces might consider either speaking friendly words to everybody or wiring up everyone’s facial muscles into perpetual smiles. Since the latter would require much more energy to achieve, laziness may cause it to prefer the former.
Another approach could be to minimize the amount of computation required to plan how to achieve its goals (which could, incidentally, also be measured in joules). It would thus prefer simple plans that it can figure out quickly over more complicated plans that might take hours of Monte-Carlo Tree Search to figure out. Simpler plans would be easier for humans to understand and react to, in theory.
Obviously, this doesn’t get anywhere close to solving alignment, and it likely won’t offer any guarantees, but I think it could still be a helpful tool in the alignment toolbox.

Jon Garcia 31 Aug 2022 2:54 UTC
2 points
1 ∶ 0
on: Can We Align a Self-Improving AGI?

Can we align a self-improving AGI?

Only if we can somehow get it to care as much about alignment as we do.

Jon Garcia 29 Aug 2022 4:53 UTC
6 points
0 ∶ 0
on: Robert Long On Why Artificial Sentience Might Matter
“I think there are a lot of things that are morally important that do seem like they require memory or involve memory. So having long term projects and long term goals, that’s something that human beings have. I wouldn’t be surprised if having memory versus not having memory is also just kind of a big determinant of what sorts of experiences you can have or affects what experiences you have in various ways. And yeah, it might be important for having an enduring self through time. So that’s one thing that people also say about large language models is they seem to have these short-lived identities that they spin up as required but nothing that lasts their time.”
There’s the interesting/tragic case of Clive Wearing, who has both retrograde and anterograde amnesia, causing him to experience consciousness only from one moment to the next. His brain is still able to construct internal narratives of his identity and experiences, which I would consider the definition of consciousness, but the lack of access to previously recorded narratives makes it seem to him that those experiences were unconscious and that he’s only just now attaining consciousness for the first time.
I would argue, as I’m sure most humans would agree, that he still has moral worth. So I’m not sure if lack of long-term memory should in itself exclude AIs from moral consideration.
Perhaps the moral worth of a system should be the product of sentience (capacity to experience suffering, at least) and consciousness (level of sophistication of the system’s internal self-narratives), where moral worth is defined as the weight we give to a system’s preferences when calculating trade-offs with other agents’ preferences in morally ambiguous situations. Of course, the problem with language models is, as you alluded to, that you can’t simply take their word for it when they declare their sentience and consciousness, even if that’s perfectly reasonable to do with humans. They’re only trained to predict what humans would say in the same context, after all. We will need to have some way of looking at their internal structures to gauge whether and to what extent they meet these criteria.

Jon Garcia 28 Aug 2022 14:25 UTC
1 point
0 ∶ 0
in reply to: Chris_Leong’s comment on: Basin broadness depends on the size and number of orthogonal features
I was thinking of paulfchristiano’s articles on corrigibility (https://www.lesswrong.com/posts/fkLYhTQteAu5SinAc/corrigibility):
In this post I claim:
1. A benign act-based agent will be robustly corrigible if we want it to be.
2. A sufficiently corrigible agent will tend to become more corrigible and benign over time. Corrigibility marks out a broad basin of attraction towards acceptable outcomes.

Jon Garcia 28 Aug 2022 9:28 UTC
2 points
0 ∶ 0
on: Basin broadness depends on the size and number of orthogonal features
Talking about broad basins in this community reminds me of the claims of corrigible AI: that if you could get an AGI that is at least a little bit willing to cooperate with humans in adjusting its own preferences to better align with ours, then it would fall into a broad basin of corrigibility and become more aligned with human values over time.

I realize than the basins you’re talking about here are more related to perception than value alignment, but do you think your work here could apply? In other words, if broad basins of network performance/generalization can be found by increasing the number of disentangled features available to the network, then would adding more independent measures of value help an AI to find broader basins in value space that would be more likely to contain human-aligned regions?

Maybe humans are able to align with each other so well because we have so many dimensions of value, all competing with each other to drive our behavior. A human brain might have an entire ensemble of empathy mechanisms, each making predictions about some independent, low-dimensional projection of other humans’ preferences, wiring up those predictions into competing goal-generating modules.

Any one estimate of value would be prone to Goodharting, but maybe adding enough dimensions could make that easier to avoid?

Jon Garcia 27 Aug 2022 6:39 UTC
4 points
2 ∶ 0
on: Help Understanding Preferences And Evil
My take is that AGI learning human preferences/values/needs/desires/goals/etc. is a necessary but not sufficient condition for achieving alignment.

If AI learns our preferences base on our behaviors it’s going to learn a lot of “bad” things like lying, stealing and cheating and other much worse things.

First of all, it’s important to note that just because the AGI learns that humans engage in certain behaviors, that by itself does not imply that it will have any inclination to engage in those behaviors itself. Secondly, there is a difference between behaviors and preferences. Lying, theft, and murder are all natural things that members of our species engage in, yes, but they are typically only ever used as a means of fulfilling actual preferences.

Preferences, values, etc. are what motivate our actions. Behaviors are reinforced that tend to satisfy our preferences, stimulate pleasure, alleviate pain, move us toward our goals and away from our antigoals. It’s those preferences, those motivators of action rather than the actions themselves, that I believe Stuart Russell is talking about.

An aligned AGI will necessarily have to take our preferences into account, whether to help us achieve them more intelligently than we could on our own or to steer us toward actions that minimize conflict with others’ preferences. It would ideally take on our values (or the coherent extrapolation or aggregation of them) as its own in some way without necessarily taking on our behavioral policies as its own (though of course it would need to learn about those too).

An unaligned AGI would almost certainly learn human values as well, once it’s past a certain level of intelligence, but it would lack the empathy to care about them beyond the need to plan around them as it pursues its own goals.

Jon Garcia 26 Aug 2022 5:18 UTC
2 points
0 ∶ 0
on: A Test for Language Model Consciousness
This is an interesting proposal. I just have a few thoughts.
I’ve come to see consciousness as the brain’s way of constructing internal narratives of experience. When I am conscious of something, I can recall it later or describe it to someone else. Most cognitive processes are unconscious not because they have no impact on our thoughts but rather because our brains don’t record a narrative of what they’re doing.
Consciousness is not the same thing as awareness or even self-awareness. I can be paying “attention” to the road unconsciously and not recall a single thing about the drive at the end of it because the route was so familiar and nothing noteworthy occurred. A robot can have an internal model of its own body position, but that type of self-awareness doesn’t feel like what we mean by the word “consciousness”. Conversely, I would argue that most nonhuman animals are conscious in a nontrivial way, even if they don’t have the language to report on it and even if they fail the mirror test of “self-awareness”. Furthermore, even a mind without long-term memory can still be considered conscious from moment to moment, in that the mind is continually constructing narratives of what it experiences, but the lack of access to previously recorded narratives makes it seem to that mind that those experiences were unconscious.
Language models can certainly construct narratives in one sense, but those narratives typically do not map to any internal narratives of anything other than the relationships among the words that came before. They are trained to say what they predict a human would say based on context. I’m not saying that language models can’t be conscious. In fact, I think they are currently the closest models we have to conscious algorithms, especially those that ground their symbols. However, I am saying that self-reports of consciousness are not what we should look for. Sure, it’s correlated with consciousness in humans (as are self-awareness, sentience, etc.), but that doesn’t generalize to an alien cognitive architecture.
I’m not sure what the test would have to be, except that you would have to somehow look for structured patterns of activity, which are generated by what the language model experience and which guide the model’s interpretations and decisions regarding what comes next. A purely text-based language model could only ever have a very low-level form of consciousness, in my opinion, even though language is the hallmark of higher-level consciousness in humans. Based on my understanding, I think we’ll start to see recognizable consciousness in the descendants of models like those that add captions to images. A model that can watch a video and then describe its contents in conversation or follow the instructions contained in the video could almost certainly be called conscious, even if the narratives it can hold in its head are not as sophisticated as those a human can handle.
Finally, on the subject of moral patienthood for conscious algorithms, I would say that consciousness is a necessary but not sufficient condition. I think it would also need to have some form of sentience, such as the ability to experience suffering (aversive behavioral motivation), before it could be said to have moral worth. Again, these are correlated in humans and nonhuman animals, but not (yet) in AI.

Jon Garcia 28 Jul 2022 20:42 UTC
1 point
in reply to: Ben’s comment on: Chapter 14: The Unknown and the Unknowable
To quote a future chapter in ROT13:
Naq jura Uneel bcrarq uvf rlrf, ur fnj whfg jung ur’q orra ubcvat gb frr, n sbyqrq cvrpr bs cncre yrsg ba gur sybbe, gur tvsg bs uvf shgher frys.
Pnyy gung cvrpr bs cncre “Cncre-2”.
Uneel gber n cvrpr bs cncre bss uvf cnq.
Pnyy gung “Cncre-1”. Vg jnf, bs pbhefr, gur fnzr cvrpr bs cncre. Lbh pbhyq rira frr, vs lbh ybbxrq pybfryl, gung gur enttrq rqtrf zngpurq.
Uneel erivrjrq va uvf zvaq gur nytbevguz gung ur jbhyq sbyybj.
Vs Uneel bcrarq hc Cncre-2 naq vg jnf oynax, gura ur jbhyq jevgr “101 k 101” qbja ba Cncre-1, sbyq vg hc, fghql sbe na ubhe, tb onpx va gvzr, qebc bss Cncre-1 (juvpu jbhyq gurerol orpbzr Cncre-2), naq urnq ba hc bhg bs gur pnirea yriry gb wbva uvf qbez zngrf sbe oernxsnfg.
Vs Uneel bcrarq hc Cncre-2 naq vg unq gjb ahzoref jevggra ba vg, Uneel jbhyq zhygvcyl gubfr ahzoref gbtrgure.
Vs gurve cebqhpg rdhnyrq 181,429, Uneel jbhyq jevgr qbja gubfr gjb ahzoref ba Cncre-1 naq fraq Cncre-1 onpx va gvzr.
Bgurejvfr Uneel jbhyq nqq 2 gb gur ahzore ba gur evtug naq jevgr qbja gur arj cnve bs ahzoref ba Cncre-1. Hayrff gung znqr gur ahzore ba gur evtug terngre guna 997, va juvpu pnfr Uneel jbhyq nqq 2 gb gur ahzore ba gur yrsg naq jevgr qbja 101 ba gur evtug.
Naq vs Cncre-2 fnvq 997 k 997, Uneel jbhyq yrnir Cncre-1 oynax.
Juvpu zrnag gung gur bayl cbffvoyr fgnoyr gvzr ybbc jnf gur bar va juvpu Cncre-2 pbagnvarq gur gjb cevzr snpgbef bs 181,429.
Vs guvf jbexrq, Uneel pbhyq hfr vg gb erpbire nal fbeg bs nafjre gung jnf rnfl gb purpx ohg uneq gb svaq. Ur jbhyqa’g unir whfg fubja gung C=AC bapr lbh unq n Gvzr-Gheare, guvf gevpx jnf zber trareny guna gung. Uneel pbhyq hfr vg gb svaq gur pbzovangvbaf ba pbzovangvba ybpxf, be cnffjbeqf bs rirel fbeg. Znlor rira svaq gur ragenapr gb Fylgureva’f Punzore bs Frpergf, vs Uneel pbhyq svther bhg fbzr flfgrzngvp jnl bs qrfpevovat nyy gur ybpngvbaf va Ubtjnegf. Vg jbhyq or na njrfbzr purng rira ol Uneel’f fgnaqneqf bs purngvat.
Uneel gbbx Cncre-2 va uvf gerzoyvat unaq, naq hasbyqrq vg.
Cncre-2 fnvq va fyvtugyl funxl unaqjevgvat:
QB ABG ZRFF JVGU GVZR
Uneel jebgr qbja “QB ABG ZRFF JVGU GVZR” ba Cncre-1 va fyvtugyl funxl unaqjevgvat, sbyqrq vg arngyl, naq erfbyirq abg gb qb nal zber gehyl oevyyvnag rkcrevzragf ba Gvzr hagvy ur jnf ng yrnfg svsgrra lrnef byq.
Gb gur orfg bs Uneel’f xabjyrqtr, gung unq orra gur fpnevrfg rkcrevzragny erfhyg va gur ragver uvfgbel bs fpvrapr.
Vg frrzf gung guvf Aneengvir Pbafgehpgvba sbepr (Ryvrmre, va ernyvgl, jura ur jnf pbzvat hc jvgu gur fgbel) bayl fgrref gvzr geniry riragf gbjneq ybbcf gung ner fgnoyr nsgre bar vgrengvba. Lrf, n tenaqsngure cnenqbk jbhyq abeznyyl pnhfr n gjb-vgrengvba ybbc, ohg vs gung jrer cbffvoyr va guvf svpgvbany havirefr, gura Uneel’f rkcrevzrag jbhyq unir fgnovyvmrq ba “457 k 397” engure guna “QB ABG ZRFF JVGU GVZR”. Vg frrzf gung gur Aneengbe fgrrerq Uneel gbjneq jevgvat gung zrffntr engure guna pbagvahvat uvf rkcrevzrag nf cynaarq.
(V pbhyq vzntvat guvf pbfzvp ragvgl vffhvat Uneel n qverpg jneavat (Ybirpensg-fglyr) nsgre gur svefg gvzryvar gung ur punatrq uvf cerivbhf shgher frys’f zrffntr, yrnqvat Uneel gb fraq uvf bja jneavat gb uvf arkg cnfg frys, jvgu gur zrffntr naq unaqjevgvat fgnovyvmvat npebff gvzryvarf fubegyl gurernsgre.)

For the same reason, I would expect any time traveler attempting a grandfather paradox would find themselves thwarted in surprising ways, resulting in no grandfathers getting killed (or realizing that the man they thought was their grandfather actually wasn’t). Even if the time traveler succeeded in the first few iterations, creating a two-iteration loop, I think Narrative forces would cause this to destabilize until a one-iteration loop was achieved.
In other words, no convergent series need be assumed. Convergent series are enforced.

Jon Garcia 5 Feb 2022 21:40 UTC
3 points
on: QNR prospects are important for AI alignment research
Thanks for this excellent overview, and all the links to resources. It’s nice to have a label for this class of models.

Recently, I’ve come to see natural language as a medium for encoding and transmitting cognitive programs. These would be analogous to computer programs, with working memory for state embeddings (heap), for intermediate predicate-call-specific data (stack), and for the causal graph itself (program code). Such programs would be used by conscious agents for simulating models of the world and for carrying out behavioral policies, depending on what part of the brain is carrying out the program.

Under this scheme, the cognitive programs themselves would run on QNR-type models, but they would be constructed or transferred between minds using (non-quasi) natural language. Language, then, involves a traversal over cognitive program space, using the rules of syntax to unravel a cognitive program when speaking/writing or to assemble one when listening/reading.

In the brain, the dynamic routing of information necessary for all this would be carried out by the basal ganglia, while the token embeddings and program code would exist as patterns in the cortex. If this is all valid, then it would seem that the brains of more intelligent species become that way due to the evolution of neural architectures that let the brain operate more like a CPU.

I would be interested to hear if you had any thoughts on this.

Jon Garcia 5 Feb 2022 5:08 UTC
6 points
AF
in reply to: Steven Byrnes’s comment on: [Intro to brain-like-AGI safety] 2. “Learning from scratch” in the brain

Memory ≠ Unstructured memory (and likewise, locally-random ≠ globally-random): [...]

Agreed. I didn’t mean to imply that you thought otherwise.

“just” a memory system + learning algorithm—with a dismissive tone of voice on the “just”: [...]

I apologize for how that came across. I had no intention of being dismissive. When I respond to a post or comment, I typically try to frame what I say for a typical reader as much for the original poster. In this case, I had a sense that a typical reader could get the wrong impression about how the neocortex does what it does if the only sorts of memory systems and learning algorithms that came to mind were things like a blank computer drive and stochastic gradient descent on a feed-forward neural network.

You are absolutely right that the neocortex is equipped to learn from scratch, starting out generating garbage and gradually learning to make sense of the world/body/other-brain-regions/etc., which can legitimately be described as a memory system + learning algorithm. I just wanted anyone reading to appreciate that, at least in biological brains, there is no clean separation between learning algorithm and memory, but that the neocortex’s role as a hierarchical, dynamic, generative simulator is precisely what makes learning from scratch so efficient, since it only has to correlate its intrinsic dynamics with the statistically similar dynamics of learned experience.

I’m sure that there are vastly more ways of implementing learning-from-scratch, maybe some much better ways in fact, and I realize that the exact implementation is probably not relevant to the arguments you plan to make in this sequence. I just feel that a basic understanding of what a real learning-from-scratch system looks like could help drive intuitions of what is possible.

Generative models can be learned from scratch [...]

Indeed, but of course including their own particular structural priors.

Dynamics is not unrelated to neural architecture: [...]

Well, what is a recurrent neural network after all but an arbitrarily deep feed-forward neural network with shared weights across layers? My comment on cortical waves was just to point out a clever way that the brain learns to organize its cortical maps and primes them to expect causality to operate (mostly) locally in space and time. For example, orientation columns in V1 may be adjacent to each other because similarly oriented edges (from moving objects) were consistently presented to the same part of the visual field close in time, such that traveling waves of excitation would teach pyramidal cell A to learn orientation A at time A and then teach neuron B to learn orientation B at time B.

Lottery ticket hypothesis: [...]

“Lottery tickets” (i.e., subnetworks with random initializations that just so happen to give them the right inductive bias to generalize well from the training data for a particular task) probably occur in the brain as much as in current deep learning architectures. However, the issue in DL is that the rest of the network often fails to contribute much to test performance beyond what the lottery ticket subnetwork was able to achieve, as though there was a chasm in model space that the other subnetworks were unable to cross to reach a solution. Evolution seems to have found a way around this problem, at least by the time the neocortex came along, in the sense that the brain seems adept at giving every subnetwork a useful job.

Again, I think the nature of the neocortex as a generative simulator is what makes this feasible with sparse training data. The structure and dynamics of the neocortex have enough in common statistically with the structure and dynamics of the natural world that it is easy for it to align with experience. In contrast, the structure and (lack of) dynamics of current DL systems makes them more brittle when trying to make sense of the natural world.

All that being said, I realize that my comments might be taking things down a rabbit hole. But I appreciate your feedback and look forward to seeing your perspective fleshed out more in the rest of this sequence.

Jon Garcia 4 Feb 2022 4:53 UTC
LW: 6 AF: 2
AF
on: [Intro to brain-like-AGI safety] 2. “Learning from scratch” in the brain
Great summary of the argument. I definitely agree that this will be an important distinction (learning-from-scratch vs. innate circuitry) for AGI alignment, as well as for developing a useful Theory of Cognition. The core of what motivates our behavior must be innate to some extent (e.g., heuristics that evolution programmed into our hypothalamus that tell us how far from homeostasis we’re veering), to act as a teaching signal to the rest of our brains (e.g., learn to generate goal states that minimize the effort required to maintain or return to homeostasis, goal states that would be far too complex to encode genetically).

However, I think that characterizing the telencephelon+cerebellum as just a memory system + learning algorithm is selling it a bit short. Even at the level of abstraction that you’re dealing with here, it seems important to recognize that the neocortex is a dynamic generative simulator. It has intrinsic dynamics that generate top-down spatiotemporal patterns, from regions that represent the highest levels of abstraction down to regions that interface directly with the senses and motor system.

At the beginning, yes, it is simulating nonsense, but the key is that it is more than just randomly initialized memory slots. The sorts of hierarchical patterns that it generates contain information about the dynamical priors that it expects to deal with in life. (Cortical waves, for instance, set priors for spatiotemporally local causal interactions. Retinal waves are probably doing something similar.)

The job of the learning algorithm is, then, to bind the dynamics of experience to the intrinsic dynamics of the neural circuitry, so that the top-down system is better able to simulate real-world dynamics in the future. It probably does so by propagating prediction errors bottom-up from the sensorimotor areas up to the regions representing abstract concepts, making adjustments and learning from them along the way.

In my opinion, some sort of predictive coding scheme using some sort of hierarchical, dynamic generative simulator will be necessary (though not sufficient) to building generally intelligent systems. For learning-from-scratch to be effective (i.e., not require millions of labelled training examples before it can make useful generalizations), it needs to have such a head start.

Jon Garcia 2 Feb 2022 20:42 UTC
23 points
in reply to: Jon Garcia’s comment on: Reflections on six months of fatherhood
One thing I forgot to mention that I meant to say: My oldest can now speak English pretty fluently, using articles and adjectives, even some compound sentences; she just has first and second person pronouns mixed up.

When she says, “Go in your bed,” she’s referring to her own bed. Or when she says, “Give me the ball,” she means that she is giving it to us.

This sort of mistake makes sense when you consider that she learned English by listening to her parents narrate things from our own perspective. Every time we said, “Thank you,” she was giving us something, so now that’s what she says when she gives us something. Every time we used first person pronouns, she correctly inferred that we were referring to ourselves (i.e., not her), and every time we used second person pronouns, we were referring to her.

When she speaks, it sounds like she’s telling us what she predicts we would say, rather than what we would expect her to say as a proper conversational partner.

I’ve heard that babies start out life not just egocentric, but actually unable to distinguish the rest of the universe from themselves. Our youngest is still at that stage. When she is sad, it’s because the universe is sad. Mommy and Daddy are not individuals out there operating independently in an external world; they’re just phenomena that the universe generates to make everything better.

For our oldest, she knows that we are different people, but she seems to see language as a process of narrating everything that happens. Sentences are things we build together, rather than a way for different people to share their own perspectives with each other. “I/me/my” is always said about the other person, and “you/your” is always said about her.

For GPT-3, I feel it’s the same way. Language is something it generates and predicts, not a conversation it participates in from its own perspective.

To correct this In our daughter, we say something like, “No, say, ‘I pooped in my potty,’” if she says, “You pooped in your potty.”

For a large language model, how would we get it to understand that it is separate from us, with its own limitations in understanding, which are different from our own limitations in understanding? Is that something we even want?

Jon Garcia 1 Feb 2022 16:31 UTC
1 point
in reply to: Jon Garcia’s comment on: Before Colour TV, People Dreamed in Black and White
Disclaimer: The preceding was just a simplified story, constructed to make sense out of the complex and confusing reality underlying the qualitative content of dreams. Please take it with a grain of salt.

Jon Garcia 1 Feb 2022 16:27 UTC
5 points
on: Before Colour TV, People Dreamed in Black and White
Even after decades of watching color TV, my Abuela still says that she dreams in black-and-white. Maybe her dream-generator was more impressionable in her younger years.

You’re probably right that a big component of this phenomenon is the vagueness of consciousness latching onto cultural cues to fill in the missing details. But there could also be something else going on.

Our conscious minds are constantly constructing narratives of our personal experiences, trying to make sense out of a complex and confusing world using relatively simple stories. These narratives are encoded by the hippocampus and regenerated for the rest of the brain during sleep as a form of offline learning and regularization.

Stories are what our hippocampus craves. The accumulation of new stories is probably what fuels our curiosity drive, and building shared narratives is central to building community identity.

When we experience life, it’s up to us to construct our own stories. When we watch movies, we’re getting a multimodal story superstimulus delivered to our hippocampus on a silver platter. The saliency is probably off the charts compared to real life.

Based on that theory, it would seem more surprising if dreams didn’t look like movies.