Jon Garcia(Jonathan William Garcia)

Karma: 324

I have a PhD in Computational Neuroscience from UCSD (Bachelor’s was in Biomedical Engineering with Math and Computer Science minors). Ever since junior high, I’ve been trying to figure out how to engineer artificial minds, and I’ve been coding up artificial neural networks ever since I first learned to program. Obviously, all my early designs were almost completely wrong/unworkable/poorly defined, but I think my experiences did prime my brain with inductive biases that are well suited for working on AGI.

Although I now work as a data scientist in R&D at a large medical device company, I continue to spend my free time studying the latest developments in AI/ML/DL/RL and neuroscience and trying to come up with models for how to bring it all together into systems that could actually be implemented. Unfortnately, I don’t seem to have much time to develop my ideas into publishable models, but I would love to have the opportunity to share ideas with those who do.

Of course, I’m also very interested in AI Alignment (hence the account here). My ideas on that front mostly fall into the “learn (invertible) generative models of human needs/goals and hook those up to the AI’s own reward signal” camp. I think methods of achieving alignment that depend on restricting the AI’s intelligence or behavior are about as destined to failure in the long term as Prohibition or the War on Drugs in the USA. We need a better theory of what reward signals are for in general (probably something to do with maximizing (minimizing) the attainable (dis)utility with respect to the survival needs of a system) before we can hope to model human values usefully. This could even extend to modeling the “values” of the ecological/socioeconomic/political supersystems in which humans are embedded or of the biological subsystems that are embedded within humans, both of which would be crucial for creating a better future.

Jon Garcia 5 Feb 2022 21:40 UTC
1 point
on: QNR prospects are important for AI alignment research
Thanks for this excellent overview, and all the links to resources. It’s nice to have a label for this class of models.

Recently, I’ve come to see natural language as a medium for encoding and transmitting cognitive programs. These would be analogous to computer programs, with working memory for state embeddings (heap), for intermediate predicate-call-specific data (stack), and for the causal graph itself (program code). Such programs would be used by conscious agents for simulating models of the world and for carrying out behavioral policies, depending on what part of the brain is carrying out the program.

Under this scheme, the cognitive programs themselves would run on QNR-type models, but they would be constructed or transferred between minds using (non-quasi) natural language. Language, then, involves a traversal over cognitive program space, using the rules of syntax to unravel a cognitive program when speaking/writing or to assemble one when listening/reading.

In the brain, the dynamic routing of information necessary for all this would be carried out by the basal ganglia, while the token embeddings and program code would exist as patterns in the cortex. If this is all valid, then it would seem that the brains of more intelligent species become that way due to the evolution of neural architectures that let the brain operate more like a CPU.

I would be interested to hear if you had any thoughts on this.

Jon Garcia 5 Feb 2022 5:08 UTC
LW: 4 AF: 3
AF
in reply to: Steven Byrnes’s comment on: [Intro to brain-like-AGI safety] 2. “Learning from scratch” in the brain

Memory ≠ Unstructured memory (and likewise, locally-random ≠ globally-random): [...]

Agreed. I didn’t mean to imply that you thought otherwise.

“just” a memory system + learning algorithm—with a dismissive tone of voice on the “just”: [...]

I apologize for how that came across. I had no intention of being dismissive. When I respond to a post or comment, I typically try to frame what I say for a typical reader as much for the original poster. In this case, I had a sense that a typical reader could get the wrong impression about how the neocortex does what it does if the only sorts of memory systems and learning algorithms that came to mind were things like a blank computer drive and stochastic gradient descent on a feed-forward neural network.

You are absolutely right that the neocortex is equipped to learn from scratch, starting out generating garbage and gradually learning to make sense of the world/body/other-brain-regions/etc., which can legitimately be described as a memory system + learning algorithm. I just wanted anyone reading to appreciate that, at least in biological brains, there is no clean separation between learning algorithm and memory, but that the neocortex’s role as a hierarchical, dynamic, generative simulator is precisely what makes learning from scratch so efficient, since it only has to correlate its intrinsic dynamics with the statistically similar dynamics of learned experience.

I’m sure that there are vastly more ways of implementing learning-from-scratch, maybe some much better ways in fact, and I realize that the exact implementation is probably not relevant to the arguments you plan to make in this sequence. I just feel that a basic understanding of what a real learning-from-scratch system looks like could help drive intuitions of what is possible.

Generative models can be learned from scratch [...]

Indeed, but of course including their own particular structural priors.

Dynamics is not unrelated to neural architecture: [...]

Well, what is a recurrent neural network after all but an arbitrarily deep feed-forward neural network with shared weights across layers? My comment on cortical waves was just to point out a clever way that the brain learns to organize its cortical maps and primes them to expect causality to operate (mostly) locally in space and time. For example, orientation columns in V1 may be adjacent to each other because similarly oriented edges (from moving objects) were consistently presented to the same part of the visual field close in time, such that traveling waves of excitation would teach pyramidal cell A to learn orientation A at time A and then teach neuron B to learn orientation B at time B.

Lottery ticket hypothesis: [...]

“Lottery tickets” (i.e., subnetworks with random initializations that just so happen to give them the right inductive bias to generalize well from the training data for a particular task) probably occur in the brain as much as in current deep learning architectures. However, the issue in DL is that the rest of the network often fails to contribute much to test performance beyond what the lottery ticket subnetwork was able to achieve, as though there was a chasm in model space that the other subnetworks were unable to cross to reach a solution. Evolution seems to have found a way around this problem, at least by the time the neocortex came along, in the sense that the brain seems adept at giving every subnetwork a useful job.

Again, I think the nature of the neocortex as a generative simulator is what makes this feasible with sparse training data. The structure and dynamics of the neocortex have enough in common statistically with the structure and dynamics of the natural world that it is easy for it to align with experience. In contrast, the structure and (lack of) dynamics of current DL systems makes them more brittle when trying to make sense of the natural world.

All that being said, I realize that my comments might be taking things down a rabbit hole. But I appreciate your feedback and look forward to seeing your perspective fleshed out more in the rest of this sequence.

Jon Garcia 4 Feb 2022 4:53 UTC
LW: 4 AF: 3
AF
on: [Intro to brain-like-AGI safety] 2. “Learning from scratch” in the brain
Great summary of the argument. I definitely agree that this will be an important distinction (learning-from-scratch vs. innate circuitry) for AGI alignment, as well as for developing a useful Theory of Cognition. The core of what motivates our behavior must be innate to some extent (e.g., heuristics that evolution programmed into our hypothalamus that tell us how far from homeostasis we’re veering), to act as a teaching signal to the rest of our brains (e.g., learn to generate goal states that minimize the effort required to maintain or return to homeostasis, goal states that would be far too complex to encode genetically).

However, I think that characterizing the telencephelon+cerebellum as just a memory system + learning algorithm is selling it a bit short. Even at the level of abstraction that you’re dealing with here, it seems important to recognize that the neocortex is a dynamic generative simulator. It has intrinsic dynamics that generate top-down spatiotemporal patterns, from regions that represent the highest levels of abstraction down to regions that interface directly with the senses and motor system.

At the beginning, yes, it is simulating nonsense, but the key is that it is more than just randomly initialized memory slots. The sorts of hierarchical patterns that it generates contain information about the dynamical priors that it expects to deal with in life. (Cortical waves, for instance, set priors for spatiotemporally local causal interactions. Retinal waves are probably doing something similar.)

The job of the learning algorithm is, then, to bind the dynamics of experience to the intrinsic dynamics of the neural circuitry, so that the top-down system is better able to simulate real-world dynamics in the future. It probably does so by propagating prediction errors bottom-up from the sensorimotor areas up to the regions representing abstract concepts, making adjustments and learning from them along the way.

In my opinion, some sort of predictive coding scheme using some sort of hierarchical, dynamic generative simulator will be necessary (though not sufficient) to building generally intelligent systems. For learning-from-scratch to be effective (i.e., not require millions of labelled training examples before it can make useful generalizations), it needs to have such a head start.

Jon Garcia 2 Feb 2022 20:42 UTC
21 points
in reply to: Jon Garcia’s comment on: Reflections on six months of fatherhood
One thing I forgot to mention that I meant to say: My oldest can now speak English pretty fluently, using articles and adjectives, even some compound sentences; she just has first and second person pronouns mixed up.

When she says, “Go in your bed,” she’s referring to her own bed. Or when she says, “Give me the ball,” she means that she is giving it to us.

This sort of mistake makes sense when you consider that she learned English by listening to her parents narrate things from our own perspective. Every time we said, “Thank you,” she was giving us something, so now that’s what she says when she gives us something. Every time we used first person pronouns, she correctly inferred that we were referring to ourselves (i.e., not her), and every time we used second person pronouns, we were referring to her.

When she speaks, it sounds like she’s telling us what she predicts we would say, rather than what we would expect her to say as a proper conversational partner.

I’ve heard that babies start out life not just egocentric, but actually unable to distinguish the rest of the universe from themselves. Our youngest is still at that stage. When she is sad, it’s because the universe is sad. Mommy and Daddy are not individuals out there operating independently in an external world; they’re just phenomena that the universe generates to make everything better.

For our oldest, she knows that we are different people, but she seems to see language as a process of narrating everything that happens. Sentences are things we build together, rather than a way for different people to share their own perspectives with each other. “I/me/my” is always said about the other person, and “you/your” is always said about her.

For GPT-3, I feel it’s the same way. Language is something it generates and predicts, not a conversation it participates in from its own perspective.

To correct this In our daughter, we say something like, “No, say, ‘I pooped in my potty,’” if she says, “You pooped in your potty.”

For a large language model, how would we get it to understand that it is separate from us, with its own limitations in understanding, which are different from our own limitations in understanding? Is that something we even want?

Jon Garcia 1 Feb 2022 16:31 UTC
1 point
in reply to: Jon Garcia’s comment on: Before Colour TV, People Dreamed in Black and White
Disclaimer: The preceding was just a simplified story, constructed to make sense out of the complex and confusing reality underlying the qualitative content of dreams. Please take it with a grain of salt.

Jon Garcia 1 Feb 2022 16:27 UTC
5 points
on: Before Colour TV, People Dreamed in Black and White
Even after decades of watching color TV, my Abuela still says that she dreams in black-and-white. Maybe her dream-generator was more impressionable in her younger years.

You’re probably right that a big component of this phenomenon is the vagueness of consciousness latching onto cultural cues to fill in the missing details. But there could also be something else going on.

Our conscious minds are constantly constructing narratives of our personal experiences, trying to make sense out of a complex and confusing world using relatively simple stories. These narratives are encoded by the hippocampus and regenerated for the rest of the brain during sleep as a form of offline learning and regularization.

Stories are what our hippocampus craves. The accumulation of new stories is probably what fuels our curiosity drive, and building shared narratives is central to building community identity.

When we experience life, it’s up to us to construct our own stories. When we watch movies, we’re getting a multimodal story superstimulus delivered to our hippocampus on a silver platter. The saliency is probably off the charts compared to real life.

Based on that theory, it would seem more surprising if dreams didn’t look like movies.

Jon Garcia 1 Feb 2022 15:59 UTC
27 points
on: Reflections on six months of fatherhood
As a father of two very young daughters (2 years old and 2 months old), I can really appreciate this. As someone with a background in computational neuroscience and some linguistics/NLP/ML/AI, I’ve loved watching them grow and making educated guesses about what sorts of computations could be going on inside their little brains at each developmental stage.

From the earliest days, when they can’t even focus on our faces, hard as they try (I can tell what you’re trying to do, superior coliculus and fusiform face area; you can do it!), to later on when they’re walking and talking (still working on that theory of mind, though).

Language development has been especially fun to watch. Early on, they love just staring at your mouth as you enunciate the various phonemic sequences of what will become their native language. As they become more aware, you can see them start to comprehend when you use simple sentences to narrate things within their field of attention. And they definitely learn to understand more complex language long before they can talk. Patterns built upon patterns, just like deep transformer models, yet still quite different.

When my daughter began to pronounce words, we started pausing intermittently while reading her favorite books or singing familiar songs, and we would have her complete the last word of each line. I couldn’t help but think of how large language models are often trained to perform next-token prediction in a similar way. Although, it’s clear that the human brain has some sort of extra bias that makes it easier to memorize songs and poetry than prose.

And it’s funny how trying to talk to babies reveals just how much of our adult-level world model we assume when we communicate. Once, when our oldest was trying to use a sippy cup on her own for the first time, we saw her putting it to her mouth like we did but failing to get any water. To help her out, I told her to lift the bottom of her cup to get at the water. She then proceeded to lift the entire cup above her head, which surprisingly did not help her. (Eventually she got it.)

For all their temporary limitations, it’s clear that there is a lot going on inside babies’ heads. You can learn a lot about the human brain and cognitive algorithms and biases by studying them carefully. It’s certainly the cutest way to do so.

Jon Garcia 29 Jan 2022 19:19 UTC
8 points
on: Deconfusing Deception
This reminds me of the proverb:

Oh what a tangled web we weave, when first we practice to deceive.

The more the algorithm probes the deception, the more effort the deceiver has to put into maintaining the illusion. Reality is completely self-consistent, so the accumulation of enough evidence into a sufficiently coherent world model will always resolve these types of deceptions if no further effort is applied to maintain them.

When I think of deception, though, I typically imagine a deceiver trying to create a strategic mismatch between their beliefs and another agent’s beliefs. This information asymmetry is typically over something that the deceiver does not expect the agent to be able to investigate easily. It then allows them to get away with something that is in the deceiver’s interest but against (what they think is probably) the other agent’s interest.

It’s usually something like a claim about the health benefits of snake oil (where the agent lacks the time or the resources to perform or look up a randomized controlled trial), rather than a claim about the existence of a physical object that’s visible and within reach.

Maybe magic tricks go that route, though. Although those sorts of deceptions seem to be more about creating a superstimulus for the audience’s curiosity drive, since the audience knows that their senses are being deceived and yet go out of their way to experience that deception.

Jon Garcia 29 Jan 2022 15:36 UTC
3 points
on: Aligned AI Needs Slack
Good point. I can see a couple main ways to give an AI slack in its pursuit of optimization:
1. Set the level of precision on the various features of the world that it optimizes for. Lower precision would then mean the AI puts forth less effort toward optimizing it and puts up less of a fight against other agents (i.e., humans) who try to steer it away from the optimum. The AI might also have a goal-generating module that tries to predict what goals would produce what level of utility. Having lower precision would mean that it would sample goal states from a broader region of latent space and therefore be willing to settle for reaching the sub-optimal goals that result.
2. Seek Pareto improvements. When the AI is trying to optimize simultaneously for an entire ensemble of human preferences (or heuristics of those preferences) with no strict ordering among them, it would only fight for goals that improve on multiple dimensions simultaneously. World states on the same Pareto frontier would be equivalent from its perspective, and it could then work with humans who want to improve things according to their hidden preferences by just constraining any changes to lie on that same manifold.

Jon Garcia 28 Jan 2022 23:52 UTC
3 points
on: Causality, Transformative AI and alignment—part I
I like the direction you’re going with this. I agree that causal reasoning is necessary, but not sufficient, for getting alignable TAI. I think just getting step 2 done (Summarize the literature on causality from an AI safety perspective) could have a huge impact in terms of creating a very helpful resource for AI alignment researches to pull insights from.

Our worry is that ML researchers, once they figure out how, will introduce a similar “overidentifying causality” inductive bias into models. This would mean that very powerful models with potentially big impacts have the causal model of a political pundit rather than a scientist.

One possible workaround for this would be to take a Bayesian approach. Bayes’ rule is all about comparing the predictions (likelihoods) of different models (hypotheses) to assign higher probability mass to those models with greater predictive power.

Consider a system that uses an ensemble of differently structured causal models, each containing slots for different factors (e.g., {A, B} (no causal relationship), {A → B}, {A ← B}, {A ← C → B}, …). Then for any given phenomenon, the system could feed in all relevant factors into the slots of the causal graph of each model, then use each model to make predictions, both about passive observations and about the results of interventions. Those causal (or acausal) models with the greatest predictive power would win out after the accumulation of enough evidence.

Of course, the question still remains about how the system chooses which factors are relevant, or about how it decides what kind of state transformations each causal arrow induces. But I think the general idea of multiple hypothesis testing should be sufficient to get any causally reasoning AI to think more like a scientist than a pundit.

Jon Garcia 28 Jan 2022 14:53 UTC
11 points
on: Newcomb’s Grandfather
Time travel is impossible. Therefore, Omega’s “going back in time” to kill your grandfather was actually just it predicting you taking both boxes and killing him based on that prediction. Or, if Omega comes from outside of spacetime and just interacts with our universe by manipulating the universal wavefunction, it could at most kill your grandfather from another Everett branch, since the one from yours definitely survived.

In the former case, yes, your existence is evidence that Omega predicted you would just take box B. In the latter case, it’s best to act as if that’s what Omega predicted anyway.

Jon Garcia 27 Jan 2022 5:22 UTC
LW: 4 AF: 3
AF
on: [Intro to brain-like-AGI safety] 1. What’s the problem & Why work on it now?
Great introduction. As someone with a background in computational neuroscience, I’m really looking forward to what you have to say on all this.

By the way, you seem to use a very broad umbrella for covering “brain-like-AGI” approaches. Would you classify something like a predictive coding network as more brain-like or more prosaic? What about a neural Turing machine? In other words, do you have a distinct boundary in mind for separating the red box from the blue box, or is your classification more fuzzy (or does it not matter)?

Jon Garcia 25 Jan 2022 20:56 UTC
1 point
on: Why do we need a NEW philosophy of progress?
I agree. Progress ought to be seen as a project that humanity collectively works towards, not an inevitable consequence of Natural Law nor a corrupting addiction to be resisted.

For a new vision of progress, I think we need something like a Gaia Initiative, like the Gaia Hypothesis except that instead of presuming that the Earth functions like a giant superorganism, we frame that as a goal that humanity should strive to achieve. To that end, if we could solve alignment, I think that AGI could be used primarily as a tool for solving coordination problems, aiming to maximize sustainable cooperative harmony, both for humanity and for the global ecosystem.

Jon Garcia 24 Jan 2022 14:38 UTC
1 point
on: Words are Imprecise Telepathy
It’s not just knowledge that we telepathize with language. With declarative sentences, we transfer whole multimodal scenes from one mind to another. With interrogative sentences, we initiate database queries (memory retrieval) in other minds. And with imperative sentences, we copy over behavioral programs into the minds of other agents for execution.

Jon Garcia 21 Jan 2022 1:46 UTC
1 point
on: Harry Potter and the Methods of Psychomagic | Chapter 3: Intelligence Explosions
The obvious first step in this setting to initiating an intelligence explosion would be to find the Lost Diadem of Ravenclaw, even if all it does is accelerate processing speed and/or expand working memory capacity. Though it might be more complicated if it’s part of Tom Riddle’s enhanced horcrux network, especially if it’s one of the items he buried deep underground/sank in the ocean/hovered invisibly in the atmosphere/etc. I’m interested to see where you take this.

Jon Garcia 19 Jan 2022 3:45 UTC
3 points
in reply to: gwern’s comment on: Thought Experiments Provide a Third Anchor
Thanks for the links!

Jon Garcia 19 Jan 2022 3:01 UTC
1 point
in reply to: gwern’s comment on: Thought Experiments Provide a Third Anchor

There has been work on constructing adversarial examples for human brains, and some interesting demonstrations of considerable neural-level control even with our extremely limited ability to observe brains

Do you have a source for this? I would be interested in looking into it. I could see this happening for isolated neurons, at least, but it would be curious if it could happen for whole circuits in vivo.

Does this go beyond just manipulating how our brains process optical illusions? I don’t see how the brain would perceive the type of pixel-level adversarial perturbations most of us think of (e.g.: https://openai.com/blog/adversarial-example-research/) as anything other than noise, if it even reaches past the threshold of perception at all. The sorts of illusions humans fall prey to are qualitatively different, taking advantage of our perceptual assumptions like structural continuity or color shifts under changing lighting conditions or 3-dimensionality. We don’t tend to go from making good guesses about what something is to being wildly, confidently incorrect when the texture changes microscopically.

My guess would be that you could get rid of a lot of adversarial susceptibility from DL systems by adding in the right kind of recurrent connectivity (as in predictive coding, where hypotheses about what the network is looking at help it to interpret low-level features), or even by finding a less extremizing nonlinearity than ReLU (e.g.: https://towardsdatascience.com/neural-networks-an-alternative-to-relu-2e75ddaef95c). Such changes might get us closer to how the brain does things.

Overparameterization, such as through making the network arbitrarily deep, might be able to get you around some of these limitations eventually (just like a fully connected NN can do the same thing as a CNN in principle), but I think we’ll have to change how we design neural networks at a fundamental level in order to avoid these issues more effectively in the long term.

Jon Garcia 18 Jan 2022 21:29 UTC
3 points
in reply to: jacob_cannell’s comment on: Thought Experiments Provide a Third Anchor
To an extent that’s true. There are certainly some similarities in how human brains work and how deep learning works, if for no other reason than that DL uses a connectionist approach to AI, which has given narrow AIs something like an intuition, rather than the hard-coded rules of GOFAI. And yes, once we start developing goal-oriented artificial agents, humans will remain, for a long time, the best model we have for approaching an understanding of them.

However, remember how susceptible current DL models can be to adversarial examples, even when the adversarial examples have no perceptible difference to non-adversarial examples as fas as humans can tell. That means that something is going on in DL systems that is qualitatively much different from how human brains process information. Something that makes them fragile in a way that is hard to anthropomorphize. Something alien.

And then there is the orthogonality thesis. Even though humans are the best example we currently have of general intelligence, there is no reason to think that the first AGIs will have goal/value structures any less alien to humans than would a superintelligent spider. Anthropomorphization of such systems carries the risk of assuming too much about how they think or what they want, where we miss critical points of misalignment.

Jon Garcia 17 Jan 2022 21:41 UTC
4 points
on: Truthful LMs as a warm-up for aligned AGI
I like this approach to alignment research. Getting AIs to be robustly truthful (producing language output that is consistent with their best models of reality, modulo uncertainty) seems like it falls in the same space as getting them to keep their goals consistent with their best estimates of human goals and values.

As for avoiding negligent falsehoods, I think it will be crucial for the AI to have explicit estimates of its uncertainty for anything it might try to say. To a first approximation, assuming the system can project statements to a consistent conceptual space, it could predict the variance in the distribution of opinions in its training data around any particular issue. Then it could use this estimate of uncertainty to decide whether to state something confidently, to add caveats to what it says, or to turn it into a question for the interlocutor.

Jon Garcia 15 Jan 2022 3:18 UTC
10 points
on: [Linkpost] [Fun] CDC To Send Pamphlet On Probabilistic Thinking

Walensky added that if Americans took away one easy lesson from the pamphlet, she hoped it would be P(H|E) = (P(E|H) * P(H)) / P(E).

Seriously, do this.