Instructor at Center for Applied Rationality
It looks to me like one can buy this vaccine online without a prescription.
Are you tempted to drop or reduce the size of this trade in light of the UK seeming to have (roughly speaking, for now at least) contained B.1.1.7?
Yeah, makes sense. Fwiw, I have encountered one purportedly 97+ CRI lamp that looked awful to me.
I really appreciate you writing this!
Just wanted to add that my informal impression from a few experiments is that the difference between 90 CRI bulbs and 95+ CRI bulbs is actually large.
Another (unlikely, but more likely than almost all other historical people) candidate for partial future revival: During the 79 AD eruption of Vesuvius, part of this man’s brain was vitrified.
Your posts about the neocortex have been a plurality of the posts I’ve been most excited reading this year. I am super interested in the questions you’re asking, and it has long driven me nuts that I don’t find these questions asked often in the neuroscience literature.
But there’s an aspect of these posts I’ve found frustrating, which is something like the ratio of “listing candidate answers” to “explaining why you think those candidate answers are promising, relative to nearby alternatives.”
Interestingly, I also have this gripe when reading Friston and Hawkins. And I feel like I also have this gripe about my own reasoning, when I think about this stuff—it feels phenomenologically like the only way I know how to generate hypotheses in this domain is by inducing a particular sort of temporary overconfidence.
I don’t feel incentivized to do this nearly as much in other domains, and I’m not sure what’s going on. My lead hypothesis is that in neuroscience, data is so abundant, and theories/frameworks so relatively scarce, that it’s unusually helpful to ignore lots of things—e.g. via the “take as given x, y, z, and p” motion—in order to make conceptual progress. And maybe there’s just so much available data here that it would be terribly sisiphean to try to justify all the things one takes as given when forming or presenting intuitions about underlying frameworks. (Indeed, my lead hypothesis for why so many neuroscientists seem to employ strategies like, “contribute to the ‘understanding road systems’ project by spending their career measuring the angles of stop-sign poles relative to the road,” is that they feel it’s professionally irresponsible, or something, to theorize about underlying frameworks without first trying to concretely falsify a sisiphean-rock-sized mountain of assumptions).
Still, I think some amount of this motion is clearly necessary to avoid accidentally deluding yourself, and the references in your posts make me think you do at least some of it already. So I guess I just want to politely—and super gratefully, I’m really glad you write these posts regardless! If trying to do this would turn you into a stop sign person, don’t do it!—suggest that explicating these more might make it easier for readers to understand and come to share your intuitions.
I have more proto-questions about your model than I have time to flesh them out well enough to describe, but here are some that currently feel top-of-mind:
Say there exist genes that confer advantage in math-ey reasoning. By what mechanism is this advantage mediated, if the neocortex is uniform? One story, popular among the “stereotypes of early 2000s cognitive scientists” section of my models, is that brains have an “especially suitable for maths” module, and that genes induce various architectural changes which can improve or degrade its quality. What would a neocortical uniformist’s story be here—that genes induce architectural changes which alter the quality of the One Learning Algorithm in general? If you explain it as genes having the ability to tweak hyperparameters or the gross wiring diagram in order to degrade or improve certain circuits’ ability to run algorithms this domain-specific, is it still explanatorily useful to describe the neocortex as uniform?
My quick, ~90 min investigation into whether neuroscience as a field buys the neocortical uniformity hypothesis suggested it’s fairly controversial. Do you know why? Are the objections mostly similar to those of Marcus et al.?
Do you have the intuition that aspects of the neocortical algorithm itself (or the subcortical algorithms themselves) might be safety-relevant? Or is your safety-relevance intuition mostly about the subcortical steering mechanism? (Fwiw, I have the former intuition—i.e., I’m suspicious that some of the features of the neocortical algorithm that cause humans to differ from “optimizers” exist for safety-relevant reasons).
In general I feel intensely frustrated with the focus in neuroscience on the implementational Marr Level, relative to the computational and algorithmic levels. I liked the mostly-computational overview here, and the algorithmic sketch in your Predictive Coding = RL + SL + Bayes + MPC post, but I feel bursting with implementational questions. For example:
As I understand it, you mention “PGM-type message-passing” as a candidate class of algorithm that might perform the “select the best from a population of models” function. Do you just mean you suspect there is something in the general vicinity of a belief propagation algorithm going on here, or is your intuition more specific? If the latter, is the Dileep George paper the main thing motivating that intuition?
I don’t currently know whether the neuroscience lit contains good descriptions of how credit assignment is implemented. Do you? Do you feel like you have a decent guess, or know whether someone else does?
I have the same question about whatever mechanism approximates Bayesian priors—I keep encountering vague descriptions of it being encoded in dopamine distributions, but I haven’t found a good explanation of how that might actually work.
Are you sure PP deemphasizes the “multiple simultaneous generative models” frame? I understood the references to e.g. the “cognitive economy” in Surfing Uncertainty to be drawing an analogy between populations of individuals exchanging resources in a market, and populations of models exchanging prediction error in the brain.
Have you thought much about whether there are parts of this research you shouldn’t publish? I notice feeling slightly nervous every time I see you’ve made a new post, I think because I basically buy the “safety and capabilities are in something of a race” hypothesis, and fear that succeeding at your goal and publishing about it might shorten timelines.
Gwern, I’m curious whether you would guess that something like mesa-optimization, broadly construed, is happening in GPT-3?
This post primarily argues that a phenomenon is evidence for [learned models being likely to encode search algorithms]
I do mention interpreting the described results “as tentative evidence” about mesa-optimization at the end of the post, and this interpretation was why I wrote the post; fwiw, my impression remains that this interpretation is correct. But the large majority of the post is just me repeating or paraphrasing claims made by DeepMind researchers, rather than making claims myself; I wrote it this way intentionally, since I didn’t feel I had sufficient domain knowledge to assess the researchers’ claims well myself.
I feel confused about why, given your model of the situation, the researchers were surprised that this phenomenon occurred, and seem to think it was a novel finding that it will inevitably occur given the three conditions described. Above, you mentioned the hypothesis that maybe they just “weren’t very familiar with AI.” Looking at the author list, and at their publications (1, 2, 3, 4, 5, 6, 7, 8), this seems implausible to me. While most of the eight co-authors are neuroscientists by training, three have CS degrees (one of whom is Demis Hassabis), and all but one have co-authored previous ML papers. It’s hard for me to imagine their surprise was due simply to them lacking basic knowledge about RL?
And this OpenAI paper (whose authors I think you would describe as familiar with ML), which the summary of Wang et al. on the DeepMind website describes as “closely related work,” and which appears to me to describe a very similar setup, describes their result in similar terms:
We structure the agent as a recurrent neural network, which receives past rewards, actions, and termination flags as inputs in addition to the normally received observations. Furthermore, its internal state is preserved across episodes, so that it has the capacity to perform learning in its own hidden activations. The learned agent thus also acts as the learning algorithm, and can adapt to the task at hand when deployed.
The OpenAI authors also seem to me to think they can gather evidence about the structure of the algorithm simply by looking at its behavior. Given a similar series of experiments (mostly bandit tasks, but also a maze solver), they conclude:
the dynamics of the recurrent network come to implement a learning algorithm entirely separate from the one used to train the network weights… the procedure the recurrent network implements is itself a full-fledged reinforcement learning algorithm, which negotiates the exploration-exploitation tradeoff and improves the agent’s policy based on reward outcomes… this learned RL procedure can differ starkly from the algorithm used to train the network’s weights.
They then run an experiment designed specifically to distinguish whether meta-RL was giving rise to a model-free system, or “a model-based system which learns an internal model of the environment and evaluates the value of actions at the time of decision-making through look-ahead planning,” and suggest the evidence implies the latter. This sounds like a description of search to me—do you think I’m confused?
I get the impression from your comments that you think it’s naive to describe this result as “learning algorithms spontaneously emerge.” You describe the lack of LW/AF pushback against that description as “a community-wide failure,” and mention updating as a result toward thinking AF members “automatically believe anything written in a post without checking it.”
But my impression is that OpenAI describes their similar result in basically the same way. Do you think my impression is wrong? Or e.g. that their description is also misleading?
I’ve been feeling very confused lately about how people talk about “search,” and have started joking that I’m a search panpsychist. Lots of interesting phenomenon look like piles of thermostats when viewed from the wrong angle, and I worry the conventional lens is deceptively narrow.
That said, when I condition on (what I understand to be) the conventional understanding, it’s difficult for me to imagine how e.g. the maze-solver described in the OpenAI paper reliably and quickly locates the exit to new mazes, without doing something reasonably describable as searching for them.
And it seems to me that Wang et al. should be taken as evidence that “learning algorithms producing other search-performing learning algorithms” is convergently useful/likely to be a common feature of future systems, even if you don’t think that’s what happened in their paper, assuming you assign some credence to their hypothesis that this is what’s going on in PFC, and to the hypothesis that search occurs in PFC.
If the primary difference between the DeepMind and OpenAI meta-RL architecture and the PFC/DA architecture is scale, then I think there’s reasonable reason to suspect that something much like mesa-optimization will emerge in future meta-RL systems, even if it hasn’t yet. That is, I interpret this result as evidence for the hypothesis that highly competent general-ish learners might tend to exhibit this feature, since (among other reasons) it increased my credence that it is already exhibited by the only existing member of that reference class.
Upthread, Evan mentions agreeing that this result is “not new evidence in favor of mesa-optimization.” But he also mentions that Risks from Learned Optimization references these two papers, describing them as “the closest to producing mesa-optimizers of any existing machine learning research.” I feel confused about how to reconcile these two claims. I didn’t realize these papers were mentioned in Risks from Learned Optimization, but if I had, I think I would have been even more inclined to post this/try to ensure people knew about the results, since my (perhaps naive, perhaps not understanding ways this is disanalogous) prior is that the closest existing example to this problem might provide evidence about its nature or likelihood.
In college, people would sometimes discuss mu-eliciting questions like, “What does it mean to be human?”
I came across this line in a paper tonight and laughed out loud, imagining it as an answer:
“Maximizing this objective is equivalent to minimizing the cumulative pseudo-regret.”
I appreciate you writing this, Rohin. I don’t work in ML, or do safety research, and it’s certainly possible I misunderstand how this meta-RL architecture works, or that I misunderstand what’s normal.
That said, I feel confused by a number of your arguments, so I’m working on a reply. Before I post it, I’d be grateful if you could help me make sure I understand your objections, so as to avoid accidentally publishing a long post in response to a position nobody holds.
I currently understand you to be making four main claims:
The system is just doing the totally normal thing “conditioning on observations,” rather than something it makes sense to describe as “giving rise to a separate learning algorithm.”
It is probably not the case that in this system, “learning is implemented in neural activation changes rather than neural weight changes.”
The system does not encode a search algorithm, so it provides “~zero evidence” about e.g. the hypothesis that mesa-optimization is convergently useful, or likely to be a common feature of future systems.
The above facts should be obvious to people familiar with ML.
Does this summary feel like it reasonably characterizes your objection?
That gwern essay was helpful, and I didn’t know about it; thanks.
The scenario I had in mind was one where death occurs as a result of damage caused by low food consumption, rather than by suicide.
One way catastrophic alignment in this sense is difficult for humans is that the PFC cannot divorce itself from the DA; I’d expect that a failure mode leading to systematically low DA rewards would usually be corrected
I’m not sure such divorce is all that rare. For example, anorexia sometimes causes people to find food anti-rewarding (repulsive/inedible, even when they’re dying and don’t wish to), and I can imagine that being because PFC actually somehow alters DAs reward function.
That said, I do share the hunch that something like a “divorce resistance” trick occurs and is helpful. I took Kaj and Steve to be gesturing at something similar elsewhere in the thread. But I notice feeling confused about how exactly this trick works. Does it scale...?
I have the intuition that it doesn’t—that as the systems increase in power, divorce will occur more easily. That is, I have the intuition that if PFC were trying, so to speak, to divorce itself from DA supervision, that it could probably find some easy-ish way to succeed, e.g. by reconfiguring itself to hide activity from DA, or to send reward-eliciting signals to DA regardless of what goal it was pursuing.
I think it makes more sense to operationalize “catastrophic” here as “leading to systematically low DA reward
Thanks—I feel pretty convinced that this operationalization makes more sense than the one I proposed.
That’s a really interesting point, and I hadn’t considered it. Thanks!
Kaj, the point I understand you to be making is: “The inner RL algorithm in this scenario seems likely to be reliably aligned with the outer RL algorithm, since the former was selected specifically on the basis of it being good at accomplishing the latter’s objective, and since if the former deviates from pursuing that objective it will receive less reward from the outer alg, leading it to reconfigure itself to be more aligned. And since the two algorithms operate on similar time scales, we should expect any such misalignment to be noticed/corrected quickly.” Does this seem like a reasonable paraphrase?
It doesn’t feel obvious to me that the outer layer will be able to reliably steer the inner layer in this sense, especially as the system becomes more powerful. For example, it seems plausible to me that the inner layer might come to optimize for its proxy estimations of outer reward more than for outer reward itself, and that those two things could become decoupled.
Ah, I see. The high death rate was what made it seem often-catastrophic to me. Is your objection that the high death rate doesn’t reflect something that might reasonably be described as “optimizing for one goal at the expense of all others”? E.g., because many of the deaths are suicides, in which case persistence may have been net negative from the perspective of the rest of their goals too? Or because deaths often result from people calibratedly taking risky but non-insane actions, who just happened to get unlucky with heart muscle integrity or whatever?
Yeah, I wrote that confusingly, sorry; edited to clarify. I just meant that of the limited set of candidate examples I’d considered, (my model, which may well be wrong) of anorexia feels most straightforwardly like an example of something capable of causing catastrophic within-brain inner alignment failure. That is, it currently feels natural to me to model anorexia as being caused by an optimizer for thinness arising in brains, which can sometimes gain sufficient power that people begin to optimize for that goal at the expense of essentially all other goals. But I don’t feel confident in this model.
I agree, in the case of evolution/humans. In the text above, I meant to highlight what seemed to me like a relative lack of catastrophic *within-mind* inner alignment failures, e.g. due to conflicts between PFC and DA. Death of the organism feels to me like a reasonable way to operationalize “catastrophic” in these cases, but I can imagine other reasonable ways.