I’d have liked if @johnswentworth has responded to Eliezer’s comment at more length, though maybe he did and I missed? Still, giving this post a 4 in the review, same as the other.
Done.
johnswentworth
Karma: 62,178
It’s now the 2024 year-in-review, and Ruby expressed interest in seeing me substantively respond to this comment, so here I am responding a year and a half later.
The concept of “easy to understand” seems like a good example, so let’s start there. My immediate reaction to that example is “yeah duh of course an AI won’t use that concept internally in a human-like way”, followed by “wait what the heck is Eliezer picturing such that that would even be relevant?”, followed by “oh… probably he’s used to noobs anthropomorphizing AI all the time”. There is an argument to be made that “easy for a human to understand” is ontologically natural in our environment for predicting humans, even for nonhuman minds insofar as they’re modelling humans (though I’m not confident). But I wouldn’t expect an AI to use that concept internally the way a human does, even if the concept is convergent for predicting humans. A similar argument would apply to lots of other reflective concepts: the human version would be (potentially) convergent for modelling humans, but the AI will use that concept in a very different way internally than humans would. That means we can’t rely on our anthropomorphizing intuitions to reason about how the AI will use such concepts. But we could still potentially rely on such concepts for alignment purposes so long as we’re not e.g. expecting the AI to magically care about them or use them in human-like ways. I would guess Eliezer mostly agrees with that, though probably we could go further down this hole before bottoming out.
But I don’t think that line of discussion is all that interesting. The immediate use-cases for a strong AI, like e.g. running uploaded humans, really shouldn’t require a bunch of reflective concepts anyway. The main things I want to immediately do with AI, as a first use-case, tend to look like hard science and engineering, not like reasoning about humans or doing anthropomorphic things. I usually don’t even imagine interfacing to the thing in natural language.
There is another interesting delta here, one which developed in the intervening year-and-a-half since Eliezer wrote this comment. I think corrigibility is maybe not actually a reflective concept.
Eliezer’s particular pointer to corrigibility is extremely reflective:
The “hard problem of corrigibility” is interesting because of the possibility that it has a relatively simple core or central principle—rather than being value-laden on the details of exactly what humans value, there may be some compact core of corrigibility that would be the same if aliens were trying to build a corrigible AI, or if an AI were trying to build another AI.
…
We can imagine, e.g., the AI imagining itself building a sub-AI while being prone to various sorts of errors, asking how it (the AI) would want the sub-AI to behave in those cases, and learning heuristics that would generalize well to how we would want the AI to behave if it suddenly gained a lot of capability or was considering deceiving its programmers and so on.
(source) But Eliezer comes at corrigibility very indirectly. In principle, there is some object-level stuff an AI would do when building a powerful sub-AI while prone to various sorts of errors. That object-level stuff is not itself necessarily all that reflective. The move of “we’re talking about whatever stuff an AI would do when building a powerful sub-AI while prone to various sorts of errors” is extremely reflective, but in principle one might identify the object-level stuff via some other method which needn’t be that reflective.
In particular, Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals gestures at a very-ontologically-natural-looking concept which sure does seem to overlap an awful lot with corrigibility, to the point where it maybe just is corrigibility (insofar as corrigibility actually has a simple core at all).
LLMs mimic human text. That is the first and primary thing they are optimized for. Humans motivatedly reason, which shows up in their text. So, LLMs trained to mimic human text will also mimic motivated reasoning, insofar as they are good at mimicking human text. This seems like the clear default thing one would expect from LLMs; it does not require hypothesizing anything about motivated reasoning being adaptive.
Addendum: Why arguments that motivated reasoning is somehow secretly rational?
Once one learns to spot motivated reasoning in one’s own head, the short term planner has a much harder problem. It’s still looking for outputs-to-rest-of-brain which will result in e.g. playing more Civ, but now the rest of the brain is alert to the basic tricks. But the short term planner is still looking for outputs, and sometimes it stumbles on a clever trick: maybe motivated reasoning is (long-term) good, actually? And then the rest of the brain goes “hmm, ok, sus, but if true then yeah we can play more Civ” and the short term planner is like “okey dokey let’s go find us an argument that motivated reasoning is (long-term) good actually!”.
In short: “motivated reasoning is somehow secretly rational” is itself the ultimate claim about which one would motivatedly-reason. It’s very much like the classic anti-inductive agent, which believes that things which have happened more often before are less likely to happen again: “but you’ve been wrong every time before!” “yes, exactly, that’s why I’m obviously going to be right this time”. Likewise, the agent which believes motivated reasoning is good actually: “but your argument for motivated reasoning sure seems pretty motivated in its own right” “yes, exactly, and motivated reasoning is good so that’s sensible”.
… which, to be clear, does not imply that all arguments in favor of motivated reasoning are terrible. This is meant to be somewhat tongue-in-cheek; there’s a reason it’s not in the post. But it’s worth keeping an eye out for motivated arguments in favor of motivated reasoning, and discounting appropriately (which does not mean dismissing completely).
There is definitely a standard story which says roughly “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”. I do not think that story stands up well under examination; when I think of standard day-to-day examples of motivated reasoning, that pattern sounds like a plausible explanation for some-but-a-lot-less-than-all of them.
For example: suppose it’s 10 pm and I’ve been playing Civ all evening. I know that I should get ready for bed now-ish. But… y’know, this turn isn’t a very natural stopping point. And it’s not that bad if I go to bed half an hour late, right? Etc. Obvious motivated reasoning. But man, that motivated reasoning sure does not seem very socially-oriented? Like, sure, you could make up a story about how I’m justifying myself to an imaginary audience or something, but it does not feel like one would have predicted the Civ example in advance from the model “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”.
Another class of examples: very often in social situations, the move which will actually get one the most points is to admit fault and apologize. And yet, instead of that, people instinctively spin a story about how they didn’t really do anything wrong. People instinctively spin that story even when it’s pretty damn obvious (if one actually stops to consider it) that apologizing would result in a better outcome for the person in question. Again, you could maybe make up some story about evolving suboptimal heuristics, but this just isn’t the behavior one would predict in advance from the model “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”.
A pattern with these examples (and many others): motivated reasoning isn’t mainly about fooling others, it’s about fooling oneself. Or at least a part of oneself. Indeed, there’s plenty of standard wisdom along those lines: “the easiest person to fool is yourself”, etc.
Here’s a model which I think much better matches real-world motivated reasoning. (Note, however, that all the above critique still stands regardless of whether this next model is correct.)
Motivated reasoning simply isn’t adaptive. Even in the ancestral environment, motivated reasoning decreased fitness. It appeared in the first place as an accidental side-effect of an overall-beneficial change in human minds relative to earlier minds, and that change was recent enough that evolution hasn’t had time to fix the anti-adaptive side effects.
There’s more than one hypothesis for what that change could be. Probably something to do with some particular functions within the mind being separated into parts, so that one part of the mind can now sometimes “cheat” by trying to trick another part of the mind, but the separation of those two functions is still overall beneficial.
An example falsifiable prediction of this model: other animals generally do not motivatedly-reason. If the relevant machinery had been around for very long, we would have expected evolution to fix the problem.
Tutoring.
This answer generated by considering what one can usefully hire other people to do full time, for oneself (or one’s family) alone.
Just got around to reading this, and I found it helpful. Thank you.
This was a cool post! I was familiar with f-divergences as a generalization of KL divergence, and of course familiar with maxent methods, but I hadn’t seen the two put together before.
Problem: I’m left unsure when or why I would ever want to use this machinery. Sufficient statistics are an intuitive concept, and I can look around at the world and make guesses about where I would want to use sufficient statistics to model things. Independence is also an intuitive concept, and I can look around the world and make guesses about where to use that. Combining those two, I can notice places in the world where (some version of) KPD should bind, and if it doesn’t bind then I’m surprised.
But I don’t see how to notice places where max-f-divergence distributions should bind to reality. I can see where sufficient statistics should exist in the world, but that’s not a sufficient condition for a max-f-divergence distribution; there are (IIUC) lots of other distributions for which sufficient statistics exist. So why focus on the max-f-divergence class specifically? What intuitive property of (some) real-world systems nails down that class of distributions? Maybe some kind of convexity condition on the distribution or on updates or something?
A practical roadblock is that the above numerical results for inference are terribly slow to compute...
Not sure exactly what you’re doing numerically, but here’s how I usually handle vanilla maxent problems numerically (in my usual notation, which is not quite the same as yours, apologies for putting marginal translation work on you). We start with
maxent subject to
Transform that to
maxent subject to
This gives a dual problem for the partition function, which I’ll write in full:
min
That’s an unconstrained optimization problem, it’s convex in the happy direction, and standard optimizers (e.g. scipy using jax for derivatives) can usually handle it very quickly.
I would guess that process generalizes just fine to other f-divergences, since it’s basically just relying on convex optimization tricks. And that should yield quite fast numerics.
A barrier for : suppose X and Y are both bitstrings of length 2k. The first k bits of the two strings are equal (i.e. X[:k] == Y[:k] in python notation); the rest are independent. Otherwise, all bits are maxentropic (i.e. IID 50⁄50 coinflips).
Then there’s an exact (deterministic) natural latent: := X[:k] = Y[:k]. But I(X; Y), H(X|Y), and H(Y|X) are all much larger than zero; each is k bits.
Maybe twice a year I go looking for this comment and can’t find it, so I’m copying it into shortform:
Oh, I can just give you a class of nontrivial predictions of expected utility theory. I have not seen any empirical results on whether these actually hold, so consider them advance predictions.
So, a bacteria needs a handful of different metabolic resources—most obviously energy (i.e. ATP), but also amino acids, membrane lipids, etc. And often bacteria can produce some metabolic resources via multiple different paths, including cyclical paths—e.g. it’s useful to be able to turn A into B but also B into A, because sometimes the environment will have lots of B and other times it will have lots of A. Now, there’s the obvious prediction that the bacteria won’t waste energy turning B into A and then back into B again—i.e. it will suppress one of those two pathways (assuming the cycle is energy-burning), depending on which metabolite is more abundant. Utility generalizes this idea to arbitrarily many reactions and products, and predicts that at any given time we can assign some (non-unique) “values” to each metabolite (including energy carriers), such that any reaction whose reactants have more total “value” than its products is suppressed (or at least not catalyzed; the cell doesn’t really have good ways to suppress spontaneous reactions other than putting things in separate compartments).
Of course in practice this will be an approximation, and there may be occasional exceptions where the cell is doing something the model doesn’t capture. If we were to do this sort of analysis in a signalling network rather than a metabolic network, for instance, there would likely be many exceptions: cells sometimes burn energy to maintain a concentration at a specific level, or to respond quickly to changes, and this particular model doesn’t capture the “value” of information-content in signals; we’d have to extend our value-function in order for the utility framework to capture that. But for metabolic networks, I expect that to mostly not be an issue.
That’s really just utility theory; expected utility theory would involve an organism storing some resources over time (like e.g. fat). Then we’d expect to be able to assign “values” such that the relative “value” assigned to stored resources which are not currently used is a weighted sum of the “values” assigned to those resources in different possible future environments (of the sort the organism might find itself in after something like its current environment, in the ancestral world), and the weights in the sums should be consistent. (This is a less-fleshed-out prediction than the other one, but hopefully it’s enough of a sketch to give the idea.)
Of course, if we understand expected utility theory deeply, then these predictions are quite trivial; they’re just saying that organisms won’t make pareto-suboptimal use of their resources! It’s one of those predictions where, if it’s false, then we’ve probably discovered something interesting—most likely some place where an organism is spending resources to do something useful which we haven’t understood yet. [EDIT-TO-ADD: This is itself intended as a falsifiable prediction—if we go look at an anomaly and don’t find any unaccounted-for phenomenon, then that’s a very big strike against expected utility theory.] And that’s the really cool prediction here: it gives us a tool to uncover unknown-unknowns in our understanding of a cell’s behavior.
I think one thing I didn’t communicate in the post is that I don’t necessarily intend to hypothesize deep nonconsent as a terminal preference. So, for instance,
women are scared men will get angry if they go from “yes” to “no”, in a way they won’t if the woman goes from “----” to “no”, so women delay being explicit until they have all the information
sounds to me like one of many possible generators of deep nonconsent preference—i.e. it’s directly explaining why women would typically have a deep-in-the-sense-of-appearing-in-lots-of-places preference for nonconsent behavior. It therefore sounds not-at-all at odds with the post, or at least what I had in mind when writing the post.
“Another I played with was e.g. “blame avoidance”, i.e. something-like-ladybrain really wants any dating/sex to happen in a way which is “not her fault”. That seems to mostly generate the same predictions.”
Do you think it has some disadvantage, such that you didn’t choose to mention it at all in the OP?
“Blame avoidance” seems like a candidate generator of deep nonconsent preference: if one never consents to anything that’s going on, then one is not to blame for any of it (or so goes the story). There are other generators one could imagine as well—e.g. Elizabeth hypothesized elsethread ‘women are scared men will get angry if they go from “yes” to “no”, in a way they won’t if the woman goes from “----” to “no”, so women delay being explicit until they have all the information’. That’s another hypothesis for what might generate deep nonconsent preference.
I settled on the term “deep nonconsent preference” because that seemed like the most direct description of the behavior-cluster, while assuming the least about what generates that behavior. I did not think (and still don’t think) I had enough information to nail down a primary generator of the behavior.
Can you gesture at what kind of data would be helpful to bring in-frame?
So, there’s this thing called Solomonoff induction. It works, provably, for anything Turing computable. And human social behavior is definitely Turing computable.
“If a theory claims to compactly generate any significant set of social dynamics, that’s evidence against the theory” is an anti-inductive prior. It’s like saying that things which have happened less often before are more likely in the future, and therefore the sun will certainly not rise tomorrow.
Look, I don’t like dealing with the sort of stuff I called “deep nonconsent” in this post. Sure, I’m quite kinky in bed, but in the rest of the mating process? When someone who’s interested won’t send any goddamn signals, or sends negative signals while hoping that I pursue, it’s just incredibly obnoxious. I strongly prefer to deal with women who actually send signals when interested, or better yet just ask me out. I want to date and fuck women who are, like, “on my team”, not trying to make everything pointlessly difficult all the time.
And maybe that will change at some point. It’s the sort of thing which sometimes seems less obnoxious as one understands it better. But man, for now, I sure prefer to just avoid women who do that shit.
Like, okay, let’s put it this way—if it were to turn out to have been true the entire time, what other generator could produce this evidence that also would produce evidence incompatible with this model? Or, in what way could “nonconsent” be missing the point about the generator? I’d sure like to see a slightly more ladybrain discussion, if that’s available.
I totally agree that there are other possible generators which look very similar to “deep nonconsent preference”. Another I played with was e.g. “blame avoidance”, i.e. something-like-ladybrain really wants any dating/sex to happen in a way which is “not her fault”. That seems to mostly generate the same predictions.
So yeah, I am totally ready to believe there’s some other nearby generator, and if you have one which also better explains some additional things then please state it I want to know it. I have not found it on my own, and one of the main points of posting this stuff online is that sometimes people come along and tell me what I’m missing. That’s what I want. If you have clean examples where the model in the post would produce incorrect interpretations of what’s going on, I also want to hear those. What I don’t want is people being like “this is problematic and missing important things” without actually saying a single thing that it’s wrong about or presenting any alternative model.
I went back and forth a bit on whether “nonconsent” or “anticonsent” was the right term to use for the bulk of this post. I’ve encountered women whose preferences look like one, and other women whose preferences look like the other, including at earlier parts of the pipeline (inconveniently). For instance, wanting a guy to pursue hard even after he’s turned down seems not-uncommon, despite it being obviously very obnoxious in cases where it’s actually unwanted. That would be an earlier-stage version of “anticonsent”, as opposed to “nonconsent/aconsent”.
I ended up going with “nonconsent” because it’s a broad enough term to capture both, and I think both are relevant.
That was a great post.
An important thing which I think is missing here: natural abstraction binds mainly to the Model, not the Critic.
Insofar as the natural abstraction hypothesis is true at all, it’s about structure which appears in the environment. That structure gets convergently mirrored in models of that environment. The Critic component is not optimized to be a model of the environment; it’s optimized to be a model of the reward signal. Insofar as the natural abstraction hypothesis binds to the Critic, it would be relevant in two ways:
The Critic largely inherits the Model’s ontology, because that’s the ontology it’s forced to use (architecturally) when querying the Model. This is roughly the Pointers thing: values take natural abstractions as inputs.
Insofar as the reward signal has a natural abstract structure, the Critic may learn that structure. But typically reward signals are a lot less rich than sensor signals (reward is a single scalar! Sensors include e.g. a whole image!), so there’s a lot less room for learning rich abstractions.
This is also an instance of the more general pattern that epistemics are a lot more constrained than values—think e.g. the orthogonality thesis.
Yes. In fact, IIRC only B depends on the choice of UTM, A can be a universal constant.
“On Green” is one of the LessWrong essays which I most often refer to in my own thoughts (along with “Deep Atheism”, which I think of as a partner to “On Green”). Many essays I absorb into my thinking, metabolize the contents, but don’t often think of the essay itself. But this one is different. The essay itself has stuck in my head as the canonical pointer to… well, Green, or whatever one wants to call the thing Joe is gesturing at.
When I first read the essay, my main thought was “Wow, Joe did a really good job pointing to a thing that I do not like. Screw Green. And thank you Joe for pointing so well at this thing I do not like.”. And when the essay has come up in my thoughts, it’s mostly been along similar lines: I encounter something in the wild, I’m like “ugh screw this thing… wait, why am I all ‘ugh’ at this, and why do other people like it?… oh it’s Carlsmith!Green, I see”.
But rereading the essay today, I’m struck by different thoughts than a year ago.
Green = Oxytocin Feeling?
Let’s start with the headline gesturing:
Today, reading that, I think “oxytocin?”. That association probably isn’t obvious to most of you; oxytocin is mostly known as the hormone associated with companionate love, which looks sort of like wholesomeness and happy villages but not so much like nature. So let me fill in the gap here.
Most people feel companionate love toward their family and (often metaphorical) village. Some people—usually more hippie-ish types—report feeling the same thing towards all of humanity. Even more hippie-ish types go further, feeling the same thing toward animals, or all living things, or the universe as a whole. Generally, the phrase which cues me in is usually “deep connection”—when people report feeling a deep connection to something, that usually seems to cash out to the oxytocin-feeling.
So when I read the post, I think “hmm, maybe Joe is basically gesturing at the oxytocin-feeling, i.e. companionate love”. That sure does at least rhyme with a lot of other pieces of the post.
… and that sure would explain why I get a big ol’ “ugh” toward Green; I already knew I’m probably genetically unable to feel oxytocin. To me, this whole Green thing is a cluster of stuff which seem basically-unremarkable in their own right, but for some reason other people go all happy-spiral about the stuff and throw away a bunch of their other values to get more Green.
Is Green instrumentally worthwhile?
That brings us to the central topic of the essay: is there a case for more Green which doesn’t require me to already value Green in its own right? Given that I don’t feel oxytocin, would it nonetheless be instrumental to the goals I do have to embrace a bit more Green?
My knee-jerk reaction to deep atheism is basically “yeah deep atheism is just straightforwardly correct”. When Joe says “Green, on its face, seems like one of the main mistakes” I’m like “yes, Green is one of the main mistakes, it still looks like that under its face too, it’s just straightforwardly a mistake”. It indeed looks like a conflation between is and ought, like somebody’s oxytocin-packed value system is trying to mess with their epistemics, like they’re believing that the world is warm and fuzzy and caring because they have oxytocin-feelings toward the world, not because the world is actually helping them.
… but one could propose that normal human brains, with proper oxytocin function, are properly calibrated about how much to trust the world. Perhaps my lack of oxytocin biases me in a way which gives me a factually-incorrect depth of atheism.
But that proposal does not stand up to cursory examination. The human brain has pretty decent general purpose capability for figuring out instrumental value; why would that machinery be wrong on the specific cluster of stuff oxytocin touches? It would be really bizarre if this one apparent reward-system hack corrects an epistemic error in a basically-working general purpose reasoning machine. It makes much more sense that babies and families just aren’t that instrumentally appealing, so evolution dropped oxytocin into the reward system to make babies and families very appealing as a terminal value. And that hack then generalized in weird ways, with people sometimes loving trees or the universe, because that’s what happens when one drops a hacky patch into a reward function.
The upshot of all that is… this Green stuff really is a terminal-ish value. And y’know, if that’s what people want, I’m not going to argue with the utility function (as the saying goes). But I am pretty darn skeptical of attempts to argue instrumental necessity of Green, and I do note that these are not my values.