I love that book! I like Robin’s essays, too, but the book was much easier for me to understand. I wish more people would read it, would review it on here, etc.
AnnaSalamon
(I don’t necessarily agree with QC’s interpretation of what was going on as people talked about “agency”—I empathize some, but empathize also with e.g. Kaj’s comment in a reply that Kaj doesn’t recognize this at from Kaj’s 2018 CFAR mentorship training, did not find pressures there to coerce particular kinds of thinking).
My point in quoting this is more like: if people don’t have much wanting of their own, and are immersed in an ambient culture that has opinions on what they should “want,” experiences such as QC’s seem sorta like the thing to expect. Which is at least a bit corroborated by QC reporting it.
-section on other ways to get inside opponent’s loop, not just speed—“more inconspicuously, more quickly, and with more irregularity” as Boyd said
this sounds interesting
-personal examples from video games: Stormgate and Friends vs. Friends
I want these
the OODA loop is not as linear as this model presents
I think the steps always go in order, but also there are many OODA loops running simultaneously
In the Observe step, one gathers information about the situation around them. In Boyd’s original context of fighter aircraft operations, we can imagine a pilot looking out the canopy, checking instruments, listening to radio communications, etc.
Gotcha. I’d assumed “observe” was more like “hear a crashing noise from the kitchen”—a kinda-automatic process that triggers the person to re-take-things-in and re-orient. Is that wrong?
Some partial responses (speaking only for myself):
1. If humans are mostly a kludge of impulses, including the humans you are training, then… what exactly are you hoping to empower using “rationality training”? I mean, what wants-or-whatever will they act on after your training? What about your “rationality training” will lead them to take actions as though they want things? What will the results be?
1b. To illustrate what I mean: once I taught a rationality technique to SPARC high schoolers (probably the first year of SPARC, not sure; I was young and naive). Once of the steps in the process involved picking a goal. After walking them through all the steps, I asked for examples of how it had gone, and was surprised to find that almost all of them had picked such goals as “start my homework earlier, instead of successfully getting it done at the last minute and doing recreational math meanwhile”… which I’m pretty sure was not their goal in any wholesome sense, but was more like ambient words floating around that they had some social allegiance to. I worry that if you “teach” “rationality” to adults who do not have wants, without properly noticing that they don’t have wants, you set them up to be better-hijacked by the local memeset (and to better camouflage themselves as “really caring about AI risk” or whatever) in ways that won’t do anybody any good because the words that are taking the place of wants don’t have enough intelligence/depth/wisdom in them.
2. My guess is that the degree of not-wanting that is seen among many members of the professional and managerial classes in today’s anglosphere is more extreme than the historical normal, on some dimensions. I think this partially because:
a. IME, my friends and I as 8-year-olds had more wanting than I see in CFAR participants a lot of the time. My friends were kids who happened to live on the same street as me growing up, so probably pretty normal. We did have more free time than typical adults.
i. I partially mean: we would’ve reported wanting things more often, and an observer with normal empathy would on my best guess have been like “yes it does seem like these kids wish they could go out and play 4-square” or whatever. (Like, wanting you can feel in your body as you watch someone, as with a dog who really wants a bone or something).
ii. I also mean: we tinkered, toward figuring out the things we wanted (e.g. rigging the rules different ways to try to make the 4-square game work in a way that was fun for kids of mixed ages, by figuring out laxer rules for the younger ones), and we had fun doing it. (It’s harder to claim this is different from the adults, but, like, it was fun and spontaneous and not because we were trying to mimic virtue; it was also this way when we saved up for toys we wanted. I agree this point may not be super persuasive though.)
b. IME, a lot of people act more like they/we want things when on a multi-day camping trip without phones/internet/work. (Maybe like Critch’s post about allowing oneself to get bored?)
c. I myself have had periods of wanting things, and have had periods of long, bleached-out not-really-wanting-things-but-acting-pretty-”agentically”-anyway. Burnout, I guess, though with all my CFAR techniques and such I could be pretty agentic-looking while quite burnt out. The latter looks to me more like the worlds a lot of people today seem to me to be in, partly from talking to them about it, though people vary of course and hard to know.
d. I have a theoretical model in which there are supposed to be cycles of yang and then yin, of goal-seeking effort and then finding the goal has become no-longer-compelling and resting / getting board / similar until a new goal comes along that is more compelling. CFAR/AIRCS participants and similar people today seem to me to often try to stop this process—people caffeinate, try to work full days, try to have goals all the time and make progress all the time, and on a large scale there’s efforts to mess with the currency to prevent economic slumps. I think there’s a pattern to where good goals/wanting come from that isn’t much respected. I also think there’s a lot of memes trying to hijack people, and a lot of memetic control structures that get upset when members of the professional and managerial classes think/talk/want without filtering their thoughts carefully through “will this be okay-looking” filters.
All of the above leaves me with a belief that the kinds of not-wanting we see are more “living human animals stuck in a matrix that leaves them very little slack to recover and have normal wants, with most of their ‘conversation’ and ‘attempts to acquire rationality techniques’ being hijacked by the matrix they’re in rather than being earnest contact with the living animals inside” and less “this is simple ignorance from critters who’re just barely figuring out intelligence but who will follow their hearts better and better as you give them more tools.”
Apologies for how I’m probably not making much sense; happy to try other formats.
I’m trying to build my own art of rationality training, and I’ve started talking to various CFAR instructors about their experiences – things that might be important for me to know but which hadn’t been written up nicely before.
Perhaps off topic here, but I want to make sure you have my biggest update if you’re gonna try to build your own art of rationality training.
It is, basically: if you want actual good to result from your efforts, it is crucial to build from and enable consciousness and caring, rather than to try to mimic their functionality.
If you’re willing, I’d be quite into being interviewed about this one point for a whole post of this format, or for a whole dialog, or to talking about it with you in some other way, means, since I don’t know how to say it well and I think it’s crucial. But, to babble:
Let’s take math education as an analogy. There’s stuff you can figure out about numbers, and how to do things with numbers, when you understand what you’re doing. (e.g., I remember figuring out as a kid, in a blinding flash about rectangles, why 2*3 was 3*2, why it would always work). And other people can take these things you can figure out, and package them as symbol-manipulation rules that others can use to “get the same results” without the accompanying insights. But… it still isn’t the same things as understanding, and it won’t get your students the same kind of ability to build new math or to have discernment about which math is any good.
Humans are automatically strategic sometimes. Maybe not all the way, but a lot more deeply than we are in “far-mode” contexts. For example, if you take almost anybody and put them in a situation where they sufficiently badly need to pee, they will become strategic about how to find a restroom. We are all capable of wanting sometimes, and we are a lot closer to strategic at such times.
My original method of proceeding in CFAR, and some other staff members’ methods also, was something like:
Find a person, such as Richard Feynman or Elon Musk or someone a bit less cool than that but still very cool who is willing to let me interview them. Try to figure out what mental processes they use.
Turn these mental processes into known, described procedures that system two / far-mode can invoke on purpose, even when the vicera do not care about a given so-called “goal.”
(For example, we taught processes such as: “notice whether you viscerally expect to achieve your goal. If you don’t, ask why not, solve that problem, and iterate until you have a plan that you do viscerally anticipate will succeed.” (aka inner sim / murphyjitsu.))
My current take is that this is no good—it teaches non-conscious processes how to imitate some of the powers of consciousness, but in a way that lacks its full discernment, and that can lead to relatively capable non-conscious, non-caring processes doing a thing that no one who was actually awake-and-caring would want to do. (And can make it harder for conscious, caring, but ignorant processes, such as youths, to tell the difference between conscious/caring intent, and memetically hijacked processes in the thrall of institutional-preservation-forces or similar.) I think it’s crucial to more like start by helping wanting/caring/consciousness to become free and to become in charge. (An Allan Bloom quote that captures some but not all of what I have in mind: “There is no real education that does not respond to felt need. All else is trifling display.”)
I’m not Critch, but to speak my own defense of the numeracy/scope sensitivity point:
IMO, one of the hallmarks of a conscious process is that it can take different actions in different circumstances (in a useful fashion), rather than simply doing things the way that process does it (following its own habits, personality, etc.). (“When the facts change, I change my mind [and actions]; what do you do, sir?”)
Numeracy / scope sensitivity is involved in, and maybe required for, the ability to do this deeply (to change actions all the way up to one’s entire life, when moved by a thing worth being moved by there).
Smaller-scale examples of scope sensitivity, such as noticing that a thing is wasting several minutes of your day each day and taking inconvenient, non-default action to fix it, can help build this power.
I am pretty far from having fully solved this problem myself, but I think I’m better at this than most people, so I’ll offer my thoughts.
My suggestion is to not attempt to “figure out goals and what to want,” but to “figure out blockers that are making it hard to have things to want, and solve those blockers, and wait to let things emerge.”
Some things this can look like:
- Critch’s “boredom for healing from burnout” procedures. Critch has some blog posts recommending boredom (and resting until quite bored) as a method for recovering one’s ability to have wants after burnout:
Physically cleaning things out. David Allen recommends cleaning out one’s literal garage (or, for those of us who don’t have one, I’d suggest one’s literal room, closet, inbox, etc.) so as to have many pieces of “stuck goal” that can resolve and leave more space in one’s mind/heart (e.g., finding an old library book from a city you don’t live in anymore, and either returning it anyhow somehow, or giving up on it and donating it to goodwill or whatever, thus freeing up whatever part of your psyche was still stuck in that goal).
Refusing that which does not “spark joy.” Marie Kondo suggests getting in touch with a thing you want your house to be like (e.g., by looking through magazines and daydreaming about your desired vibe/life), and then throwing out whatever does not “spark joy”, after thanking those objects for their service thus far.
Analogously, a friend of mine has spent the last several months refusing all requests to which they are not a “hell yes,” basically to get in touch with their ability to be a “hell yes” to things.
Repeatedly asking one’s viscera “would there be anything wrong with just not doing this?”. I’ve personally gotten a fair bit of mileage from repeatedly dropping my goals and seeing if they regenerate. For example, I would sit down at my desk, would notice at some point that I was trying to “do work” instead of to actually accomplish anything, and then I would vividly imagine simply ceasing work for the week, and would ask my viscera if there would be any trouble with that or if it would in fact be chill to simply go to the park and stare at clouds or whatever. Generally I would get back some concrete answer my viscera cared about, such as “no! then there won’t be any food at the upcoming workshop, which would be terrible,” whereupon I could take that as a goal (“okay, new plan: I have an hour of chance to do actual work before becoming unable to do work for the rest of the week; I should let my goal of making sure there’s food at the workshop come out through my fingertips and let me contact the caterers” or whatever.
Gendlin’s “Focusing.” For me and at least some others I’ve watched, doing this procedure (which is easier with a skilled partner/facilitator—consider the sessions or classes here if you’re fairly new to Focusing and want to learn it well) is reliably useful for clearing out the barriers to wanting, if I do it regularly (once every week or two) for some period of time.
Grieving in general. Not sure how to operationalize this one. But allowing despair to be processed, and to leave my current conceptions of myself and of my identity and plans, is sort of the connecting thread through all of the above imo. Letting go of that which I no longer believe in.
I think the above works much better in contact also with something beautiful or worth believing in, which for me can mean walking in nature, reading good books of any sort, having contact with people who are alive and not despairing, etc.
Okay, maybe? But I’ve also often been “real into that” in the sense that it resolves a dissonance in my ego-structure-or-something, or in the ego-structure-analog of CFAR or some other group-level structure I’ve been trying to defend, and I’ve been more into “so you don’t get to claim I should do things differently” than into whether my so-called “goal” would work. Cf “people don’t seem to want things.”
. The specific operation that happened was applying ooda loops to the concept of ooda loops.
I love this!
Surprise 4: How much people didn’t seem to want things
And, the degree to which people wanted things was even more incoherent than I thought. I thought people wanted things but didn’t know how to pursue them.
[I think Critch trailed off here, but implication seemed to be “basically people just didn’t want things in the first place”]
I concur. From my current POV, this is the key observation that should’ve, and should still, instigate a basic attempt to model what humans actually are and what is actually up in today’s humans. It’s too basic a confusion/surprise to respond to by patching the symptoms without understanding what’s underneath.
I also quite appreciate the interview as a whole; thanks, Raemon and Critch!
I’m curious to hear how you arrived at the conclusion that a belief is a prediction.
I got this in part from Eliezer’s post Make your beliefs pay rent in anticipated experiences. IMO, this premise (that beliefs should try to be predictions, and should try to be accurate predictions) is one of the cornerstones that LessWrong has been based on.
I love this post. (Somehow only just read it.)
My fav part:
> In the context of quantilization, we apply limited steam to projects to protect ourselves from Goodhart. “Full steam” is classically rational, but we do not always want that. We might even conjecture that we never want that.To elaborate a bit:
It seems to me that when I let projects pull me insofar as they pull me, and when I find a thing that is interesting enough that it naturally “gains steam” in my head, it somehow increases the extent to which I am locally immune from Goodhardt (e.g., my actions/writing goes deeper than I might’ve expected). OTOH, when I try hard on a thing despite losing steam as I do it, I am more subject to Goodhardt (e.g., I complete something with the same keywords and external checksums as I thought I needed to hit, but it has less use and less depth than I might’ve expected given that).I want better models of this.
Oh, man, yes, I hadn’t seen that post before and it is an awesome post and concept. I think maybe “believing in”s, and prediction-market-like structures of believing-ins, are my attempt to model how Steam gets allocated.
So, I agree there’s something in common—Wittgenstein is interested in “language games” that have function without having literal truth-about-predictions, and “believing in”s are games played with language that have function and that do not map onto literal truth-about-predictions. And I appreciate the link in to the literature.
The main difference between what I’m going for here, and at least this summary of Wittgenstein (I haven’t read Wittgenstein and may well be shortchanging him and you) is that I’m trying to argue that “believing in”s pay a specific kind of rent—they endorse particular projects capable of taking investment, they claim the speaker will themself invest resources in that project, they predict that that project will get yield ROI.
Like: anticipations (wordless expectations, that lead to surprise / not-surprise) are a thing animals do by default, that works pretty well and doesn’t get all that buggy. Humans expand on this by allowing sentences such as “objects in Earth’s gravity accelerate at a rate of 9.8m/s^2,” which… pays rent in anticipated experience in a way that “Wulky Wilkisen is a post-utopian” doesn’t, in Eliezer’s example. I’m hoping to cleave off, here, a different set of sentences that are also not like “Wulky Wilkinsen is a post-utopian” and that pay a different and well-defined kind of rent.
A point I didn’t get to very clearly in the OP, that I’ll throw into the comments:
When shared endeavors are complicated, it often makes sense for them to coordinate internally via a shared set of ~“beliefs”, for much the same reason that organisms acquire beliefs in the first place (rather than simply learning lots of stimulus-response patterns or something).
This sometimes makes it useful for various collaborators in a project to act on a common set of “as if beliefs,” that are not their own individual beliefs.
I gave an example of this in the OP:
If my various timeslices are collaborating in writing a single email, it’s useful to somehow hold in mind, as a target, a single coherent notion of how I want to trade off between quality-of-email and time-cost-of-writing. Otherwise I leave value on the table.
The above was an example within me, across my subagents. But there are also examples held across sets of people, e.g. how much a given project needs money vs insights on problem X vs data on puzzle Y, and what the cruxes are that’ll let us update about that, and so on.
A “believing in” is basically a set of ~beliefs that some portion of your effort-or-other-resources is invested in taking as a premise, that usually differ from your base-level beliefs.
(Except, sometimes people coordinate more easily via things that’re more like goals or deontologies or whatever, and English uses the phrase “believing in” for marking investment in a set of ~beliefs, or in a set of ~goals, or in a set of ~deontologies.)
I made a new post just now, “Believing In,” which offers a different account of some of the above phenomena.
My current take is that my old concept of “narrative syncing” describes the behaviorist outside of a pattern of relating that pops up a lot, but doesn’t describe the earnest inside that that pattern is kind of designed around.
(I still think “narrative syncing” is often done without an earnest inside, by people keeping words around an old icon after the icon has lost its original earnest meaning (e.g., to manipulate others), so I still want a term for that part; I, weirdly, do not often think using the term “narrative syncing,” it doesn’t quite do it for me, not sure what would. Some term that is to “believing in” as lying/deceiving is to “beliefs/predictions”.)
Totally. Yes.