Introducing the Instrumental Rationality Sequence
What is this project?
I am going to be writing a new sequence of articles on instrumental rationality. The end goal is to have a compiled ebook of all the essays, so the articles themselves are intended to be chapters in the finalized book. There will also be pictures.
I intend for the majority of the articles to be backed by somewhat rigorous research, similar in quality to Planning 101 (with perhaps a few less citations). Broadly speaking, the plan is to introduce a topic, summarize the research on it, give some models and mechanisms, and finish off with some techniques to leverage the models.
The rest of the sequence will be interspersed with general essays on dealing with these concepts, similar to In Defense of the Obvious. Lastly, there will be a few experimental essays on my attempt to synthesize existing models into useful-but-likely-wrong models of my own, like Attractor Theory.
I will likely also recycle / cannibalize some of my older writings for this new project, but I obviously won’t post the repeated material here again as new stuff.
What topics will I cover?
Here is a broad overview of the three main topics I hope to go over:
(Ordering is not set.)
Overconfidence in Planning: I’ll be stealing stuff from Planning 101 and rewrite a bit for clarity, so not much will be changed. I’ll likely add more on the actual models of how overconfidence creeps into our plans.
Motivation: I’ll try to go over procrastination, akrasia, and behavioral economics (hyperbolic discounting, decision instability, precommitment, etc.)
Habituation: This will try to cover what habits are, conditioning, incentives, and ways to take the above areas and habituate them, i.e. actually putting instrumental rationality techniques into practice.
Other areas I may want to cover:
Assorted Object-Level Things: The Boring Advice Repository has a whole bunch of assorted ways to improve life that I think might be useful to reiterate in some fashion.
Aversions and Ugh Fields: I don’t know too much about these things from a domain knowledge perspective, but it’s my impression that being able to debug these sorts of internal sticky situations is a very powerful skill. If I were to write this section, I’d try to focus on Focusing and some assorted S1/S2 communication things. And maybe also epistemics.
Ultimately, the point here isn’t to offer polished rationality techniques people can immediately apply, but rather to give people an overview of the relevant fields with enough techniques that they get the hang of what it means to start making their own rationality.
Why am I doing this?
Niche Role: On LessWrong, there currently doesn’t appear to be a good in-depth series on instrumental rationality. Rationality: From AI to Zombies seems very strong for giving people a worldview that enables things like deeper analysis, but it leans very much into the epistemic side of things.
It’s my opinion that, aside from perhaps Nate Soares’s series on Replacing Guilt (which I would be somewhat hesitant to recommend to everyone), there is no in-depth repository/sequence that ties together these ideas of motivation, planning, procrastination, etc.
Granted, there have been many excellent posts here on several areas, but they’ve been fairly directed. Luke’s stuff on beating procrastination, for example, is fantastic. I’m aiming for a broader overview that hits the current models and research on different things.
I think this means that creating this sequence could add a lot of value, especially to people trying to create their own techniques.
Open-Sourcing Rationality: It’s clear that work is being done on furthering rationality by groups like Leverage and CFAR. However, for various reasons, the work they do is not always available to the public. I’d like to give people who are interested but unable to directly work with these organization something they can use to jump start their own investigations.
I’d like this to become a similar Schelling Point that we could direct people to if they want to get started.
I don’t meant to imply that what I’ll produce is the same caliber, but I do think it makes sense to have some sort of pipeline to get rationalists up to speed with the areas that (in my mind) tie into figuring out instrumental rationality. When I first began looking into this field, there was a lot of information that was scattered in many places.
I’d like to create something cohesive that people can point to when newcomers want to get started with instrumental rationality that similarly gives them a high level overview of the many tools at their disposal.
Revitalizing LessWrong: It’s my impression that independent essays on instrumental rationality have slowed over the years. (But also, as I mentioned above, this doesn’t mean stuff hasn’t happened. CFAR’s been hard at work iterating their own techniques, for example.) As LW 2.0 is being talked about, this seems like an opportune time to provide some new content and help with our reorientation towards LW becoming once again a discussion hub for rationality.
Where does LW fit in?
Crowd-sourcing Content: I fully expect that many other people will have fantastic ideas that they want to contribute. I think that’s a good idea. Given some basic things like formatting / roughly consistent writing style throughout, I think it’d be great if other potential writers see this post as an invitation to start thinking about things they’d like to write / research about instrumental rationality.
Feedback: I’ll be doing all this writing on a public Google Doc with posts that feature chapters once they’re done, so hopefully there’s ample room to improve and take in constructive criticism. Feedback on LW is often high-quality, and I expect that to definitely improve what I will be writing.
Other Help: I probably can’t come through every single research paper out there, so if you see relevant information I didn’t or want to help with the research process, let me know! Likewise, if you think there are other cool ways you can contribute, feel free to either send me a PM or leave a comment below.
Why am I the best person to do this?
I’m probably not the best person to be doing this project, obviously.
But, as a student, I have a lot of time on my hands, and time appears to be a major limiting reactant in this whole process.
Additionally, I’ve been somewhat involved with CFAR, so I have some mental models about their flavor of instrumental rationality; I hope this translates into meaning I’m writing about stuff that isn’t just a direct rehash of their workshop content.
Lastly, I’m very excited about this project, so you can expect me to put in about 10,000 words (~40 pages) before I take some minor breaks to reset. My short-term goals (for the next month) will be on note-taking and finding research for habits, specifically, and outlining more of the sequence.
On the Replication Crisis and examining research:
My knowledge of statistics and research is fairly elementary. I know what the different tests / conditions are, so I can do some minor discrimination, but does anyone know anything about figuring out which things are real? (i.e. what to look for when seeing if research papers check out)
My current understanding / heuristics look like:
Try to analyze the experiment yourself.
Look for multiple studies that try to confirm the phenomenon.
Look for meta-analyses of the thing.
Check on OSF if replication has happened.
Priming, power stances, willpower-as-a-resource, and everything in Cialdini is on shaky ground.
It’s difficult—you are basically asking for a well-calibrated bullshit detector and those mostly get calibrated by wading through a lot of bullshit and paying careful attention to it X-)
Andrew Gelman, a stats professor at Columbia, has a sideline in snickering at bad papers, so scanning his blog might give you ideas about good approaches. See e.g. this.
Thanks! I’ve bumped into Gelman’s stuff a few times. I’ll check it out.
I Ankified Influence—Cialdini.
How wrong am I now?
Also, check out Keith Stanovich’s book How to think straight about Psychology. I am a rationality noob and find it amazing, but maybe it will help you because Keith talks about how to properly interpret scientific results in the later section of the book, AFAIK.
Gwern wrote a post about how the replication crisis means that the kind of methods that produced a lot of what’s in Cialdini’s books likely shoudn’t be trusted. The post wasn’t about specific arguments that indicated that Cialdini’s claims have been shown to be wrong. If you follow Gwern’s view than it’s unknown how wrong you are.
I’m not sure to what extent I follow that argument. A principle like scarcity is strongly reduced by many sleasy online marketers who do a lot of multivariate testing. If it wouldn’t work, they wouldn’t use it.
When it comes to detecting bullshit there’s more than just looking at the statistics. It’s very worthwhile to ask yourself whether the thesis actually makes sense based on what you know to be true based on how you live your life.
If you look at willpower-as-a-resource-that’s-glucose, it’s quite easy to see problems with that thesis. The paleo folks that try to minimize their blood glucose levels don’t tend to suffer from low willpower. The brain doesn’t consume more glucose when it’s “mentally active” than when it doesn’t. It doesn’t really fit the experience I have with how willpower works in myself.
Willpower also has the problem that it’s like Chi. Having it in the ontology can be useful for some task but there are no willpower particles or atoms just like there are no Chi particles or atoms. Baumeister et al seem to forget that willpower is just an abstraction.
Before the replication crisis, I was talking with a psychology Ph.D. about the thesis you believed on it while I didn’t. A few years later I meet him at an LW event and he told me that I was right all along. I wasn’t right because I was better at understanding the underlying statistics but because I thought about how the finding relates to other things that are true.
It might be worthwhile to read Baumeister’s book to get a feeling of how it looks like when an eminent psychologist defends a wrong thesis in the 21st century.
I find this hard to believe.
Cialdini? I’m finishing “Influence” right now. I was extra skeptical during reading it since I’m freshly acquainted with the replication crisis, but googling each citation and reading through the paper is way too much work. He supports many of his claims with multiple studies and real-life anecdotes (for all that’s worth). Could you point me to the criticism of Cialdini you have read?
Cialdini is based off a comment I think I saw by Scott Alexander along the lines of “everything in Cialdini now seems to be bunk”. This is low confidence and I’m happy to revise in light of new info.
My priors on Cialdini are mainly based on how priming, which seems similar to many of his claims, doesn’t replicate well.
That is indeed very low weight. My prior is pretty shaky as-is, but that evidence shouldn’t move it much.
I thought about priming a lot while reading. Many of the results he lists are similar to priming, but priming being false doesn’t mean all results similar to it are false. One could consider a broader hypothesis encompassing all that, namely “humans can be influenced by subtle clues to their subconsciousness to a significant degree”. That’s the similarity I see with priming, both it and many of Caldini’s hypothesis follow from this premise. The priming failure would suggest it’s false, but those experiments used extremely subtle subliminal clues, as if they were designed not to work. Much of Caldini’s work affirms this broader thesis. It’s no metastudy, but the guy lists a lot of studies, all affirming it. A lot of Kahneman’s work does, too. Surely it is acceptable that humans often act on instinct (unconsciously) and that they are subconsciously influenced by their surroundings. This follows from System 1 being so prevalent in our thought.
SSC has a new open thread right now, I should ask there. Maybe Scott can clear it up.
Meta: please don’t override the font and other styling of this site.
I’m not-too-familiar w/ the LW editor, and I don’t quite know how to remove my original styling.
If I’m copying-pasting in, is the only work-around to paste in plain text and apply styling afterwards?
Yes, paste as text and use markdown for formatting.
If you feel like you don’t know much about Focusing, I would advocate that you work to understand it deeply before writing an article about it.
Try to lead other people through the process. If a person has an uncomfortable feeling at the beginning of the process and that feeling is released at the end of the process.
There’s a lot of inferential distance for many rationalists for terms like felt sense, felt shift and resonate. I would be happy to proof read any post on Focusing that tries to explain them to this audience.
That sounds good. Thanks for the offer!
Upvoted! The instrumental side of things appeals to me a lot more and having more content/an actual sequence developed for it is exciting!
Cool! Anything you’re especially interested in seeing? I trust my judgment / discretion in finding relevant topics to an extent, but I really do want to know what else the community might like to see articles on (in the instrumental sphere.)
Comment here if there are specific areas / sub-topics you’d like for me to try and cover:
Focus on covering areas that you feel you understand the best. It’s better to write something really great even if a bit fewer people are interested in the topic than writing another mediocre article on a topic in which there’s a lot of interest.
Seems good. Thanks for the feedback!