Here is my brain dump: I have mostly given up on the Berkeley rationality community as a possible village. I think the people who showed up here were mostly selected for being bad at villaging, and that the awful shit that’s been happening around here lately is downstream of that. I think there is something toxic / dysfunctional woven deep into the community fabric (which has something to do with the ways in which the Mission interacts poorly with people’s psychologies) and I don’t feel very hopeful about even being able to show it clearly to half of the community, let alone doing anything about it.
In February I wrote a 20-page Google Doc describing what I think is wrong with Berkeley in more detail, which I’ve shared with some of you but don’t plan to make public. (Message me on Facebook if you’d like to request access to a PDF of it; I might not say yes, though.) I’d like to get around to writing a second public draft but again, I’ve been feeling less hopeful, so… we’ll see.
I upvoted this because it gave me some concepts to use to look at some experiences I’ve had. The speculations at the level of physical mechanism aren’t really cruxes for me so I mostly don’t care about them, and same with facts of the matter about what any particular Pali text actually says. What’s interesting to me is what Romeo gets out of a combination of reading them and reflecting on his own experience, that might be relevant to me reflecting on my own experience.
Why should I believe any of this?
Gut reaction to this question is that it’s the wrong question. I don’t view this post as telling you anything you’re supposed to believe on Romeo’s word.
It’s goodharting from the point of view of natural selection’s values but it doesn’t have to be goodharting from the point of view of your values. We can enjoy art even if art is in some sense goodharting on e.g. being in beautiful places or whatever.
This is fantastic and absolutely the conversation I want to be having. Resonates quite a lot with my experience, especially as a description of what it is exactly that I got out of circling.
In your language circling naturally stirs up sankharas because relational shit is happening (e.g. people are paying attention to you or ignoring you, liking or disliking what you say, etc) and then hopefully, if the circle is being well-facilitated, you sometimes get coached into a state where you can notice and work with your “causal links in the perceptual system between physical sensations, feelings, and mental reactions,” e.g. by the facilitator remaining very calm and holding space, then gently pointing out their observations about your causal links, that kind of thing. Unfortunately with less skilled facilitation this doesn’t happen and shit just gets stirred up and not resolved; worst case people get retraumatized.
Very excited to talk to you more about this.
Yeah, I agree that at the earlier stages it’s not clear that ambition is a thing to aim for, and I would also advise people to prioritize health broadly.
I agree that encouragement and guidance is good, and more generally think that mentorship is really, really deeply important. I am not about this “individual rationality” life anymore. It’s group rationality or nothing.
Right, this is the kind of thing I had in mind with the phrase “pathological need to do something.” Cf. people who are obsessed with making way more money than they could ever possibly spend.
He says that “Politics is an extension of war by other means. Arguments are soldiers”. I’d update this to say that “ONE WAY OF THINKING ABOUT politics is AS IF IT IS an extension of war by other means. Arguments CAN BE THOUGHT OF AS soldiers”.
This is a good shift for you to have made and I’m glad you can make it. Now you can just do this mentally to everyone’s writing (and speaking, for that matter) all the time.
But asking writers to do it themselves is crippling. The new sentence you propose is obviously technically more accurate but it’s also clunky, anemic prose.
Someone ages ago (maybe Viliam_Bur?) said that people thought they were drawn to LW by the good ideas but they were equally if not more drawn to LW by the quality of Eliezer’s writing, and a key quality of Eliezer’s writing is that he knew how to be punchy when he needed to be.
“Arguments are soldiers” is, and was always, poetry, not a mathematical identity. The point was to shock you into a frame shift in how you look at arguments (away from e.g. an implicit frame of “arguments are neutral tools we use to search for truth”), not tell you a Platonic Truth that you Believe Forever. And punchy writing is key to provoking these kinds of frame shifts; clunky writing just doesn’t actually get through to any part of you that matters.
Terence Tao is great; I haven’t read that book but I like his writing a lot in general. I am a big fan of the Princeton Lectures in Analysis by Stein and Shakarchi; clear writing, good exercises, great diagrams, focus on examples and applications.
(Edit: also, fun fact, Stein was Tao’s advisor.)
Epistemic status: ex-math grad student
I think you’re equivocating between two possible meanings of “choose” here. There’s “choose” as in you start telling people “I want to write a book” and then there’s “choose” as in you actually decide to actually write the book, which is quite different. I think Ray is asking about something like how to cultivate the capacity to do the latter. It is not at all trivially easy. Most goals are fake; making them real is a genuine skill.
Tongue-in-cheek: “when their pathological need to do something outweighs their pathological need to do nothing.”
In more detail: there are several different kinds of deep-rooted psychological needs that ambition might be powered by, and I think the resulting different kinds of ambition are different enough to discuss as distinct entities (in particular, they vary in how prosocial they are). Some possibilities off the top of my head, not mutually exclusive, inspired by Enneagram types:
1. Reinforce a particular identity / self-narrative (e.g. “I’m special” → strive to become a celebrity or w/e, see Instagram influencers); Enneagram 4
2. Get people to like you (again see Instagram influencers); Enneagram 3
3. Have power over people (some politicians, maybe); Enneagram 8
4. Have fun to avoid feeling bad; Enneagram 7
(One way to probe this in a given ambitious person is to look at what coping mechanisms they turn to when they fail. E.g. if it’s about reinforcing a given identity through some ambitious project, when that project fails do they start reinforcing that identity in other ways?)
Then there’s genuine compassion, which is the cleanest power source for ambition I’ve found so far, and arguably the most prosocial (there might be others, e.g. childlike joy and wonder). I am quite concerned that most of the ambition in the rationality / EA space is not being powered by genuine compassion; personally, most of the time I’ve been here I’ve been powered by a combination of #1 and #2.
There are also several different kinds of deep-rooted psychological needs that lack of ambition might be powered by. Again, some possibilities off the top of my head, inspired by Enneagram types:
1. Not drawing criticism / pissing people off; Enneagram 2, Enneagram 3, or Enneagram 9
2. Avoiding the feeling of not knowing what to do; Enneagram 5, Enneagram 6
3. Sense that ambition is morally wrong / corrupting; Enneagram 1
4. Sense that ambition is not your place / not the sort of thing people like you are allowed to do; Enneagram 2, Enneagram 3, Enneagram 4
Historically I think a lot of my lack of ambition was powered by a combination of #1, #2, and #4, although it’s hard to disentangle. There were also less psychological obstacles, e.g. I was tired all the time because I was eating, sleeping, and exercising poorly, and had an awful social life; real hard to be ambitious or agentic in that state.
To summarize, I mostly relate to ambition as a relatively surface-level psychological phenomenon that’s being powered by deeper dynamics, and I think at least as much in terms of obstacles to ambition as in terms of ways to cultivate ambition.
Epistemic status: based on lots of personal development work and looking at other people’s psychology and personal development, e.g. via circling; especially, noticing my own level of ambition increase drastically the more work I do on myself, and looking at what seem to be the gears of that.
I also don’t have much of this skill and made it through life without needing to have it; I was able to coast on raw intelligence for quite a long time, up through my 2nd year of grad school or so. Welp.
Except in romantic relationships; I’ve historically consistently found it easy to have commitment, follow-through, reliability, focused attention, etc. in that context (although it was kinda being fueled by neediness so there were other things going on there).
It feels like I have not yet found e.g. a job that I deeply value enough to commit to in the same way that I valued my relationships enough to commit to them, and that I can take my lack of commitment to various things I’ve attempted to do so far as evidence that at least some part of me didn’t find them worth committing to. I think I’m okay with having high standards for what I commit to in this way, although I might be out of practice committing as a result.
Yes, I strongly agree that this is missing and it sucks. I have a lot to say about why I think this is happening, which hopefully will be converted from a Google Doc into a series of blog posts soonish.
There’s an interesting thing authentic relating people do at workshops that they call “setting intentions,” and I think it works in a different way than either of these. The difference seems to me to be that the intention is being “held by the group.” I’m not sure how to explain what I mean by this. There are at least two visible signs of it:
1) people remind each other about the intention, and
2) people reward each other for following through on the intention.
If everyone knows the intention is being held by the group in this way, it both comes to mind more easily and feels more socially acceptable to follow through on (the latter might be causing the former). In my experience group intentions also require almost no willpower, but they also don’t feel quite like policies to me (that would be “agreements”) - they’re more like especially salient affordances.
The key ritual is that at some point someone asks “so, do we all agree to hold this intention?” and we raise our hands if so—and we look around the room so we can see each other’s hands. That way the collective holding of the intention can enter common knowledge.
Said another way, it’s something like trying to define the values of the group-mind we’re attempting to come together to form.
I relate to your resistance to willpower-based intentions. It’s something like, a lot of people have an “inner authoritarian” or “inner tyrant” that is the internalized avatar of other people making them do stuff when they were younger (parents, teachers, etc.), whose job it is to suppress the parts of them that are unacceptable according to outer tyrants. You can live under your inner tyrant’s reign of terror, which works as long as submitting to the inner tyrant keeps your life running smoothly, e.g. it placates your outer tyrants and they feed you and stuff.
At some point this strategy can stop working, and then other parts of you might engage in an internal rebellion against your inner tyrant; I think I was in a state like this for most of 2017 and 2018, and probably still am now to some extent. At this stage using willpower can feel like giving in to the inner tyrant.
Then I think there’s some further stage of development that involves developing “internal leadership,” whatever that is.
There’s a bit in the Guru Papers about this. One quote, I think in the context of submitting to renunciate morality (e.g. Christian morality):
Maintaining denial actually requires constant surveillance of the thing you are pretending isn’t there. This deepens the internal splits that renunciation promises to heal. It requires the construction of a covert inner authoritarian to keep control over the “bad” stuff you reject. This inner tyrant is probably not strong enough to do the job on its own, so you submit to an external authority whose job is to strengthen the internal tyrant.
Glad to see you’re writing about this! I think motivation is a really central topic and there’s lots more to be said about it than has been said so far around here.
When we’re struggling with motivation to do X, it’s because only S2 predicts/believes that X will lead to reward. S1 isn’t convinced. Your S2 models say X is good, but S1 models don’t see it. This isn’t necessarily a bug. S2 reasoning can be really shitty, people can come up with all kinds of dumb plans for things that won’t help, and it’s not so bad that their S1 models don’t go along with them.
I think these days S1 and S2 have become semantic stopsigns, and in general I recommend that people stop using these terms both in their thinking and verbally, and instead try to get more specific about what parts of their mind actually disagree and why. I can report, for example, that CFAR doesn’t use these terms internally.
Anna Salamon used to say, in the context of teaching internal double crux at CFAR workshops, that there’s no such thing as an S1 vs. S2 conflict. All conflicts are “S1 vs. S1.” “Your S2,” whatever that means, may be capable of engaging in logical reasoning and having explicit verbal models about things, but the part of you that cares about the output of all of that reasoning is a part of your S1 (in my internal dialect, just “a part of you”), and you’ll make more progress once you start identifying what part that is.
Here’s an example of what getting more specific might look like. Suppose I’m a high school student and “my S1” says play video games and “my S2″ says do my homework. What is actually going on here?
One version could be that I know I get social reinforcement from my parents and my teachers to do homework, or more sinisterly that I get socially punished for not doing it. So in this case “my S2” is a stopsign blocking an inquiry into the power structure of school, and generally the lack of power children have in modern western society, which is both harder and less comfortable to think about than “akrasia.”
Another version is someone told me to do my homework so I’ll go to a good college so I’ll get a good job. In this case “my S2” is a stopsign blocking an inquiry into why I care about any of the nodes in this causal diagram—maybe I want to go to a good college because it’ll make me feel better about myself, maybe I want to get a good job to avoid disappointing my parents, etc.
That’s on the S2 side, but “my S1” is also blocking inquiry. Why do I want to play video games? Not “my S1,” just me; I can own that desire. There are obvious stories to tell about video games being more immediately pleasurable and addictive than most other things I could do, and those stories have some weight, but they’re also a distraction; much easier to think about than why I wouldn’t rather do anything else. In my actual experience, the times in my life I have played video games the most, the reasons were mostly emotional: I was really depressed and lonely and felt like a failure, and video games (and lots of other stuff) distracted me from feeling those things. Those feelings were very painful to think about, and that pain prevented me from even looking at the structure of this problem, let alone debugging it, for a long time.
(One sign I was doing this is that the video games I chose were not optimized for pleasure. I deliberately avoided video games that could be fun in a challenging way, because I didn’t want to feel bad about doing poorly at them. Another sign is that everything else I did was also chosen for its ability to distract: for example, watching anime (never live-action TV, too uncomfortably close to real life), reading fiction (never nonfiction, again too uncomfortably close to real life), etc.)
An inference I take from this model is that you are best able to focus on long-term S2 goals (those which have the least inherent ability to influence you) if you have taken care of the rest of things which motivate you. Eat enough, sleep enough, spend time with friends. When you’re trying to fight the desire to address those things, you’re using willpower, and willpower is a stopgap measure.
Strongly agree, except that I wouldn’t use the term “S2 goals.” That’s a stopsign. Again I suggest getting more specific: what part of you has those goals and why? Where did they come from?
So part of what I need to do now is really figure out how to do green-brain, growth-orientation motivation over red-brain, deficit-reduction motivation.
If I understand correctly what you mean by this, I have a lot of thoughts about how to do this. The short, unsatisfying version, which will probably surprise no one, is “find out what you actually want by learning how to have feelings.”
The long version can be explained in terms of Internal Family Systems. The deal is that procrastinative behaviors like playing a lot of video games are evidence that you’re trying to avoid feeling a bad feeling, and that that bad feeling is being generated by a part of you that IFS calls an “exile.” Exiles are producing bad feelings in order to get you to avoid a catastrophic situation that resembles a catastrophic situation earlier in your life that you weren’t prepared for; for example, if you were overwhelmed by criticism from your parents as a child, you might have an exile that floods you with pain whenever people criticize you, especially people you really respect in a way that might cause you to project your parents onto them.
Exiles are paired with parts called protectors, whose job it is to protect exiles from being triggered. In the criticism example, that might look like avoiding people who criticize you, avoiding doing things you might get criticized for, or feeling sleepy or confused when someone manages to criticize you anyway.
Behavior that’s driven by exile / protector dynamics (approximately “red-brain, deficit-reduction,” if I understand you correctly) can become very dysfunctional, as the goal of avoiding psychological pain becomes a worse and worse proxy for avoiding bad situations. In extreme cases it can almost completely block your access to what you want, as that becomes less of a priority than avoiding pain. In the criticism example, you might be so paralyzed by the possibility that someone could criticize you for doing things that you stop doing anything.
There are lots of different ways to get exiles and protectors to chill the fuck out, and once they do you get to find out what you actually want when you aren’t just trying to avoid pain. It’s good times. See also my comment on Kaj’s IFS post.
This seems like a bad idea to me; I think people who are trying to have good ideas should develop courage instead. If you don’t have courage your ideas are being pushed around by fear in general, and asking for a particular source of that fear to be ameliorated will not solve the general problem.
Thanks for writing this up! I’m excited to see more people running experiments like this.
When you say “if I take X as a prompt, I get Y,” how many trials did you wait? In my own experimentation I’ve found lil’ GPT-2′s performance to be really variable across trials, and I’ve needed to wait 5 trials in some cases to get results I even sort of liked.
My sense overall of how lil’ GPT-2 functions after playing with it for awhile on several different kinds of prompts is that it has a strong sense of genre, and has done something like learned a bunch of different genre conventions for the different types of texts in the training set. If the prompt strongly resembles a genre familiar from training it will run with that genre, although sometimes it’ll wander off into another genre. It does quite poorly on prompts that I suspect don’t strongly match to a genre in the training set.
For example, I tried to run a Turing test (mostly as a joke) by prompting with “Question: what is your name? Answer:” and I got this on my ~2nd to 4th trial (don’t remember), with my speculations as to genre in [square brackets]:
I’m a Guardian Huntress, a deviation of the <Here> Monks keyword. I’m like a little “mechanic-creature hire”, but is Disassembler which is general clique-of-guy? I usually live in forms of Priests or Whispers of the Old Gods, but if it’s Asian Brotherhood of Magi..… [RP forum?]
Reply ~29000 0 ~13 min 0 By TrueBeliever For MA teenage adventurer: Ooops It looks stupid but I’m given the wrong answer, and this logic is corrupt. Other first hand studies: sometimes your famed mechanical fiends attack other nvrock’s—I had such too—but you’re supposed to have flying paths to defend them would not it? This alien spawn is being unnecessary and a waste of time Man—you stole the Saalis… [gaming forum?]
Reply ~120200 0 ~30 min 0 By john : I know there’s no better examples of
a landing back off 2lands ( ou ever?) Low challenge difficulty
a lot of new cards in my deck,
which doesn’t draw cards: stacks high levels Some need life if w/o rest of deck already Defense Emperor’s | Inferno/Neck Scorer Necronomicon Mysticpetal Mana Enigma Mortalswords Mysticmist Scout Zarthai Sniper [MtG or Hearthstone forum?]
It’s a multiplier and it increases the yield on a lv301 fighter fighter, next to nothing it’s probably never in my deck or some random deck Dofrone’s | Boltforge Fastboat Sling dmt3 Efreet Flattestalker Infernal Eater Toxic Water Hurricane For another Holy orb suggested… [gaming forum? LoL?]
Was planning on posting a longer (mostly humorous) post with my own results but that post is low priority so I don’t know when it’s going to happen.
Thanks for writing this! I am very excited that this post exists. I think what this model suggests about procrastination and addiction alone (namely, that they’re things that managers and firefighters are doing to protect exiles) are already huge, and resonate strongly with my experience.
In the beginning of 2018 I experienced a dramatic shift that I still don’t quite understand; my sense of it at the time was that there was this crippling fear / shame that had been preventing me from doing almost anything, that suddenly lifted (for several reasons, it’s a long story). That had many dramatic effects, and one of the most noticeable ones was that I almost completely stopped wanting to watch TV, read manga, play video games, or any of my other addiction / procrastination behaviors. It became very clear that the purpose of all of those behaviors was numbing and distraction (“general purpose feeling obliterators” used by firefighters, as waveman says in another comment) from how shitty I felt all the time, and after the shift I basically felt so good that I didn’t want or need to do that anymore.
(This lasted for awhile but not forever; I crashed hard in September (long story again) before experiencing a very similar shift again a few weeks ago.)
Another closely related effect is that many things that had been too scary for me to think about became thinkable (e.g. regrettable dynamics in my romantic relationships), and I think this is a crucial observation for the rationality project. When you have exile-manager-firefighter dynamics going on and you don’t know how to unblend from them, you cannot think clearly about anything that triggers the exile, and trying to make yourself do it anyway will generate tremendous internal resistance in one form or another (getting angry, getting bored, getting sleepy, getting confused, all sorts of crap), first from managers trying to block the thoughts and then from firefighters trying to distract you from the thoughts. Top priority is noticing that this is happening and then attending to the underlying emotional dynamics.
I like this reading and don’t have much of an objection to it.
This is a bad argument for transhumanism; it proves way too much. I’m a little surprised that this needs to be said.
Consider: “having food is good. Having more and tastier food is better. This is common sense. Transfoodism is the philosophy that we should take this common sense seriously, and have as much food as possible, as tasty as we can make it, even if doing so involves strange new technology.” But we tried that, and what happened was obesity, addiction, terrible things happening to our gut flora, etc. It is just blatantly false in general that having more of a good thing is better.
As for “common sense”: in many human societies it was “common sense” to own slaves, to beat your children, again etc. Today it’s “common sense” to circumcise male babies, to eat meat, to send people who commit petty crimes to jail, etc., to pick some examples of things that might be considered morally repugnant by future human societies. Common sense is mostly moral fashion, or if you prefer it’s mostly the memes that were most virulent when you were growing up, and it’s clearly unreliable as a guide to moral behavior in general.
Figuring out the right thing to do is hard, and it’s hard for comprehensible reasons. Value is complex and fragile; you were the one who told us that!
In the direction of what I actually believe: I think that there’s a huge difference between preventing a bad thing happening and making a good thing happen, e.g. I don’t consider preventing an IQ drop equivalent to raising IQ. The boy has had120 IQ his entire life and we want to preserve that, but the girl has had 110 IQ her entire life and we want to change that. Preserving and changing are different, and preserving vs. changing people in particular is morally complicated. Again the argument Eliezer uses here is bad and proves too much:
Either it’s better to have an IQ of 110 than 120, in which case we should strive to decrease IQs of 120 to 110. Or it’s better to have an IQ of 120 than 110, in which case we should raise the sister’s IQ if possible. As far as I can see, the obvious answer is the correct one.
Consider: “either it’s better to be male than female, in which case we should transition all women to men. Or it’s better to be female than male, in which case we should transition all men to women.”
What I can appreciate about this post is that it’s an attempt to puncture bad arguments against transhumanism, and if it had been written more explicitly to do that as opposed to presenting an argument for transhumanism, I wouldn’t have a problem with it.
This whole conversation makes me deeply uncomfortable. I expect to strongly disagree at pretty low levels with almost anyone else trying to have this conversation, I don’t know how to resolve those disagreements, and meanwhile I worry about people seriously advocating for positions that seem deeply confused to me and those positions spreading memetically.
For example: why do people think consciousness has anything to do with moral weight?