AnnaSalamon

Karma: 19,842

AnnaSalamon 11 Dec 2025 0:22 UTC
12 points
2
in reply to: TsviBT’s comment on: Eliezer’s Unteachable Methods of Sanity
A further wrinkle / another example is that a question like “what should I think about (in particular, what to gather information about / update about)”, during the design process, wants these predictions.
Yes; this (or something similar) is why I suspect that “‘believing in’ atoms” may involve the same cognitive structure as “‘believing in’ this bakery I am helping to create” or “‘believing in’ honesty” (and a different cognitive structure, at least for ideal minds, from predictions about outside events). The question of whether to “believe in” atoms can be a question of whether to invest in building out and maintaining/tuning an ontology that includes atoms.

AnnaSalamon 10 Dec 2025 15:33 UTC
19 points
7
in reply to: Eliezer Yudkowsky’s comment on: Eliezer’s Unteachable Methods of Sanity
Prediction and planning remain incredibly distinct as structures of cognitive work,
I disagree. (Partially.) For a unitary agent who is working with a small number of possible hypotheses (e.g., 3), and a small number of possible actions, I agree with your quoted sentence.
But let’s say you’re dealing with a space of possible actions that’s much too large to let you consider each exhaustively, e.g. what blog post to write (considered concretely, as a long string of characters).
It’d be nice to have some way to consider recombinable pieces, e.g. “my blog post could include idea X”, “my blog post could open with joke J”, “my blog post could be aimed at a reader similar to Alice”.
Now consider the situation as seen by the line of thinking that is determining: “should my blog post be aimed mostly at readers similar to Alice, or at readers similar to Bob?”. For this line of thinking to do a good estimate of ExpectedUtility(post is aimed at Alice), it needs predictions about whether the post will contain idea X. However, for the line of thinking that is determining whether to include idea X (or the unified agent, at those moments when it is actively considering this), it’’ll of course need good plans (not predictions) about whether to include X, and how exactly to include X.
I don’t fully know what a good structure is for navigating this sort of recombinable plan space, but it might involve a lot of toggling between “this is a planning question, from the inside: shall I include X?” and “this is a prediction question, from the outside: is it likely that I’m going to end up including X, such that I should plan other things around that assumption?”.
My own cognition seems to me to toggle many combinatorial pieces back and forth between planning-from-the-inside and predicting-from-the-outside, like this. I agree with your point that human brains and bodies have all kinds of silly entanglements. But this part seems to me like a plausible way for other intelligences to evolve/grow too, not a purely one-off humans idiosyncrasy like having childbirth through the hips.

AnnaSalamon 14 Oct 2025 2:13 UTC
11 points
17
in reply to: cata’s comment on: What is Lesswrong good for?
Also we understand basic arithmetic around here, which goes a long way sometimes.

AnnaSalamon 13 Oct 2025 4:39 UTC
4 points
0
in reply to: ChristianKl’s comment on: Ethical Design Patterns
It’s a good point, re: some of the gap being that it’s hard to concretely visualize the world in which AGI isn’t built. And also about the “we” being part of the lack of concreteness.
I suspect there’re lots of kinds of ethical heuristics that’re supposed to interweave, and that some are supposed to be more like “checksums” (indicators everyone can use in an embodied way to see whether there’s a problem, even though they don’t say how to address it if there is a problem), and others are supposed to be more concrete.
For some more traditional examples:
- There’re heuristics for how to tell whether a person or organization is of bad character (even though these heuristics don’t tell how how to respond if a person is of bad character). Eg JK Rowling’s character Sirius’s claim that you can see the measure of a person by how they treat their house-elves (which has classical Christian antecedents, I’m just mentioning a contemporary phrasing).
- There’re heuristics for how countries should be, e.g. “should have freedom of speech and press” or (longer ago) “should have a monarch who inherited legitimately.”
It would be too hard to try to equip humans and human groups for changing circumstances via only a “here’s what you do in situation X”. It’s somewhat easier to do it (and traditional ethical heuristics did do it) by a combination of “you can probably do well by [various what-to-do heuristics]” and “you can tell if you’re doing well by [various other checksum-type heuristics]. Ethics is help to let us design our way to better plans, not to only always give us those plans.

AnnaSalamon 13 Oct 2025 4:19 UTC
2 points
0
in reply to: AnnaSalamon’s comment on: Adele Lopez’s Shortform
(Nevermind, after thinking about it a bit more I think I get it.)

AnnaSalamon 13 Oct 2025 3:53 UTC
6 points
0
in reply to: Adele Lopez’s comment on: Adele Lopez’s Shortform
Another place where I’ll think and act somewhat differently as a result of this conversation:
- It’s now higher on my priority list to try to make sure CFAR doesn’t act as a “gateway” to all kinds of weird “mental techniques” (or quasi-cults who use “mental techniques”). Both for CFAR’s new alumni, and for social contacts of CFAR’s new alumni. (This was already on some lists I’d made, but seeing Adele derive it independently bumped it higher for me.)
What links here?
- AnnaSalamon's comment on Adele Lopez’s Shortform by Adele Lopez (13 Oct 2025 4:19 UTC; 2 points)

AnnaSalamon 13 Oct 2025 3:36 UTC
2 points
0
in reply to: Wei Dai’s comment on: Ethical Design Patterns
Okay, but: it’s also find individuals who are willing to speak for heuristic C, in a way I suspect differs from what it was like for leaded gasoline and from what I remember as a kid in the late 80′s about the ozone layer.
It’s a fair point that I shouldn’t expect “consensus”, and should’ve written and conceptualized that part differently, but I think heuristic C is also colliding with competing ethical heuristics in ways the ozone situation didn’t.

AnnaSalamon 13 Oct 2025 1:26 UTC
8 points
0
in reply to: Adele Lopez’s comment on: Adele Lopez’s Shortform
I listed the cases I could easily list of full-blown manic/psychotic episodes in the extended bay area rationalist community (episodes strong enough that the person in most cases ended up hospitalized, and in all cases ended up having extremely false beliefs about their immediate surroundings for days or longer, eg “that’s the room of death, if I walk in there I’ll die”; “this is my car” (said of the neighbor’s car)).
I counted 11 cases. (I expect I’m forgetting some, and that there are others I plain never knew about; count this as a convenience sample, not an exhaustive inventory.)
Of these, 5 are known to me to have involved a psychedelic or pot in the precipitating event.
3 are known to me to have *not* involved that.
In the other 3 cases I’m unsure.
In 1 of the cases where I’m unsure about whether there were drugs involved, the person had taken part in a several-weeks experiment in polyphasic sleep as part of a Leverage internship, which seemed to be part of the precipitating event from my POV.
So I’m counting [between 6 and 8] out of 11 for “precipitated by drugs or an imprudent extended sleep-deprivation experiment” and [between 3 and 5] out of 11 for “not precipitated by doing anything unusually physiologically risky.”
(I’m not here counting other serious mental health events, but there were also many of those in the several-thousand-person community across the last ten years, including several suicides; I’m not trying here to be exhaustive.)
(Things can have multiple causes, and having an obvious precipitating physiological cause doesn’t mean there weren’t other changeable risk factors also at play.)

AnnaSalamon 12 Oct 2025 22:29 UTC
2 points
0
in reply to: plex’s comment on: Adele Lopez’s Shortform
Your guess above, plus: the person’s “main/egoic part”, who has have mastered far-mode reasoning and the rationalist/Bayesian toolkit, and who is out to “listen patiently to the dumb near-mode parts that foolishly want to do things other than save the world,” can in some people, with social “support” from outside them, help those parts to overpower other bits of the psyche in ways that’re more like tricking and less like “tug of wars”, without realizing they’re doing this.

AnnaSalamon 12 Oct 2025 21:27 UTC
8 points
7
in reply to: AnnaSalamon’s comment on: Adele Lopez’s Shortform
My own guesses are that CFAR mostly paid an [amount of attention that made sense] to reducing psychosis/mania risks in the workshop context, after our initial bad experience with the mania/psychosis episode at an early workshop when we did not yet realize this could be a thing.
The things we did:
- tried to screen for instablity;
- tried to warn people who we thought might have some risk factors (but not enough risk factors that we were screening them out) after accepting them to the workshop, and before they’d had a chance to say yes. (We’d standardly say something like: “we don’t ask questions this nosy, and you’re already in regardless, but, just so you know, there’s some evidence that workshops of all sorts, probably including CFAR workshops, may increase risks of mania or psychosis in people with vulnerability to that, so if you have any sort of psychiatric history you may want to consider either not coming, or talking about it with a psychiatrist before coming.”)
- try to train our instructors and “mentors” (curriculum volunteers) to notice warning signs. check in as a staff regularly to see if anyone had noticed any warning signs for any participants. if sensible, talk to the participant to encourage them to sleep more, skip classes, avoid recreational drugs for awhile, do normal grounding activities, etc. (This happened relatively often — maybe once every three workshops — but was usually a relatively minor matter. Eg this would be a person who was having trouble sleeping and who perhaps thought they had a chance at solving [some long-standing personal problem they’d previously given up on] “right now” a way that weirded us out, but who also seemed pretty normal and reasonable still.)
I separately think I put a reasonable amount of effort into organizing basic community support and first aid for those who were socially contiguous with me/CFAR who were having acutely bad mental health times, although my own capacities weren’t enough for a growing community and I mostly gave up on the less near-me parts around 2018.
It mostly did not occur to me to contemplate our cultural impact on the community’s overall psychosis rate (except for trying for awhile to discourage tulpas and other risky practices, and to discourage associating with people who did such things, and then giving up on this around 2018 when it seemed to me there was no real remaining chance of quarantining these practices).
I like the line of inquiry about “what art of rationality might be both good in itself, and increase peoples’ robustness / decrease their vulnerability to mania/psychosis-type failure modes, including much milder versions that may be fairly common in these parts and that are still bad”. I’ll be pursuing it. I take your point that I could in principle have pursued it earlier.
If we are going to be doing a fault analysis in which we give me and CFAR responsibility for some of our downstream memetic effects, I’d like CFAR to also get some credit for any good downstream memetic effects we had. My own guess is that CFAR workshops:
- made it possible for EA and “the rationalist community” to expand a great deal without becoming nearly as “diluted”/“normie” as would’ve happened by default, with that level of immigration-per-year;
- helped many “straw lesswrongers” to become more “agenty” and realize “problems are for solving” instead of sort of staring helplessly at their todo lists and desires, and that this part made the rationalist community stronger and healthier
- helped a fair number of people to become less “straw EA” in the sense of “my only duty is to do the greatest good for the greatest number, while ignoring my feelings”, and to tune in a bit more to some of the basics of healthy life, sometimes.
I acknowledge that these alleged benefits are my personal guesses and may be wrong. But these guesses seem on par to me with my personal guess that patterns of messing with one’s own functioning (as from “CFAR techniques”) can erode psychological wholeness, and I’m afraid it’ll be confusing if I voice only the negative parts of my personal guesses.

AnnaSalamon 12 Oct 2025 20:40 UTC
2 points
0
in reply to: Adele Lopez’s comment on: Adele Lopez’s Shortform
I’m most worried about 2, I think there’s potentially something toxic about the framing of “rationality habits” in general, which has previously led to a culture of there being all these rationality “tricks” that would solve all your problems … which in turn leads to people uncritically trying dubious techniques that fuck them up.
Could you say a bit more here, please?
(not a direct response, but:) My belief has been that there are loads of people in the bay area doing dubious things that mess them up (eg tulpas, drugs, weird sex things, weird cult things—both in the rationalist diaspora, and in the bay area broadly), but this is mostly people aiming to be edgy and do “weird/cool/powerful” things, not people trying CFAR techniques as such.

AnnaSalamon 12 Oct 2025 19:27 UTC
2 points
−2
in reply to: plex’s comment on: Adele Lopez’s Shortform
though IDC feels similar flavoured and is an original.
Awkwardly, while IDC is indeed similar-flavored and original to CFAR, I eventually campaigned (successfully) to get it out of our workshops because I believe, based on multiple anecdotes, that IDC tends to produce less health rather than more, especially if used frequently. AWC believes Focusing should only be used for dialog between a part and the whole (the “Self”), and I now believe she is correct there.

AnnaSalamon 12 Oct 2025 19:23 UTC
18 points
0
in reply to: Adele Lopez’s comment on: Adele Lopez’s Shortform
I’d like to talk a bit about the sense in which the rationalist community does or doesn’t have “people in positions of leadership”, and how this compares to eg an LDS ward (per Adele’s comparison). I’m unfortunately not sure how to be brief here, but I’d appreciate thoughts anyway from those who have them, because, as CFAR and I re-enter the public space, I am unsure what role to try to occupy exactly, and I am also unsure how to accurately communicate what roles I am and am not willing to be in (so as to not cause others to inaccurately believe I’ll catch things).
(This discussion isn’t directly to do with psychosis; but it bears on Adele’s questions about what CFAR leadership or other rationality community leaders are responsible for, and what to predict from us, and what would be good here.)
On my understanding, church parishes, and some other traditional communities, often have people who intentionally:
1. are taken as a role model by many, especially young people;
2. try to act in such a way that it’ll be fine for people to imitate them;
3. try to care for the well-being of the community as a whole (“is our parish healthy? what small nudges might make us a little healthier or more thriving? what minor trouble-spots are beginning, that might ease up if I or another volunteer heads over and listens and tries to help everyone act well?”).
a/b/c are here their primary duty: they attend to the community for its own sake, as a calling and public role. (Maybe they also have a day job, but a/b/c are primary while they are doing their parish duties, at least.)
Relatedly, they are trusted by many in the community, and many in the community will follow their requests, partly because their requests make sense: “So-and-so had a death in the family recently, and Martha is organizing meals for them; please see Martha if you’re willing to provide some meals.” So they are coordinating a larger effort (that many, many contribute to) to keep the parish healthy and whole. (Also relatedly: the parish is already fairly healthy, and a large majority of those in it would like it to be healthier, and sees this as a matter of small nudges rather than large upsets or revolutions.)
Eliezer is clearly *not* in a position of community leadership in this sense. His primary focus (even during those hours in which he is interacting with the rationalist community) is not the rationalist community’s health or wholeness, but is rather AI risk. He does make occasional posts to try to nudge the community toward health and wholeness, but overall his interactions with us are not those of someone who is trying to be a role model or community-tender/leader.
My guess is that almost nobody, perhaps actually nobody, sees themself as in a position of community leadership in this sense. (Or maybe Oliver and Lightcone do? I am not sure and would be interested to hear from them here).
Complicating the issue is the question of who is/isn’t “in” “the” rationalist community that a given set of leaders is aiming to tend. I believe many of the bad mental health events have historically happened in the rationalist community’s periphery, in group houses with mostly unemployed people who lack the tethers or stablising influences that people employed by mainstream EA or rationalist organizations, or by eg Google, have. (My guess is that here there are even fewer “community leaders”; eg I doubt Oli and Lightcone see themselves as tenders of this space.)
(A quick google suggests that LDS wards typically have between 200 and 500 members, many of which I assume are organized into families; the bay area “rationalist-adjacent” community includes several thousand people, mostly not in families.)
In the early days of CFAR (2012-2017, basically), and in the earlier days of the 2009-2011 pre-CFAR rationality community, I felt some of this duty. I tried to track bad mental health events in the broader rationalist community and to help organize people to care for people who acutely needed caring for, where I could. This is how I came to spend 200+ hours on psychosis. My main felt duty wasn’t on community wholeness — it was on AI risk, and on recruiting for AI risk — but I felt a bit as though it was my backyard and I wanted my own backyard to be good, partly because I cared, partly because I thought people might expect it of me, partly because I thought people might blame/credit me for it.
I mostly quit doing this around 2018, due partly to Michael Vassar seeming to declare memetic war on me in a way I didn’t know how to deal with, partly to some parts of EA also saying that CFAR and I were bad in the wake of the Brent Dill fiasco, partly to “the community” having gotten huge in a way that was harder and harder for me to keep track of or to feel as much connection to, and partly to having less personal psychological slack for personal reasons.
(I’m not saying any of my motivations or actions here were good necessarily; I’m trying to be accurate.)
(TBC, I still felt responsible for the well-being of people at CFAR events, and of people in the immediate aftermath of CFAR events, just not for “the rationalist community” broadly. And I still tried to help when I found out about a bad situation where I thought I might have some traction, but I stopped proactively extending feelers of the sort that would put me in that situation.)
Another part of the puzzle is that Eliezer’s Sequences and HPMOR cast a huge “come here if you want to be meaningful; everything is meaningless except this work, and it’s happening here” narrative beacon, and many many came who nobody regarded themself as having ~any responsibility for, I think. (In contrast, EY’s and Nate’s recent book does not do this; it still says AI risks are important, but it actively doesn’t undermine peoples’ local meaning-making and lives.)
I and CFAR should probably figure out better what my and our roles will be going forward, and should try hard to *visibly* not take up more responsibility than we’re expecting to meet. I’m interested also in what our responsibilities are, here.
I’m currently keen on:
1. Actively attend to the well-being of those at our events, or those in the immediate aftermath of our events, where we can; and
2. Put some thought into which “rationality habits” or similar, if percolated out from CFAR’s workshops or for that matter from my LW posts and/or my actions broadly, will make the community healthier (eg, will reduce or at least not increase any local psychosis-prone-ness of these communities);
3. Put a little bit of listening-effort into understanding the broader state of the communities our participants come from, and return to, and spread our memes in, since this is necessary for (2). (Only a little, because it is hard and I am lazy.)
4. Don’t otherwise attempt to tend the various overlapping rationality or bay area rationality communities.
Thoughts appreciated.

AnnaSalamon 12 Oct 2025 17:17 UTC
6 points
2
in reply to: Viliam’s comment on: Adele Lopez’s Shortform
The surprising part is that you consider them a part of the rationalist community. What did they do to deserve this honor?
(Noting again that I’m speaking only of the pre-2020 situation, as I lack much recent info) Many don’t consider them part of “the” community. This is part of how they come to be not-helped by the more mainstream/healthy parts.
However: they are seeded by people who were deeply affected by Eliezer’s writing, and who wanted to matter for AI risk, and who grabbed some tools and practices from what you would regard as the rationality community, and who then showed their friends their “cool mind-tools” etc., with the memes evolving from there.
Also, it at least used to be that there was no crisp available boundary: one’s friends will sometimes have friendships that reach beyond, and so habits will move from what I’m calling the “periphery” into the “mainstream” and back.
The social puzzle faced by bay area rationalists is harder than that faced by eg Boston-area rationalists, owing mostly I think to the sheer size of the bay area rationality community.

AnnaSalamon 12 Oct 2025 2:33 UTC
4 points
0
in reply to: AnnaSalamon’s comment on: Adele Lopez’s Shortform
(I still quite like this idea on my second pass ~two weeks later; I guess I should try to interview people / observe people and see if I can figure out in detail what they are and aren’t doing here.)

AnnaSalamon 12 Oct 2025 2:30 UTC
10 points
2
in reply to: TsviBT’s comment on: Adele Lopez’s Shortform
I love this, yes. Straw rationalists believe we should update our beliefs ~instantly (even foundational ones, even ones where we’ve never seen someone functional believe it and so have no good structures to copy, such as “what if this is all a simulation with [particular purpose X]”), and don’t have an adequate model of, nor adequate respect for, the work involved in staying sane and whole through this process.

AnnaSalamon 12 Oct 2025 2:22 UTC
7 points
3
in reply to: Viliam’s comment on: Adele Lopez’s Shortform
I think there’s a broader property that makes people not-psychotic, that many things in the bay area and in the practice of “rationality” (not the ideal art, but the thing folks do) chip away at.
I believe the situation is worse among houses full of unemployed/underemployed people at the outskirts of the community than it is among people who work at central rationalist/EA/etc organizations or among people who could pay for a CFAR workshop. (At least, I believe this was so before covid; I’ve been mostly out of touch since leaving the bay in early 2020.)
This “broader property” is something like: “the world makes sense to me (on many levels: intuitive, emotional, cognitive, etc), and I have meaningful work that is mundane and full of feedback loops and that I can tell does useful things (eg I can tell that after I feed my dog he is fed), and many people are counting on me in mundane ways, and my friends will express surprise and check in with me if I start suddenly acting weird, and my rough models are in rough synchrony also with the social world around me and with the physical systems I am interacting with, and my friends are themselves sane and reasonable and oriented to my world such that it works fine for me to update off their opinions, and lots of different things offer useful checksums on lots of different aspects of my functioning in a non-totalizing fashion.”
I think there are ways of doing debate (even “where nothing is taboo”) that are relatively more supportive of this “broader property.” Eg, it seems helpful to me to spend some time naming common ground (“we disagree about X, and we’ll spend some time trying to convince each other of X/not-X, but regardless, here’s some neighboring things we agree about and are likely to keep agreeing about”). Also to notice that material reality has a lot of detail, and that there are many different questions and factors that may affect (AI or whatever) that don’t correlate that much with each other.

AnnaSalamon 12 Oct 2025 1:51 UTC
9 points
9
in reply to: plex’s comment on: Adele Lopez’s Shortform
I like this point, particularly the “controlling vs opening” bit. I believe I’ve seen this happen, in a fairly internally-grown way in people within the wider rationalist millieu. I believe I’ve also seen (mostly via hearsay, so, error bars) a more interpersonal “high stakes, therefore [tolerate bad/crazy things that someone else in the group claims has some chance at helping somehow with AI]” happen in several different quasi-cults on the outskirts of the rationalists.

Fear is part of where controlling (vs opening) dynamics come from, sometimes, I think. (In principle, one can have an intellectual stance of “there’s something precious that may be lost here” without the emotion of fear; it’s the emotion that I think inclines people toward the narrowing/controlling dynamic.) I also think there’s something in the notion that we should aspire toward being “Bayesian agents” that lends itself toward controlling dynamics (Joe Carlsmith gets at some of this in his excellent “Otherness and control in the age of AI” sequence, IMO.)
I agree Focusing helps some, when done well. (Occasionally it even helps dramatically.) It’s not just a CFAR thing; we got it from Gendlin, and his student Ann Weiser Cornell and her students are excellent at it, are unrelated to the rationalists, and offer sessions and courses that’re excellent IMO. I also think nature walks and/or exercise help some people, as does eg having a dog, doing concrete things that matter for other people even if they’re small, etc. Stuff that helps people regain a grounding in how to care about normal things.
I suspect also it would be good to have a better conceptual handle on the whole thing. (I tried with my Emergencies post, and it’s better than not having tried, but it … more like argued “here’s why it’s counterproductive to be in a controlling/panicky way about AI risk” and did not provide “here’s some actually accessible way to do something else”.)

AnnaSalamon 11 Oct 2025 23:54 UTC
13 points
0
on: CFAR update, and New CFAR workshops
Two CFAR applicants told me our emails (an acceptance email and a “here’s how you schedule a financial aid interview or ask further questions” email) landed in their spam folders. So, if you or your friend applied and are wondering why you haven’t heard from us, try checking your spam folder. Thanks!

AnnaSalamon 8 Oct 2025 23:59 UTC
2 points
0
in reply to: Drake Morrison’s comment on: CFAR update, and New CFAR workshops
Thanks! I’ll take a look.