I’m an EA who benefitted from rationality
This is my personal take, not an organizational one. Originally written May 2025, revived for the EA Forum’s Draft Amnesty Week. Cross-posted from the EA Forum but cross-posting didn’t work the first time.
When I discovered the beginnings of EA, it felt like coming home. I had finally found a group of people who cared as much as I did about saving lives.
When I discovered rationality not long after, it felt new. The intensity of LessWrong was both offputting and fascinating to me, and I read a lot of the site in the summer of 2011. I had studied sociology and was used to thinking in terms of ethics and emotions, but not in terms of statistics or economics. LessWrongers were picky and disagreeable, but I learned reasoning skills from them.
Before EA even had a name, LessWrong was one of the best places to talk about ideas now associated with EA. My first post in 2011 was about what would later be called earning to give.
I attended LessWrong meetups in Boston because it was fun to toss ideas around, but I hosted EA meetups because I wanted to strengthen a movement I believed was saving lives. After having kids I continued to prioritize EA, but I didn’t stay involved in rationality as a community.
It did continue to influence how I thought, though. If I wrote something that didn’t quite feel right, I’d imagine how a rationalist critic would respond. Often this steered me away from sloppy reasoning.
……
At their best, the two movements have in common a willingness to dig into weird possibilities and to take ideas seriously. How literally should you take expected value calculations? What if AI is much less predictable than we hope? Does this research study really say what the headlines report?
I’ve also heard plenty of criticisms of both sides:
EAs have adopted key ideas (existential risk, astronomical waste) from people in weird corners of the internet while holding their nose at those same people and spaces
Rationalists prioritize ideas and navel-gazing over real-world impact, and squander their weirdness points
EA cares too little for accuracy or what’s really important, and too much for image and respectability
Rationality tolerates or encourages repulsive ideas and behaviors
……
So how do these movements currently relate to each other? How should they relate?
Among individuals, there’s overlap, but most people who identify with one don’t identify with the other.
A quarter of respondents on the 2024 LessWrong survey identify as EAs.
7% of respondents on the 2024 EA Survey learned about EA via LessWrong; this is more than the combined number of respondents who came via GiveWell and Giving What We Can.
Some cities have basically no overlap between the two scenes, and others have a lot, especially the Bay Area. (In general I think the Bay Area is shaped by having an unusual number of subcultures that cross-pollinate each other.)
Personally, EA is my main focus, but rationality continues to be one of the spaces I draw value from. I want to stay open to good ideas and techniques from thinkers in the rationality space, even if I object to other ideas or actions from the same thinkers. Some ideas I’ve valued:
“Pulling the rope sideways,” popularizing prediction markets, and takes on how humans fool ourselves from Robin Hanson. I find his writing infuriating and illuminating by turns.
Goal factoring and other techniques from CFAR; I don’t endorse everything that ever came from their org, but they did spread some useful tools. I think goal factoring may in turn have been inspired by Geoff Anders’ goal mapping.
The fact that some people in EA and rationality get value from the other space doesn’t mean everyone has to do the same. If a space gives you the creeps, you don’t have to engage with it.
My best guess is that organizations and projects in EA and rationalist spaces are best off with some freedom from each other, so they can pursue projects in their different ways.
I’d refine this a little to: EA slips into missing crucial considerations somewhat more easily than Rationality.
To overly compress, it feels like EA puts Doing Good as the core motivation, then sees Truth-Seeking as a high priority to do good well. Rationality somewhat inverts this, with Truth-Seeking as the central virtue, and Doing Good (or Doing Things) as a high but not quite as central virtue.
My read is this leads some EA orgs to stay somewhat more stuck to approaches which look like they’re good but are very plausibly harmful because of a weird high context crucial consideration that would get called out somewhat more effectively in Rationalist circles.
Looks to me like the two movements are natural symbiotes, glad you wrote this post.
Pretty intriguing, sounds right although the only high-profile example that immediately comes to my mind is Nuno Sempere’s critical review of Open Phil’s bet on criminal justice reform. Any examples you find most salient?
OpenPhil’s technical alignment grantmaking missing a key step in the plan it looks like they’re trying. In particular the part where they’re banking on AI assisted alignment using alignment techniques that absolutely won’t scale to superintelligence, progress towards which you can buy from academics who don’t deeply understand the challenge and can assess with relative ease due to nice feedback loops.
But if you have ~human level weakly aligned system, you can’t ask it to solve alignment and get a result that’s safe without understanding the question you’re asking your system much better than we do currently. And that requires theory (technical philosophy/maths), which they’re underinvesting in to an extent that lots of promising people who would prefer to work on the critical bottleneck in their plan are actually doing things that they can tell are not going to matter, because that’s where the jobs are. I’ve met two people who’ve kinda burned out due to this, recently. And on the ground in the highest competence/strategic clarity circles I know, takes like this are pretty common.
Occupying as much space in the funding landscape as they do, it’s easy to imbalance the field and give others the impression that technical research is covered, and non empirical work being mostly missing from their portfolio is imo a crucial consideration so severe it plausibly flips the sign of their overall AI safety grantmaking work.
The fact that alignment and capabilities are so entangled makes this a lot worse. If you build more aligned but not deeply aligned systems, without the plan and progress you would need to use that to get a deeply aligned system, you end up shortening timelines. A system strong and aligned enough to make a best effort attempt to solve alignment is likely above the power and directability level required to trigger a software singularity and end the world. Building towards that before you have well formulated questions you want to ask the system which result in world saving is not good for p(doom).
I think this was mostly convergent development, rather than a clear lineage.
Though Geoff did teach a version of Goal Factoring that was much more like CT-charting at some early CFAR workshops.