Why CFAR’s Mission?

Related to:

Briefly put, CFAR’s mission is to improve the sanity/​thinking skill of those who are most likely to actually usefully impact the world.
I’d like to explain what this mission means to me, and why I think a high-quality effort of this sort is essential, possible, and urgent.
I used a Q&A format (with imaginary Q’s) to keep things readable; I would also be very glad to Skype 1-on-1 if you’d like something about CFAR to make sense, as would Pete Michaud. You can schedule a conversation automatically with me or Pete.


Q: Why not focus exclusively on spreading altruism? Or else on “raising awareness” for some particular known cause?

Briefly put: because historical roads to hell have been powered in part by good intentions; because the contemporary world seems bottlenecked by its ability to figure out what to do and how to do it (i.e. by ideas/​creativity/​capacity) more than by folks’ willingness to sacrifice; and because rationality skill and epistemic hygiene seem like skills that may distinguish actually useful ideas from ineffective or harmful ones in a way that “good intentions” cannot.

Q: Even given the above—why focus extra on sanity, or true beliefs? Why not focus instead on, say, competence/​usefulness as the key determinant of how much do-gooding impact a motivated person can have? (Also, have you ever met a Less Wronger? I hear they are annoying and have lots of problems with “akrasia”, even while priding themselves on their high “epistemic” skills; and I know lots of people who seem “less rational” than Less Wrongers on some axes who would nevertheless be more useful in many jobs; is this “epistemic rationality” thingy actually the thing we need for this world-impact thingy?...)

This is an interesting one, IMO.

Basically, it seems to me that epistemic rationality, and skills for forming accurate explicit world-models, become more useful the more ambitious and confusing a problem one is tackling.

For example:

If I have one floor to sweep, it would be best to hire a person who has pre-existing skill at sweeping floors.

If I have 250 floors to sweep, it would be best to have someone energetic and perceptive, who will stick to the task, notice whether they are succeeding, and improve their efficiency over time. An “all round competent human being”, maybe.

If I have 10^25 floors to sweep, it would… be rather difficult to win at all, actually. But if I can win, it probably isn’t by harnessing my pre-existing skill at floor-sweeping, nor even (I’d claim) my pre-existing skill at “general human competence”. It’s probably by using the foundations of science and/​or politics to (somehow) create some totally crazy method of getting the floors swept (a process that would probably require actually accurate beliefs, and thus epistemic rationality).

The world’s most important problems look to me more like that third example. And, again, it seems to me that to solve problems of that sort—to iterate through many wrong guesses and somehow piece together an accurate model until one finds a workable pathway for doing what originally looked impossible—without getting stuck in dead ends or wrong turns or inborn or societal prejudices—it is damn helpful to have something like epistemic rationality. (Competence is pretty darn helpful too—it’s good to e.g. be able to go out there and get data; to be able to form networking relations with folks who already know things; etc. -- but epistemic rationality is necessary in a more fundamental way.)

For the sake of concreteness, I will claim that AI-related existential risk is among humanity’s most important problems, and that it is damn confusing, damn hard, and really really needs something like epistemic rationality and not just something like altruism and competence if one is to impact it positively, rather than just, say, randomly impacting it. I’d be glad to discuss in the comments.

Q: Why suppose “sanity skill” can be increased?

Let’s start with an easier question: why suppose thinking skills (of any sort) can be increased?

The answer to that one is easy: Because we see it done all the time.

The math student who arrives at college and does math for the first time with others is absorbing a kind of thinking skill; thus mathematicians discuss a person’s “mathematical maturity”, as a property distinct from (although related to) their learning of this and that math theorem.

Similarly, the coder who hacks her way through a bunch of software projects and learns several programming languages will have a much easier time learning her 8th language than she did her first; basically because, somewhere along the line, she learned to “think like a computer scientist”...

The claim that “sanity skill” is a type of thinking skill and that it can be increased is somewhat less obvious. I am personally convinced that the LW Sequences /​ AI to Zombies gave me something, and gave something similar to others I know, and that hanging out in person with Eliezer Yudkowsky, Michael Vassar, Carl Shulman, Nick Bostrom, and others gave me more of that same thing; a “same thing” that included e.g. actually trying to figure it out, making beliefs pay rent in anticipated experience; using arithmetic to entangle different pieces of my beliefs; and so on.

I similarly have the strong impression that e.g. Feynman’s and Munger’s popular writings often pass on pieces of this same thing; that the convergence between the LW Sequences and Tetlock’s Superforecasting training is non-coincidental; that the convergence between CFAR’s workshop contents and a typical MBA program’s contents is non-coincidental (though we were unaware of it when creating our initial draft); and more generally that there are many types of thinking skill that are routinely learned/​taught and that non-trivially aid the process of coming to accurate beliefs in tricky domains. I update toward this partly from the above convergences; from the fact that Tetlock’s training seems to work; from the fact that e.g. Feynman and Munger (and for that matter Thiel, Ray Dalio, Francis Bacon, and a number of others) were shockingly conventionally successful and advocated similar things; and from the fact that there is quite a bit of “sanity” advice that is obviously correct once stated, but that we don’t automatically do (advice like “bother to look at the data; and try to update if the data doesn’t match your predictions”).

So, yes, I suspect that there is some portion of sanity that can sometimes be learned and taught. And I suspect this portion can be increased further with work.

Q. Even if you can train skills: Why go through all the trouble and complications of trying to do this, rather than trying to find and recruit people who already have the skills?

The short version: because there don’t seem to be enough people out there with the relevant skills (yet), and because it does seem to be possible to help people increase their skills with training.
How can I tell there aren’t enough people out there, instead of supposing that we haven’t yet figured out how to find and recruit them?
Basically, because it seems to me that if people had really huge amounts of epistemic rationality + competence + caring, they would already be impacting these problems. Their huge amounts of epistemic rationality and competence would allow them to find a path to high impact; and their caring would compel them to do it.
I realize this is a somewhat odd thing to say, and may not seem true to most readers. It didn’t used to seem true to me. I myself found the idea of existential risk partly through luck, and so, being unable to picture thinkers who had skills I lacked, I felt that anyone would need luck to find out about existential risk. Recruiters, I figured, could fix that luck-gap by finding people and telling them about the risks.
However, there are folks who can reason their own way to the concept. And there are folks who, having noticed how much these issues matter, can locate novel paths that may plausibly impact human outcomes. And even if there weren’t folks of this sort, there would be hypothetical people of this sort; much of what is luck at some skill-levels can in principle be made into skill. So, finding that only a limited number of people are working effectively on these problems (provided one is right about that assessment) does offer an upper bound on how many people today can exceed a certain combined level of epistemic rationality, competence, and caring about the world.
And of course, we should also work to recruit competence and epistemic skill and caring wherever we can find them. (To help such folk find their highest impact path, whatever that turns out to be.) We just shouldn’t expect such recruiting to be enough, and shouldn’t stop there. Especially given the stakes. And given the things that may happen if e.g. political competence, “raised awareness”, and recruited good intentions are allowed outpace insight and good epistemic norms on e.g. AI-related existential risk, or other similarly confusing issues. Andrew Critch makes this point well with respect to AI safety in a recent post; I agree with basically everything he says there.
Q. Why do you sometimes talk of CFAR’s mission as “improving thinking skill”, and other times talk of improving all of epistemic rationality, competence, and do-gooding? Which one are you after?

The main goal is thinking skill. Specifically, thinking skill among those most likely to successfully use it to positively impact the world.

Competence and caring are relevant secondary goals: some of us have a conjecture that deep epistemic rationality can be useful for creating competence and caring, and of course competence and caring about the world are also directly useful for impacting the world’s problems. But CFAR wants to increase competence and caring via teaching relevant pieces of thinking skill, and not via special-case hacks. For example, we want to help people stay tuned into what they care about even when this is painful, and to help people notice their aversions and sort through which of their aversions are and aren’t based in accurate implicit models. We do not want to use random emotional appeals to boost specific cause areas, nor to use other special-case hacks that happen to boost efficacy in a manner opaque to participants.

Why focus primarily on thinking skill? Partly so we can have focus enough as an organization so as to actually do anything at all. (Organizations that try to accomplish several things at once risk accomplishing none—and “epistemic rationality” is more of a single thing.) Partly so our workshop participants and other learners can similarly have focus as learners. And partly because, as discussed above, it is very very hard to intervene in global affairs in such a way as to actually have positive outcomes, and not merely outcomes one pretends will be positive; and focusing on actual thinking skill seems like a better bet for problems as confusing as e.g. existential risk.

Why include competence and caring at all, then? Because high-performing humans make use of large portions of their minds (I think), and if we focus only on “accurate beliefs” in a narrow sense (e.g., doing analogs of Tetlocks forecasting training and nothing else), we are apt to generate “straw lesswrongers” whose “rationality” applies mainly to their explicit beliefs… people who can nitpick incorrect statements and can in this way attempt accurate verbal statements, but who are not creatively generative, do not have the iterated energy/​competence/​rapid iteration required to launch a startup, and cannot run good fast realtime social skills. We aim to do better. And we suspect that working to hit competence and caring via what one might call “deep epistemic rationality” is a route in.

Q. Can a small organization realistically do all that without losing Pomodoro virtue? (By “Pomodoro virtue”, I mean the ability to focus on one thing at a time and so to actually make progress, instead of losing oneself amidst the distraction of 20 goals.)

We think so, and we think the new core/​labs division within CFAR will help. Basically, Core will be working to scale up the workshops and related infrastructure, which should give a nice trackable set of numbers to optimize—numbers that, if grown, will enable better financial health for CFAR and will also enable a much larger set of people to train in rationality.

Labs will be focusing on impacting smaller numbers of people who are poised to impact existential risk (mainly), and on seeing whether our “impacts” on these folk do in fact seem to help with their impact on the world.

We will continue to work together on many projects and to trade ideas frequently, but I suspect that this separation into two goals will give more “pomorodo virtue” to the whole organization.

Q. What is CFAR’s relationship to existential risk? And what should it be?

CFAR’s mission is to improve the sanity/​thinking skill of those who are most likely to actually usefully impact the world—via whatever cause areas may be most important. Many of us suspect that AI-related existential risk is an area with huge potential for useful impact; and so we are focusing partly on helping meet talent gaps in that field. This focus also gives us more “pomodoro virtue”—it is easier to track whether e.g. the MIRI Summer Fellows Program helped boost research on AI safety, than it is to track whether a workshop had “good impacts on the world” in some more general sense.

It is important to us that the focus remain on “high impact pathways, whatever those turn out to be”, that we do not propagandize for particular pre-set answers (rather, that we assist folks in thinking things through in an unhindered way), and that we work toward a kind of thinking skill that may let people better assess what paths are actually high impact for having positive effects in the world, and to overcome flaws in our current thinking.

Q. Should I do “Earning to Give”? Also: I heard that there are big funders around now and so “earning to give” is no longer a sensible thing for most people to do; is that true? And what does all this have to do with CFAR?

IMO, earning to give remains a pathway for creating very large positive impacts on the world.
For example, CFAR, MIRI, and many of the organizations under the CEA umbrella seem to me to be both high potential impact, and substantially funding limited right now, such that further donation is likely to cause a substantial increase in how much good these organizations accomplish.
This is an interesting claim to make in a world where e.g. Good Ventures, Elon Musk, and others are already putting very large sums of money into making the world better; if you expect to make a large difference via individual donation, you must implicitly expect that you can pick donation targets for your own smallish sum better than they could, at least at the margin. (One factor that makes this more plausible is that you can afford to put in far more time/​attention/​thought per dollar than they can.)
Alternately stated: the world seems to me to be short, not so much on money, as on understanding of what to do with money; and an “Earning to Give” career seems to me to potentially make sense insofar as it is a decision to really actually try to figure out how to get humanity to a win-state (or how to otherwise do whatever it is that is actually worthwhile), especially insofar as you seem to see low-hanging fruit for high-impact donation. Arguing with varied others who have thought about it is perhaps the fastest route toward a better-developed inside view, together with setting a 5 minute timer and attempting to solve it yourself. “Earning to give”ers, IMO, contribute in proportion to the amount of non-standard epistemic skill they develop, and not just in proportion to their giving amount; and, much as with Giving Games, knowing that you matter today can provide a reason to develop a worldview for real.
This has to do with CFAR both because I expect us to accomplish far more good if we have money to do it, and because folks actually trying to figure out how to impact the world are building the kind of thinking skill CFAR is all about.