Sorry for leaving a comment after only reading the summary, so maybe this is addressed in the text, but I think I have a more concrete version of what I read as the theory being falling into a trap of a local maximum.
CFAR is just too weird.
I know lots of people like weird, but weird is self-limiting. And I don’t mean cute, “lol i have so many plants, i’m so weird”, within the normal person overton window weird, but proper normal-people-won’t-really-understand-you weird.
One of the great lessons I’ve learned from my years of Zen practice is “don’t talk about the weird”. There that means stuff like don’t talk about enlightenment, what happens during meditation (except with your teacher or a close dharma friend), or the things you can only know by experiencing them for yourself. It’s a distraction and only rarely helpful. Mostly you need to just keep at the everyday practice of Zen.
I claim rationality needs the same lesson. Lots of this stuff about rationality is actually, properly weird. And for the kind of person who enjoys it, they want to lean into the weird. This is a mistake. This kind of weird is for adepts and teachers to talk shop about on rare occasion. Everyday folks need to hear about the normal, everyday practice of a thing in order to do it and have it relate to their life.
Weird seems fine to Less Wrong because it’s tiered: you can find the weird on all posts, the normal on curated (or at least relatively normal). CFAR, to really succeed at what I see as its mission (bring rationality to the masses), needed to be the most normal version of rationality possible and as best I can tell it totally failed at this (e.g. CFAR seems single handedly responsible for dozens of bits of new jargon that could have been talked about in normal language without jargon).
The good news is I think it’s possible to start over if someone wanted. There’s some evidence this could work. For example, in the last 20 years cognitive bias training has become common. A lot of it is BS presentations rather than actual training, but there’s roots of rationality stuff out in the water among normal folks working jobs in big corporations. So we have some proof-of-concept that this is possible, but it requires optimizing for understanding by normal people, not the sort of weird people who are willing to spend their own money to get better at rationality.
(Note: I think CFAR succeeded in some important other ways. Those are out of scope for this comment, though.)
CFAR, to really succeed at what I see as its mission (bring rationality to the masses), needed...
IMO (and the opinions of Davis and Vaniver, who I was just chatting with), CFAR doesn’t and didn’t have this as much of its mission.
We were and are (from our founding in 2012 through the present) more focused on rationality education for fairly small sets of people who we thought might strongly benefit the world, e.g. by contributing to AI safety or other high-impact things, or by adding enrichment to a community that included such people. (Though with the notable exception of Julia writing the IMO excellent book “Scout Mindset,” which she started while at CFAR and which I suspect reached a somewhat larger audience.)
I do think we should have chosen our name better, and written our fundraising/year-end-report blog posts more clearly, so as to not leave you and a fair number of others with the impression we were aiming to “raise the sanity waterline” broadly. I furthermore think it was not an accident that we failed at this sort of clarity; people seemed to like us and to give us money / positive sentences / etc. when we sounded like we were going to do all the things, and I failed to adjust our course away from that local reward of “sound like you’re doing all the things, so nobody gets mad” to “communicate what’s actually up, even when that looks bad, so you’ll be building on firm ground.”
One Particular Center for Helping A Specific Nerdy Demographic Bridge Common Sense and Singularity Scenarios And Maybe Do Alignment Research Better But Not Necessarily The Only Or Primary Center Doing Those Things
We were and are (from our founding in 2012 through the present) more focused on rationality education for fairly small sets of people who we thought might strongly benefit the world, e.g. by contributing to AI safety or other high-impact things, or by adding enrichment to a community that included such people.
Maybe this was a wrong strategy even given your goals.
Imagine that your goal is to train 10 superheroes, and you have the following options:
A: Identify 10 people with greatest talent, and train them.
B: Focus on scaling. Train 10 000 people.
It seems possible to me that the 10 best heroes in strategy B might actually be better than the 10 heroes in strategy A. Depends on how good you are at identifying talented heroes, whether the ones you choose actually agree to get trained by you, what kinds of people self-select for the scaled-up training, etc.
Furthermore, this is actually a false dilemma. If you find a way to scale, you can still have a part of your team identify and individually approach the talented individuals. They might be even more likely to join if you tell them that you already trained 10 000 people but they will get an individualized elite training.
Sorry for leaving a comment after only reading the summary, so maybe this is addressed in the text, but I think I have a more concrete version of what I read as the theory being falling into a trap of a local maximum.
CFAR is just too weird.
I know lots of people like weird, but weird is self-limiting. And I don’t mean cute, “lol i have so many plants, i’m so weird”, within the normal person overton window weird, but proper normal-people-won’t-really-understand-you weird.
One of the great lessons I’ve learned from my years of Zen practice is “don’t talk about the weird”. There that means stuff like don’t talk about enlightenment, what happens during meditation (except with your teacher or a close dharma friend), or the things you can only know by experiencing them for yourself. It’s a distraction and only rarely helpful. Mostly you need to just keep at the everyday practice of Zen.
I claim rationality needs the same lesson. Lots of this stuff about rationality is actually, properly weird. And for the kind of person who enjoys it, they want to lean into the weird. This is a mistake. This kind of weird is for adepts and teachers to talk shop about on rare occasion. Everyday folks need to hear about the normal, everyday practice of a thing in order to do it and have it relate to their life.
Weird seems fine to Less Wrong because it’s tiered: you can find the weird on all posts, the normal on curated (or at least relatively normal). CFAR, to really succeed at what I see as its mission (bring rationality to the masses), needed to be the most normal version of rationality possible and as best I can tell it totally failed at this (e.g. CFAR seems single handedly responsible for dozens of bits of new jargon that could have been talked about in normal language without jargon).
The good news is I think it’s possible to start over if someone wanted. There’s some evidence this could work. For example, in the last 20 years cognitive bias training has become common. A lot of it is BS presentations rather than actual training, but there’s roots of rationality stuff out in the water among normal folks working jobs in big corporations. So we have some proof-of-concept that this is possible, but it requires optimizing for understanding by normal people, not the sort of weird people who are willing to spend their own money to get better at rationality.
(Note: I think CFAR succeeded in some important other ways. Those are out of scope for this comment, though.)
IMO (and the opinions of Davis and Vaniver, who I was just chatting with), CFAR doesn’t and didn’t have this as much of its mission.
We were and are (from our founding in 2012 through the present) more focused on rationality education for fairly small sets of people who we thought might strongly benefit the world, e.g. by contributing to AI safety or other high-impact things, or by adding enrichment to a community that included such people. (Though with the notable exception of Julia writing the IMO excellent book “Scout Mindset,” which she started while at CFAR and which I suspect reached a somewhat larger audience.)
I do think we should have chosen our name better, and written our fundraising/year-end-report blog posts more clearly, so as to not leave you and a fair number of others with the impression we were aiming to “raise the sanity waterline” broadly. I furthermore think it was not an accident that we failed at this sort of clarity; people seemed to like us and to give us money / positive sentences / etc. when we sounded like we were going to do all the things, and I failed to adjust our course away from that local reward of “sound like you’re doing all the things, so nobody gets mad” to “communicate what’s actually up, even when that looks bad, so you’ll be building on firm ground.”
One Particular Center for Helping A Specific Nerdy Demographic Bridge Common Sense and Singularity Scenarios And Maybe Do Alignment Research Better But Not Necessarily The Only Or Primary Center Doing Those Things
A Center for Trying to Improve Our Non-Ideal Cognitive Inclinations for Navigating to Gigayears
ACTION ICING
I’m liking the uptick in “Gigayears”
Maybe this was a wrong strategy even given your goals.
Imagine that your goal is to train 10 superheroes, and you have the following options:
A: Identify 10 people with greatest talent, and train them.
B: Focus on scaling. Train 10 000 people.
It seems possible to me that the 10 best heroes in strategy B might actually be better than the 10 heroes in strategy A. Depends on how good you are at identifying talented heroes, whether the ones you choose actually agree to get trained by you, what kinds of people self-select for the scaled-up training, etc.
Furthermore, this is actually a false dilemma. If you find a way to scale, you can still have a part of your team identify and individually approach the talented individuals. They might be even more likely to join if you tell them that you already trained 10 000 people but they will get an individualized elite training.