Very good question about health care. I agree completely that we need more rationality in health care. I am very disturbed at the number of physicians who treat medicine as a job and a profession with rules to be followed rather than as a way of thinking and understanding. I really, really would like to find a scientifically minded, rational PCP. (It occurs to me that I do know a bunch of folks at Metamed. I should probably ask them.)
My meta-question for CFAR is what are they doing/planning to bring heavy-duty rationality skills into fields that need them: medicine, education, government, jurisprudence, charity, software development, etc.? Teaching workshops, no matter how life-changing, to 20 people a month doesn’t scale.
Second meta-question for CFAR: does it make sense to focus on younger folks at the start of their careers, or even earlier (as SPARC does) so there’s a longer compounding payoff over a lifetime or should there be more focus on established professionals, so there’s more payoff sooner? or both? If both, do the same workshops, venues, and curriculum make sense for early, mid, and late-career people? E.g. Anna Salamon mentioned that “One person left early from the last camp; he said his main disappointment was that he expected an organized operation with suits.” Should CFAR offer workshops with instructors in suits in D.C. to talk to bureaucrats? Should CFAR offer cheaper workshops on college campuses to talk to undergraduates?
To date it seems to me that the most scalable means of expanding the world’s net stock of instrumental and epistemic rationality has been written communication. E.g.
The Popular Literature on Behavioral Economics, particularly the books of Dan Ariely
Less Wrong
Harry Potter and the Methods of Rationality
Of course not everyone learns the same way. The great benefit of Harry Potter and the Methods of Rationality is that it attracts people who would never be reached by any other product of the community. What else can we do that reaches new people and grows rationality skills? Some folks can get immense value from in-person instruction but never from a book. And different means of instruction can reinforce techniques. But in-person instruction doesn’t scale like books and blogs do. Where does the comparative advantage lie? What is the most effective next thing that can be done to increase rationality?
Written communication has many advantages, but it typically does not make you actually do the exercises. Typically, one just looks briefly at the exercise, thinks “yeah, I see what they are trying to do” and then clicks another hyperlink or switches to another browser tab.
Having five minutes without internet access and with a social pressure to actually do the exercise can make people actually do the exercises they found on internet a decade ago but never tried.
Sure, everyone is different, but I would expect most people who spend a lot of time on internet to be like this. (And the people who don’t spend a lot of time on internet won’t see LifeHacker or LessWrong, unless a book form is published.)
I see MOOC’s as a big educaational improvement because of this—sure, I could get the same educational info without the MOOC structure; just by reading the field best textbooks and academic papers; but having a specific “course” with the quizzes/homework makes me actually do the excercises, which I wouldn’t have done otherwise; and the course schedule forces me to do them now, instead of postponing them for weeks/months/forever.
Absolutely true. Some people, perhaps most, don’t do the exercises. Also true that some people (myself included) do. Even if only a small percentage of people do the exercises they read about, and only some of the time, that still scales better than in-person classes.
On continuing reflection it occurs to me that there’s a third scalable technique for increasing rationality, at least in theory: software. I’ve seen a few attempts to set up tools like the calibration game in software. So far they haven’t caught on, but it might be worth exploring further, especially if this can be worked into games the way HpMOR works rationality into really gripping fiction.
Thinking back on my life, board games, card games, and D&D type RPGs helped me learn basic probability and game theory without explicitly attempting to do so. I’m not sure today’s videogames, fun as they are, have the same virtuous and useful side effects. I wonder if it would be worthwhile to develop a really gripping game that rewarded rational play and probability based reasoning and implicitly taught it.
Even if only a small percentage of people do the exercises they read about, and only some of the time, that still scales better than in-person classes.
That’s a good point . I am not sure about the numbers today: how many people read LW, how many percent of them would do the exercises, is it more than minicamp participants? But these numbers could be improved by e.g. converting the minicamps into series of online lessons.
I guess this is a great opportunity for a CFAR volunteer with video-editing skills.
The recent XCOM game, to some extent meets your criteria (a few bugs aside). Every move matters and must be carefully planned, there are very few actions you can carry out that don’t carry a chance of failure. You quickly learn when you can afford to be ambitious, and when you need to have a back up plan if things go wrong. Even better, in Ironman mode your ~30 hour save can easily be more or less ruined in a single turn if you make particularly poor choices (or get very, very unlucky) and you have no save game to resume—you have to start over from the beginning.
My experience talking to other people playing it isn’t, however, particularly promising when it comes to implicit teaching. One friend has complained every single time he’s missed a 98% chance (“it’s bullshit”), even when I pointed out that you make thousands of attacks over the course of a game and should expect to see multiple misses at those odds.
If you haven’t seen it before, this is an entertaining video series that demonstrates the salient points quickly.
“Doing the exercise” is not something that the student does alone, with the result compared to the answer key in the teacher’s edition textbook. To perform the exercises, the student needs other people with enough understanding of the subject to provide short-cycle feedback, and I don’t know anyone who can do that for themselves.
This is basically our problem. We could definitely teach the theory of, say, our Installing Habits class remotely, but we spend a lot of it troubleshooting people’s actual plans for setting up new habits and giving rapid, personalized feedback, and it’d be quite hard to build that into automated exercises.
On further reflection it occurs to me that research done over the last 50 or so years has also been extremely valuable at increasing rationality. Thanks to the work of Kahneman, Tversky, Ariely, Pearl, Zadeh, and others there are things we know today about cognitive biases and rationality that no one knew in 1963. So original research in certain fields also has the potential to be both scalable and useful.
My meta-question for CFAR is what are they doing/planning to bring heavy-duty rationality skills into fields that need them: medicine, education, government, jurisprudence, charity, software development, etc.? Teaching workshops, no matter how life-changing, to 20 people a month doesn’t scale.
Workshops scale quite well if you have a specific curriculum and don’t rely on the fact that a specific instructor is an awesome person that can’t be replaced. You hire more instructors and you can teach more people.
CFAR basically has to make business decisions about being able to provide rationality workshops at a price that enough people can afford to pay for CFAR to grow the number of participants.
There no reason why CFAR shouldn’t be able to grow the number of participants exponentially.
To date it seems to me that the most scalable means of expanding the world’s net stock of instrumental and epistemic rationality has been written communication. E.g.
How many people follow paradigms such as Yoga, the Landmark Forum or NLP that get taught primarily through in person training and how many people follow proper epistemic rationality?
People who promote epistemic rationality were quite bad at spreading it in the last decades compared to how paradigms that get taught in person spread.
$3,900 for a three day seminar is probably no price point at which you can teach the workshop to tens of thousands of people. Over time CFAR is probably well advised to streamline their process in a way that makes the workshop cheaper.
There no reason why CFAR shouldn’t be able to grow the number of participants exponentially.
I do not concur. CFAR is currently a small organization using small-organization logistics. Expanding to many more instructors would require a management layer different from the implementation layer, and selecting the best implementers to become management has a long history of failure.
One possible solution would be to spin off groups roughly the current size, preferably geographically diverse. That adds more dimensions of complexity but still allows for virtually everybody to be directly involved with the immediate returns of teaching and curriculum development.
At this moment there are already regular LW meetups in different cities around the world. We could find willing instructors in many of them, send them educational materials (one PDF they give to each student, one PDF with the instructions for the teacher), let them teach the lessons and send back the feedback.
The remote teachers and students are already there, and they wouldn’t cost CFAR anything. The costs for CFAR at this moment would be: creating the PDF materials from the lessons, and evaluating the feedback.
(I need to think about it some more, and perhaps I will volunteer to make one such example lesson. And publish it on LW, and process the feedback.)
That was my first plan back when things were getting started, but it turned out to be hard to develop instructional materials that worked without a developed professional instructor.
Moving the weight from instructor to material is always a lot of work. A lot of tacit knowledge needs to be made explicit.
These days I am having (as a student) an online lecture about some Java technology. It’s 3 days, 8 hours each, we received in total 600 pages of PDF. That is 12 pages per 30 minutes; minus covers and TOC it’s 9 pages of useful text.
Years ago I tried to make a non-interactive lesson for high-school students where I just gave them a PDF file with explanations and exercises, and then they worked everyone at their own speed. I needed 8-10 pages for a lesson, and I spent the whole evening just writing what I already perfectly knew. Students liked it, but I gave up doing this because it was too much work for one-time use. However if I had to teach the same thing to many classes (or just the same thing for many years), then it would be less work doing it this way. And the materials can be updated when necessary.
With the rationality exercises it will be even more complicated because we are not even 100% sure about the topic, and there can be more unexpected questions and reactions during the lesson. But I still think it is possible, and that given enough students it may be even more efficient. -- I guess the detailed material needs at least 10 repetitions to be more efficient. So if you design a lesson at CFAR, do it 2 or 3 times and then redesign it, it is not worth making detailed materials. But if we want to use the lessons at meetups, then it could be worth doing.
Yes, this is what we first tried before finding out that it was way below the level of working with late-2011-level knowledge and ability to produce lessons. Might be worth retrying once the lessons have been highly polished at the CFAR level.
If so, that would be evidence that it is not the best way to implement. The ability to improve a class by redesigning it is a feature of the organization.
How hard is it to create a developed professional instructor? I was under the impression that less than all of CFAR’s instructors were primarily educators...
I think quite a few meetups have at least one person that has gone to a workshop. There could be some teaching how to teach at the workshop so that when the go back to the meetups, they can teach there.
As one of the CFAR initiates and a meetup person, the minimum CFAR could do for us is:
A single well-contained booklet of everything with soft copy (they’ve done this since I went)
Point form script and notes that is used by the CFAR instructors when running the classes. (AFAIK, this is outstanding still)
Mostly just to prompt memory. Once you’ve been through it, it’s not hard to remember how it was taught and duplicate it so long as the right prompts are there.
Schedule concerns are as follows: I reside on Nantucket and my work schedule for the foreseeable future is Thursday-Monday, with little chance to get a day off before Memorial Day. Monthly recurring is not an option.
If you can do a meetup on a Monday night, Tuesday, or Wednesday morning (and crash space is available), I can catch a boat and bus and probably the subway.
Growing an organisation such as CFAR isn’t easy. It takes some skill. I don’t know the CFAR people personally but I have no reason to assume that they are up for that task.
I have met them, although in a limited context, and using that information I have estimated if they were to try to expand, they would be somewhat more likely to do so successfully than the typical group that attempts to expand.
I think that my prior is poorly grounded because I have experience with only a few small organizations that tried to expand and failed, and a larger number that tried successfully. However, I didn’t know of any of the successful groups before they grew.
Small-organization (everybody knows everybody) scales to a finite size. Other networking patterns tend to scale better, and I think that cellular organization might work better than hierarchical organization.
True, and I can’t see any benefit from hierarchical organisation. There isn’t a central authority of rationality any more than there is one for chemistry or calculus.
But CFAR maybe hasn’t scaled to its maximum size yet, and as it approaches it, it will probably become clearer what the ideal size is, and there will be more people with experience in training who can split off another group.
The world is almost entirely controlled by hierarchical organizations ( corporates and governments). Hiercharchal organizations have “won” to a greater extent than pretty much anything else on earth. It’s a model with flaws, but it clearly works. A person would need a whole lot of willful blindness to argue with those results.
Now as to the question of if those organizations would be good at teaching rationality, that’s another question...
Yeah, I can see how hierarchical organisations benefit certain goals and activities. I was speaking specifically about the goal of teaching rationality, in case that wasn’t clear from context. You don’t need a central authority to control what is being taught so much unless you are teaching irrationality (c.f. Scientology, Roman Catholicism or any political organisation).
You could probably run a million rationality courses a year using just a wiki and a smartphone app. (Left as an exercise for the reader)
Very good question about health care. I agree completely that we need more rationality in health care. I am very disturbed at the number of physicians who treat medicine as a job and a profession with rules to be followed rather than as a way of thinking and understanding. I really, really would like to find a scientifically minded, rational PCP. (It occurs to me that I do know a bunch of folks at Metamed. I should probably ask them.)
My meta-question for CFAR is what are they doing/planning to bring heavy-duty rationality skills into fields that need them: medicine, education, government, jurisprudence, charity, software development, etc.? Teaching workshops, no matter how life-changing, to 20 people a month doesn’t scale.
Second meta-question for CFAR: does it make sense to focus on younger folks at the start of their careers, or even earlier (as SPARC does) so there’s a longer compounding payoff over a lifetime or should there be more focus on established professionals, so there’s more payoff sooner? or both? If both, do the same workshops, venues, and curriculum make sense for early, mid, and late-career people? E.g. Anna Salamon mentioned that “One person left early from the last camp; he said his main disappointment was that he expected an organized operation with suits.” Should CFAR offer workshops with instructors in suits in D.C. to talk to bureaucrats? Should CFAR offer cheaper workshops on college campuses to talk to undergraduates?
To date it seems to me that the most scalable means of expanding the world’s net stock of instrumental and epistemic rationality has been written communication. E.g.
David Allen’s Getting Things Done
Productivity Blogs such as LifeHacker
The Popular Literature on Behavioral Economics, particularly the books of Dan Ariely
Less Wrong
Harry Potter and the Methods of Rationality
Of course not everyone learns the same way. The great benefit of Harry Potter and the Methods of Rationality is that it attracts people who would never be reached by any other product of the community. What else can we do that reaches new people and grows rationality skills? Some folks can get immense value from in-person instruction but never from a book. And different means of instruction can reinforce techniques. But in-person instruction doesn’t scale like books and blogs do. Where does the comparative advantage lie? What is the most effective next thing that can be done to increase rationality?
Written communication has many advantages, but it typically does not make you actually do the exercises. Typically, one just looks briefly at the exercise, thinks “yeah, I see what they are trying to do” and then clicks another hyperlink or switches to another browser tab.
Having five minutes without internet access and with a social pressure to actually do the exercise can make people actually do the exercises they found on internet a decade ago but never tried.
Sure, everyone is different, but I would expect most people who spend a lot of time on internet to be like this. (And the people who don’t spend a lot of time on internet won’t see LifeHacker or LessWrong, unless a book form is published.)
I see MOOC’s as a big educaational improvement because of this—sure, I could get the same educational info without the MOOC structure; just by reading the field best textbooks and academic papers; but having a specific “course” with the quizzes/homework makes me actually do the excercises, which I wouldn’t have done otherwise; and the course schedule forces me to do them now, instead of postponing them for weeks/months/forever.
Absolutely true. Some people, perhaps most, don’t do the exercises. Also true that some people (myself included) do. Even if only a small percentage of people do the exercises they read about, and only some of the time, that still scales better than in-person classes.
On continuing reflection it occurs to me that there’s a third scalable technique for increasing rationality, at least in theory: software. I’ve seen a few attempts to set up tools like the calibration game in software. So far they haven’t caught on, but it might be worth exploring further, especially if this can be worked into games the way HpMOR works rationality into really gripping fiction.
Thinking back on my life, board games, card games, and D&D type RPGs helped me learn basic probability and game theory without explicitly attempting to do so. I’m not sure today’s videogames, fun as they are, have the same virtuous and useful side effects. I wonder if it would be worthwhile to develop a really gripping game that rewarded rational play and probability based reasoning and implicitly taught it.
That’s a good point . I am not sure about the numbers today: how many people read LW, how many percent of them would do the exercises, is it more than minicamp participants? But these numbers could be improved by e.g. converting the minicamps into series of online lessons.
I guess this is a great opportunity for a CFAR volunteer with video-editing skills.
The recent XCOM game, to some extent meets your criteria (a few bugs aside). Every move matters and must be carefully planned, there are very few actions you can carry out that don’t carry a chance of failure. You quickly learn when you can afford to be ambitious, and when you need to have a back up plan if things go wrong. Even better, in Ironman mode your ~30 hour save can easily be more or less ruined in a single turn if you make particularly poor choices (or get very, very unlucky) and you have no save game to resume—you have to start over from the beginning.
My experience talking to other people playing it isn’t, however, particularly promising when it comes to implicit teaching. One friend has complained every single time he’s missed a 98% chance (“it’s bullshit”), even when I pointed out that you make thousands of attacks over the course of a game and should expect to see multiple misses at those odds.
If you haven’t seen it before, this is an entertaining video series that demonstrates the salient points quickly.
“Doing the exercise” is not something that the student does alone, with the result compared to the answer key in the teacher’s edition textbook. To perform the exercises, the student needs other people with enough understanding of the subject to provide short-cycle feedback, and I don’t know anyone who can do that for themselves.
This is basically our problem. We could definitely teach the theory of, say, our Installing Habits class remotely, but we spend a lot of it troubleshooting people’s actual plans for setting up new habits and giving rapid, personalized feedback, and it’d be quite hard to build that into automated exercises.
On further reflection it occurs to me that research done over the last 50 or so years has also been extremely valuable at increasing rationality. Thanks to the work of Kahneman, Tversky, Ariely, Pearl, Zadeh, and others there are things we know today about cognitive biases and rationality that no one knew in 1963. So original research in certain fields also has the potential to be both scalable and useful.
Workshops scale quite well if you have a specific curriculum and don’t rely on the fact that a specific instructor is an awesome person that can’t be replaced. You hire more instructors and you can teach more people.
CFAR basically has to make business decisions about being able to provide rationality workshops at a price that enough people can afford to pay for CFAR to grow the number of participants.
There no reason why CFAR shouldn’t be able to grow the number of participants exponentially.
How many people follow paradigms such as Yoga, the Landmark Forum or NLP that get taught primarily through in person training and how many people follow proper epistemic rationality?
People who promote epistemic rationality were quite bad at spreading it in the last decades compared to how paradigms that get taught in person spread.
$3,900 for a three day seminar is probably no price point at which you can teach the workshop to tens of thousands of people. Over time CFAR is probably well advised to streamline their process in a way that makes the workshop cheaper.
I do not concur. CFAR is currently a small organization using small-organization logistics. Expanding to many more instructors would require a management layer different from the implementation layer, and selecting the best implementers to become management has a long history of failure.
One possible solution would be to spin off groups roughly the current size, preferably geographically diverse. That adds more dimensions of complexity but still allows for virtually everybody to be directly involved with the immediate returns of teaching and curriculum development.
At this moment there are already regular LW meetups in different cities around the world. We could find willing instructors in many of them, send them educational materials (one PDF they give to each student, one PDF with the instructions for the teacher), let them teach the lessons and send back the feedback.
The remote teachers and students are already there, and they wouldn’t cost CFAR anything. The costs for CFAR at this moment would be: creating the PDF materials from the lessons, and evaluating the feedback.
(I need to think about it some more, and perhaps I will volunteer to make one such example lesson. And publish it on LW, and process the feedback.)
That was my first plan back when things were getting started, but it turned out to be hard to develop instructional materials that worked without a developed professional instructor.
Moving the weight from instructor to material is always a lot of work. A lot of tacit knowledge needs to be made explicit.
These days I am having (as a student) an online lecture about some Java technology. It’s 3 days, 8 hours each, we received in total 600 pages of PDF. That is 12 pages per 30 minutes; minus covers and TOC it’s 9 pages of useful text.
Years ago I tried to make a non-interactive lesson for high-school students where I just gave them a PDF file with explanations and exercises, and then they worked everyone at their own speed. I needed 8-10 pages for a lesson, and I spent the whole evening just writing what I already perfectly knew. Students liked it, but I gave up doing this because it was too much work for one-time use. However if I had to teach the same thing to many classes (or just the same thing for many years), then it would be less work doing it this way. And the materials can be updated when necessary.
With the rationality exercises it will be even more complicated because we are not even 100% sure about the topic, and there can be more unexpected questions and reactions during the lesson. But I still think it is possible, and that given enough students it may be even more efficient. -- I guess the detailed material needs at least 10 repetitions to be more efficient. So if you design a lesson at CFAR, do it 2 or 3 times and then redesign it, it is not worth making detailed materials. But if we want to use the lessons at meetups, then it could be worth doing.
Yes, this is what we first tried before finding out that it was way below the level of working with late-2011-level knowledge and ability to produce lessons. Might be worth retrying once the lessons have been highly polished at the CFAR level.
I wonder if a Kumon-style approach, with lots and lots of small steps and exercises done at one’s own pace would be resistant to redesign.
If so, that would be evidence that it is not the best way to implement. The ability to improve a class by redesigning it is a feature of the organization.
How hard is it to create a developed professional instructor? I was under the impression that less than all of CFAR’s instructors were primarily educators...
I think quite a few meetups have at least one person that has gone to a workshop. There could be some teaching how to teach at the workshop so that when the go back to the meetups, they can teach there.
As one of the CFAR initiates and a meetup person, the minimum CFAR could do for us is:
A single well-contained booklet of everything with soft copy (they’ve done this since I went)
Point form script and notes that is used by the CFAR instructors when running the classes. (AFAIK, this is outstanding still)
Mostly just to prompt memory. Once you’ve been through it, it’s not hard to remember how it was taught and duplicate it so long as the right prompts are there.
If you do this, I’ll run the lesson with the Boston group and give feedback.
If you can do it in Boston, I’d be willing to attend and provide feedback, schedule concerns permitting.
join usssss
Schedule concerns are as follows: I reside on Nantucket and my work schedule for the foreseeable future is Thursday-Monday, with little chance to get a day off before Memorial Day. Monthly recurring is not an option.
If you can do a meetup on a Monday night, Tuesday, or Wednesday morning (and crash space is available), I can catch a boat and bus and probably the subway.
Crash space is certainly available for traveling rationalists, but non-weekend meetups are very unlikely.
Growing an organisation such as CFAR isn’t easy. It takes some skill. I don’t know the CFAR people personally but I have no reason to assume that they are up for that task.
I have met them, although in a limited context, and using that information I have estimated if they were to try to expand, they would be somewhat more likely to do so successfully than the typical group that attempts to expand.
I think that my prior is poorly grounded because I have experience with only a few small organizations that tried to expand and failed, and a larger number that tried successfully. However, I didn’t know of any of the successful groups before they grew.
As usual it depends on the exponent.
Small-organization (everybody knows everybody) scales to a finite size. Other networking patterns tend to scale better, and I think that cellular organization might work better than hierarchical organization.
True, and I can’t see any benefit from hierarchical organisation. There isn’t a central authority of rationality any more than there is one for chemistry or calculus.
But CFAR maybe hasn’t scaled to its maximum size yet, and as it approaches it, it will probably become clearer what the ideal size is, and there will be more people with experience in training who can split off another group.
Unified PR, distribution of some costs (e.g. advertising, website administration), and dispute resolution (e.g. trademark issues) come to mind.
The world is almost entirely controlled by hierarchical organizations ( corporates and governments). Hiercharchal organizations have “won” to a greater extent than pretty much anything else on earth. It’s a model with flaws, but it clearly works. A person would need a whole lot of willful blindness to argue with those results.
Now as to the question of if those organizations would be good at teaching rationality, that’s another question...
Yeah, I can see how hierarchical organisations benefit certain goals and activities. I was speaking specifically about the goal of teaching rationality, in case that wasn’t clear from context. You don’t need a central authority to control what is being taught so much unless you are teaching irrationality (c.f. Scientology, Roman Catholicism or any political organisation).
You could probably run a million rationality courses a year using just a wiki and a smartphone app. (Left as an exercise for the reader)