Among the ultimate criteria for the minicamps is their impact on long-term life success. To assess this, both minicamp participants and a control group completed a long, anonymous survey containing many indicators of life success (income, self-reported happiness and anxiety levels, many questions about degree of social connectedness and satisfaction with relationships, etc.); we plan to give it again to both groups a year after mini-camp, to see whether minicampers improved more than controls. I’m eager to see and update from those results, but we’re only a couple months into the year’s waiting period. (The reason we decided ahead of time to wait a year is that minicamp aimed to give participants tools for personal change; and, for example, it takes time for improved social skills, strategicness, and career plans to translate into income.)
Meanwhile, we’re working with self-report measures because they are what we have. But they are more positive than I anticipated, and that can’t be a bad sign. I was also positively surprised by the number of rationality, productivity, and social effectiveness habits that participants reported using regularly, in response to my email asking, two months out. To quote a significant fraction of the numerical data from the exit survey (from the last day of minicamp), for those who haven’t seen participants’ ratings:
In answer to “Zero to ten, are you glad you came?”, the median answer was 10 (mean was 9.3).
In answer to “Zero to ten, will your life go significantly differently because you came to mini-camp?” the median answer was 7.5 (the mean was 6.9). [This was the response that was most positively surprising to me.]
In answer to “Zero to ten, has your epistemic rationality improved?”, the median answer was 7 (mean 6.9)
In answer to “Zero to ten, are you more motivated to learn epistemic rationality, than you were when you came?”, the median answer was 8.5 (mean 8.1)
In answer to “Zero to ten, have you become more skilled at modifying your emotions and dispositions?”, the median answer was 7 (mean 6.3).
In answer to “Zero to ten, are you more motivated to modify your emotions and dispositions, than you were when you came?”, the median answer was 9 (mean 8.3).
In answer to “Zero to ten, are you more motivated to gain social skills than you were when you came?”, the median answer was 8 (mean 7.7).
In answer to “Zero to ten, have you gained social skills since coming?”, the median answer was 7.5 (mean 7.2).
In answer to “zero to ten, did you like Luke’s sessions?”, the median answer was 9 (mean answer 8.7).
Some excerpts from the survey, about about Luke’s sessions in particular:
“Luke is an excellent presenter. These sessions exceeded my expectations: I am convinced I have under-valued social interaction and techniques and that I can accelerate my success curve by aggressively adopting them. ”
“I really liked Luke’s sessions. They were fun and interactive and well put together. There is an effect of being a bit more personally interested in the material.”
“Very useful content. Great presentation of it. Very good at handling the practical camp-issues and also useful fashion tips.”
“Luke’s sessions were concise, and well structured. Good PPT templates!”
“The social effectiveness and fashion sessions were very useful for me. ”
“Some parts of some sessions i felt went too slowly… but mostly extremely valuable information. wish we could have more social skills sessions—i would take another camp just for these super low-hanging fruit.”
“Luke gave concrete examples and advice. It was very helpful.”
“Luke was great as a session leader. His sessions were very clearly, cleanly organized, and discussions in his sessions were handled very well. Luke has, by far, the presence to lead a discussion among 16 people. :)”
“Luke was great. His sessions hit the relevant points in an effective manner.”
“Luke was very helpful and knowledgeable. The pace of his sessions was really good, and there was a lot of room for discussion. Luke also gave some helpful and specific fashion advice. ”
“Pretty much everything with Luke was phenomenal… Luke really made this whole camp worthwhile. I know this is more praise than constructive feedback, but I legitimately can’t think of anything!”
I worked on mini-camp with Luke, and I can honestly say that it’s only because of Luke that we were able to hold minicamp at all, and also that he was a phenomenal work partner in organizing the camp, getting all the logistics together, and generally making it a positive and, for many, life-changing experience.
More generally: In minicamp and other SingInst projects, Luke combines energy, reliable ability to carry projects to completion, and strategicness as to which projects make sense and which aspects of those projects are most worth the extra effort; if you’re looking to reduce existential risk, making it possible for SingInst to stably hire Luke seems to me to offer unusually good bang for your buck.
How was the control group selected? Did you select a pool of candidates larger than you could accept then randomly take a subset of these as a control? If not then calling it a ‘control group’ is borderline at best.
The prior expectation of the influence of one week of training on personal success over a year is far lower than that of various personal and environmental qualities in the individual. This being the case it is more reasonable to attribute differences in progress between the groups to the higher potential for growth in the chosen minicampers. This primarily reflects well on the ability of the Singinst rationality trainers to identify indicators of future success—a rather important skill in its own right!
A good point. The control group was of folks who made it through the initial screening but not the final screening, so, yes, there are differences. We explicitly discussed the possibility of randomizing admissions, but, for our first go, elected to admit the 25 people we most wanted, and to try randomizing some future events if the first worked well enough to warrant follow-ups (which it did). It is a bit of a hit to the data-gathering, but it wasn’t growth potential as such that we were selecting for—for example, younger applicants were less likely to have cool accomplishments and therefore less likely to get in, although they probably have more growth potential—so there should still be evidence in the results.
Also, we marked down which not-admitted applicants were closest to the cut-off line (there were a number of close calls; I really wished we had more than 25 spaces), so we can gain a bit of data by seeing if they were similar to the minicamp group or to the rest of the controls.
I have a real hard time deciding how seriously I should take this survey.
The halo effect for doing anything around awesome people like are found in a selected group of Lesswrongians is probably pretty strong. I fear at least some of the participants may have mixed up being with awesome people with becoming awesome. Don’t get me wrong being with awesome people in of itself will work … for a while, until you leave that group.
I’m not that sceptical of the claims, but from the outside its hard to tell the difference between this scenario and the rationality camps working as intended.
You’re right to suspect that this could have happened. That said: I was a mini-camp participant, and I actually became more awesome as a result. Since mini-camp, I’ve:
used Fermi calculations (something we practiced) to decide to graduate from school early.
started making more money than I had before.
started negotiating for things, which saved me over $1000 this summer.
begun the incredibly fucking useful practice of rejection therapy, which multiplied my confidence and caused the above two points.
rapidly improved my social abilities, including the easily measurable ‘success with women’ factor. This was mostly caused by a session about physical contact by Will Ryan, and from two major improvements in wardrobe caused by the great and eminent lukeprog (in whose name I just donated). I wasn’t bad at social stuff before—this was a step from good to great.
resolved my feelings about a bad relationship, mostly as a result of boosted confidence from increased social success.
I stuck around in California for the summer, and gained a lot from long conversations with other SIAI-related people. The vigor and insight of the community was a major factor in showing me how much more was possible and helping me stick to plans I initiated.
But, that said—the points listed above appear to be a direct result of the specific things I learned at mini-camp.
I suspect that it’s precisely because of concerns like these that they didn’t present these numbers until now. It’s hard to see what other evidence they could have for the efficacy of the “minicamp” at this stage.
(Edited to replace “bootcamp” with “minicamp” as per wedrified’s correction)
Don’t get me wrong being with awesome people in of itself will work … for a while, until you leave that group.
I’m not that sceptical of the claims, but from the outside its hard to tell the difference between this scenario and the rationality camps working as intended.
Indeed. SIAI is conducting a year-later follow up which should provide the information needed to differentiate. Answering that question now is probably not possible to the degree of certainty required.
Yes; what I meant by “success” was more like a successful party or conference; Luke pulled off an event that nearly all the attendees were extremely glad they came to, gave presentations that held interest and influenced behavior for at least the upcoming weeks, etc. It was successful enough that, when combined with Luke’s other accomplishments, I know we want Luke, for his project-completion, social effectiveness, strategicness, fast learning curves, and ability to fit all these qualities into SingInst in a manner that boosts our overall effectiveness. I don’t mean “Minicamp definitely successfully created new uber-rationalists”; that would be a weird call from this data, given priors.
Sure, but Konkvistador’s post is about how the survey might be contaminated by awesome-people-halo-effect, not that we shouldn’t be calling it a success. That’s a separate concern addressed elsewhere. My post was addressing how we would tell the difference between “working” and “near awesome people”.
Since you’re using self-reporting anyway, it would have been good if you had a ‘how invested do you feel in minicamp’s success?’ question. Of course I say that having seen the results already.
Among the ultimate criteria for the minicamps is their impact on long-term life success. To assess this, both minicamp participants and a control group completed a long, anonymous survey containing many indicators of life success (income, self-reported happiness and anxiety levels, many questions about degree of social connectedness and satisfaction with relationships, etc.); we plan to give it again to both groups a year after mini-camp, to see whether minicampers improved more than controls. I’m eager to see and update from those results, but we’re only a couple months into the year’s waiting period. (The reason we decided ahead of time to wait a year is that minicamp aimed to give participants tools for personal change; and, for example, it takes time for improved social skills, strategicness, and career plans to translate into income.)
Meanwhile, we’re working with self-report measures because they are what we have. But they are more positive than I anticipated, and that can’t be a bad sign. I was also positively surprised by the number of rationality, productivity, and social effectiveness habits that participants reported using regularly, in response to my email asking, two months out. To quote a significant fraction of the numerical data from the exit survey (from the last day of minicamp), for those who haven’t seen participants’ ratings:
In answer to “Zero to ten, are you glad you came?”, the median answer was 10 (mean was 9.3).
In answer to “Zero to ten, will your life go significantly differently because you came to mini-camp?” the median answer was 7.5 (the mean was 6.9). [This was the response that was most positively surprising to me.]
In answer to “Zero to ten, has your epistemic rationality improved?”, the median answer was 7 (mean 6.9)
In answer to “Zero to ten, are you more motivated to learn epistemic rationality, than you were when you came?”, the median answer was 8.5 (mean 8.1)
In answer to “Zero to ten, have you become more skilled at modifying your emotions and dispositions?”, the median answer was 7 (mean 6.3).
In answer to “Zero to ten, are you more motivated to modify your emotions and dispositions, than you were when you came?”, the median answer was 9 (mean 8.3).
In answer to “Zero to ten, are you more motivated to gain social skills than you were when you came?”, the median answer was 8 (mean 7.7).
In answer to “Zero to ten, have you gained social skills since coming?”, the median answer was 7.5 (mean 7.2).
In answer to “zero to ten, did you like Luke’s sessions?”, the median answer was 9 (mean answer 8.7).
Some excerpts from the survey, about about Luke’s sessions in particular:
“Luke is an excellent presenter. These sessions exceeded my expectations: I am convinced I have under-valued social interaction and techniques and that I can accelerate my success curve by aggressively adopting them. ”
“I really liked Luke’s sessions. They were fun and interactive and well put together. There is an effect of being a bit more personally interested in the material.”
“Very useful content. Great presentation of it. Very good at handling the practical camp-issues and also useful fashion tips.”
“Luke’s sessions were concise, and well structured. Good PPT templates!”
“The social effectiveness and fashion sessions were very useful for me. ”
“Some parts of some sessions i felt went too slowly… but mostly extremely valuable information. wish we could have more social skills sessions—i would take another camp just for these super low-hanging fruit.”
“Luke gave concrete examples and advice. It was very helpful.”
“Luke was great as a session leader. His sessions were very clearly, cleanly organized, and discussions in his sessions were handled very well. Luke has, by far, the presence to lead a discussion among 16 people. :)”
“Luke was great. His sessions hit the relevant points in an effective manner.”
“Luke was very helpful and knowledgeable. The pace of his sessions was really good, and there was a lot of room for discussion. Luke also gave some helpful and specific fashion advice. ”
“Pretty much everything with Luke was phenomenal… Luke really made this whole camp worthwhile. I know this is more praise than constructive feedback, but I legitimately can’t think of anything!”
I worked on mini-camp with Luke, and I can honestly say that it’s only because of Luke that we were able to hold minicamp at all, and also that he was a phenomenal work partner in organizing the camp, getting all the logistics together, and generally making it a positive and, for many, life-changing experience.
More generally: In minicamp and other SingInst projects, Luke combines energy, reliable ability to carry projects to completion, and strategicness as to which projects make sense and which aspects of those projects are most worth the extra effort; if you’re looking to reduce existential risk, making it possible for SingInst to stably hire Luke seems to me to offer unusually good bang for your buck.
How was the control group selected? Did you select a pool of candidates larger than you could accept then randomly take a subset of these as a control? If not then calling it a ‘control group’ is borderline at best.
The prior expectation of the influence of one week of training on personal success over a year is far lower than that of various personal and environmental qualities in the individual. This being the case it is more reasonable to attribute differences in progress between the groups to the higher potential for growth in the chosen minicampers. This primarily reflects well on the ability of the Singinst rationality trainers to identify indicators of future success—a rather important skill in its own right!
A good point. The control group was of folks who made it through the initial screening but not the final screening, so, yes, there are differences. We explicitly discussed the possibility of randomizing admissions, but, for our first go, elected to admit the 25 people we most wanted, and to try randomizing some future events if the first worked well enough to warrant follow-ups (which it did). It is a bit of a hit to the data-gathering, but it wasn’t growth potential as such that we were selecting for—for example, younger applicants were less likely to have cool accomplishments and therefore less likely to get in, although they probably have more growth potential—so there should still be evidence in the results.
Also, we marked down which not-admitted applicants were closest to the cut-off line (there were a number of close calls; I really wished we had more than 25 spaces), so we can gain a bit of data by seeing if they were similar to the minicamp group or to the rest of the controls.
I have a real hard time deciding how seriously I should take this survey.
The halo effect for doing anything around awesome people like are found in a selected group of Lesswrongians is probably pretty strong. I fear at least some of the participants may have mixed up being with awesome people with becoming awesome. Don’t get me wrong being with awesome people in of itself will work … for a while, until you leave that group.
I’m not that sceptical of the claims, but from the outside its hard to tell the difference between this scenario and the rationality camps working as intended.
You’re right to suspect that this could have happened. That said: I was a mini-camp participant, and I actually became more awesome as a result. Since mini-camp, I’ve:
used Fermi calculations (something we practiced) to decide to graduate from school early.
started making more money than I had before.
started negotiating for things, which saved me over $1000 this summer.
begun the incredibly fucking useful practice of rejection therapy, which multiplied my confidence and caused the above two points.
rapidly improved my social abilities, including the easily measurable ‘success with women’ factor. This was mostly caused by a session about physical contact by Will Ryan, and from two major improvements in wardrobe caused by the great and eminent lukeprog (in whose name I just donated). I wasn’t bad at social stuff before—this was a step from good to great.
resolved my feelings about a bad relationship, mostly as a result of boosted confidence from increased social success.
I stuck around in California for the summer, and gained a lot from long conversations with other SIAI-related people. The vigor and insight of the community was a major factor in showing me how much more was possible and helping me stick to plans I initiated.
But, that said—the points listed above appear to be a direct result of the specific things I learned at mini-camp.
I suspect that it’s precisely because of concerns like these that they didn’t present these numbers until now. It’s hard to see what other evidence they could have for the efficacy of the “minicamp” at this stage.
(Edited to replace “bootcamp” with “minicamp” as per wedrified’s correction)
Indeed. SIAI is conducting a year-later follow up which should provide the information needed to differentiate. Answering that question now is probably not possible to the degree of certainty required.
That’s exactly the complaint though—many people have described it as a success, before the data is available.
I think people are seeing drastically different things in the word ‘success’.
Yes; what I meant by “success” was more like a successful party or conference; Luke pulled off an event that nearly all the attendees were extremely glad they came to, gave presentations that held interest and influenced behavior for at least the upcoming weeks, etc. It was successful enough that, when combined with Luke’s other accomplishments, I know we want Luke, for his project-completion, social effectiveness, strategicness, fast learning curves, and ability to fit all these qualities into SingInst in a manner that boosts our overall effectiveness. I don’t mean “Minicamp definitely successfully created new uber-rationalists”; that would be a weird call from this data, given priors.
Sure, but Konkvistador’s post is about how the survey might be contaminated by awesome-people-halo-effect, not that we shouldn’t be calling it a success. That’s a separate concern addressed elsewhere. My post was addressing how we would tell the difference between “working” and “near awesome people”.
Thanks for putting that together, Anna!
Since you’re using self-reporting anyway, it would have been good if you had a ‘how invested do you feel in minicamp’s success?’ question. Of course I say that having seen the results already.