Practical rationality questionnaire
EDIT, 4/18: I’m closing the survey. I’ll post analysis and a better anonymized version of the raw data in a day or so. 236 people responded; thanks very much to all who did.
For survey participants curious about the calibration questions, the answers are:
Number of republics the USSR broke up into, following the output of the cold war: 15.
The year in which the global population reached 1 billion: 1804.
The average percentage of a watermelon’s weight that comes from water: 92.
There has been much discussion of the extent to which rationality is or isn’t practically useful. There have also been many calls for better empirical evidence.
In an attempt to produce empirical evidence for or against rationality’s usefulness for LW-ers, I have here a rationality questionnaire. It takes about 15 minutes to complete, according to myself and to Katja Grace, who kindly helped me with it. I tried to hug the query of “Are there OB/LW-like techniques, or similar techniques, that actually help LW-ers achieve their goals?” This isn’t a test—we’re not measuring individuals’ rationality—we’re just looking for correlations and noisy indicators that may nevertheless tell us give us useful info in aggeragate, when used on groups.
Fill in the survey—by following
this survey link [Survey is now closed. Though the link will still let you see the questions.]
-- and know your next 15 minutes will contribute to science, truth, rationality, and the future practical successes of LW-ers. =) (… at least as far as expected value is concerned, if you assign some probablity to this data being useful.)
ADDED: Please hold off on discussing the implications of different responses for a day or two, until the rate of survey-completions dies down. Unless you’re sure your discussion won’t prejudice others’ answers.
There are questions asking us to compare ourselves with populations of other people about whom we are likely to have only limited and unreliable knowledge, to which the answers “about the same, so far as I know” and “of course I have no idea, you silly survey-author” are not available. I think this is unfortunate.
Thanks for noting the difficulty. I’m not sure how to improve it, though. I want to know comparisons to people with “basically your starting-advantages” (e.g., not being born in rural Africa), but… when I tried to phrase things in that vicinity it became much more dependent on the survey respondant’s interpretation (e.g., on whether they included IQ, or compared themselves to others with their neurological diagnosis).
Any suggestions?
Well, you could at least give “about the same” and “no idea” as options.
In general, I worry that the questionnaire is going to have trouble disentangling variations in people’s actual success and in their descriptions of their success. Of course this is very difficult to avoid.
I didn’t think of the rural Africans. I’d have shot more towards the blissful end if I had.
The IQ question is broken in the sense that it doesn’t specify the standard deviation people should give their responses in. If someone answers “148” (or any other single number), you have no way of knowing what that actually means. On some tests it is 2 standard deviations above the mean, on others 3 standard deviations, etc.
Nearly all tests in your country may have set the numerical scale in an identical way (I don’t know), but there are other scales in use elsewhere.
Even considering this, it may well be that they don’t know the SD, especially for people like me who haven’t taken an IQ test in over 20 years.
Good survey, and I like the calibration questions: coming up with good ones is tough. But I suggest the following:
Create an “as happy as other people” for the happiness questions, instead of just “less” or “more”
Define “auto accident” better (I had one that sort of nicked my bumper)
Define “exercise” better (I walk everywhere I go, totalling at least an hour per day, but I don’t go to gyms. Does that count?)
In “have you earned more or less than those you grew up with?”, include an option for “I am in school”—I think a lot of us are earning less than our peers, but only because we’re in some kind of graduate education while they’re out making money
Yvain, I’m slightly off-topic here, but did you ever do anything with your LW survey? I’m curious to see peoples’ responses.
I thought that your survey was going to be the same sort of thing, so I didn’t develop mine further. Now that I see your survey is after something different, I’ll probably try mine again.
Oh. Damn. I should have communicated better back when I started toward this survey.
Your survey looked like “expert elicitation”—seeing what LW members think about controversial issues we actually care about, and especially what those most likely to have informed opinions think, as data on what might actually be true about those issues. I’d love to know the results but don’t plan to do it myself.
In this here survey, I’m after seeing what kinds of rationality do/don’t bear practical fruit in individuals’ lives. I’d long-run also like to investigate (though not with today’s survey) the degree to which there is/isn’t a single trait “rationality” that predicts accurate belief-formation across domains, how to measure such a trait, and what helps build that trait.
Thanks for the suggestions. I added a separate question asking if respondants are currently in school. I added a happiness option per your suggestion (which means the first 45 responses will be to a different questionnaire, but we can keep that in mind). Your other suggestions make sense, but I’m refraining to avoid changing questions midstream.
Should those of us that know we are in the first 45 responses retake the survey? It looks like a number of things have changed since I took it. My default assumption is no as I don’t want to duplicate, but I thought I’d ask.
Don’t duplicate, please.
It’s mostly the same, and better to have a couple slightly-different questions (with known populations that answered one vs. the other) than to have repeated rows.
Can I go ahead and call a bias? People who earn less, are less happy, or believe they are less successful than their peers are less likely to fill out this survey.
Conversely, how many people might say that they’re happier and more successful than their peers, when it isn’t objectively true?
Sorry, SoullessAutomaton: I just deleted a perfectly good comment of yours because I don’t want it prejudicing peoples’ survey responses. I’ll undelete it in a day or two when I close the survey.
Ah, I neglected to consider that aspect. Mea culpa.
Any progress/news on this?
Good survey. My comments on what could be improved:
“Do most people your age, who live where you live, have a driver’s license?” A “I don’t know” answer would have been appropriate here. I know many who do have, and some who don’t, and loads that I have no idea of. But I ended up answering “no”, because I assumed this was meant to control for the good/bad mass transport issue. If that interpretation was correct, I’m not entirely sure that this wording is the best way to control for it.
I had the same issue as Yvain on the “exercise” question.
“Do you plan to sign up for cryonics” could use both “maybe”, and a “maybe, if it becomes better available in my country” options.
I found it odd that the “relative to others in your work...” question didn’t have an “about average” option.
On the questions where you’ll compare yourself to others, I expect that many people will use the “average / similar” replies as a stand-in for “don’t know”. Might be good to add a “how certain are you of this, on a scale from 1 to 3” subquestion to control for those.
I found it a bit odd that we were asked for “self-help, business, or productivity books”, but nothing about other kinds of books.
The calibration questions were good.
A general comments box would be nice at the end to explain some of the end cases. I had to guess at a lot of numbers and gave some word answers to a few. Also, I had to guess at what some of the terms meant. I hope that was okay.
(Note) Take the survey before reading further.
Basic assumptions that I found amusing: questions comparing me to those I grew up with. I was homeschooled, so most of the people I grew up with were family.
It would be nice to list the answers to the confidence questions after the survey was submitted. Now I have to go look them up. :)
All told, interesting questions and I look forward to the results.
Good suggestion about the general comments box. I added it. Thanks.
I’ll list the confidence answers in a couple days, after most have filled out the survey. I’ll also post the total results, both the raw data and my own analysis of it.
17 people have filled it out so far, which is wonderful. If people keep this up, we’ll get some really good data.
Awesome. Thanks for all the work.
If you do this again in the future, I have a list of potential questions that may be interesting to study. Do you want me to:
Send you a message
Post here to allow full discussion
Wait until you post the data and post so as to allow full discussion
Do nothing (because you are not planning on doing this again; you do not need suggestions; you do not want suggestions; some other reason)
Do something else
1 or 3 are good without reservation. 2 is good if you’re fairly sure it won’t prejudice people’s answers to the current survey questions.
If you opt for route 1, my email is annasalamon at gmail.
Some objections / suggestions from a non-American, not-very-young white nerdy male:
Are you romantically involved with anyone?
I’m married for about 12 years – does this count as a romantic involvement?
How satisfied are you with your current romantic arrangements, or lack of arrangements?
Again, this question should be broadened to include marriages. (Actually, this block of questions could explore more interesting opportunities, such as “What is your ratio of the number of romantic / sexual partners you had before marriage to your self-perceived sexual attractiveness rated 1 to 10?”)
How many friends have you talked to in the last week?
Is the purpose of this question to figure out how many friends do I have, or how much actual conversations did I have with my friends? What if I consciously cut back on chit-chat by switching my Skype / ICQ / etc to ‘Do not disturb’ or not even launching them on start-up?
Edit: Also, do you mean strictly personal friends? Are other kinds of conversation partners included? For example, I don’t chat much with personal friends, but have a lot of friendly conversations with co-workers (actually employees) -- does this count?
Have you ever been in an auto accident?
Does this question imply that I was a driver? I don’t drive (will explain later), but I have been in an auto accident.
Do you have a current driver’s license? Do most people your age, who live where you live, have a driver’s license?
Is having a driver license a universally good thing for a person in an environment where everyone else has one?
I don’t drive because the traffic accident mortality per million of population in Russia is 2.5 times higher than that of Europe (the Europe has about three times as much cars as Russia, so the per-car number is even more scary). That’s an official statistic (which traditionally gets lowered here, as showing real raw stats would ruin the public performance ratings of involved officials), and some sources give greater numbers like 15 to 16 times (which is pretty believable given how much morons and noobs I see on the roads every day).
Some anecdotal data points:
Three members of my (extended) family have been involved in traffic accidents that happened due to someone else’s disregard for traffic rules. For example, a father of my wife has spent a month in a hospital after being hit by a car while walking through a marked pedestrian crossing on his green light.
Every one of my driving co-workers regularly sees high-status people (huge black SUVs with license plates like “A-001-AA”) blatantly and openly violating traffic rules and getting away with that.
My co-worker has met an illiterate guy in a driving school. Yes, I mean that, the guy was illiterate and innumerate – he couldn’t dial a number on a mobile phone!
To sum up, I consider driving in today’s Russia to be too dangerous for me. I consciously minimize time I spend in proximity to cars and high-traffic roads, and when I need to get somewhere quick, I contact a tested driver or, if he is busy, my preferred taxi service (for which I also have some criteria).
As for my ability to obtain a driver’s license, I could do that easily – my father always was a driver, so I’m familiar with the inner workings of combustion engines since childhood, and my hand-eye coordination is more than enough for driving (I used to dominate the score table in Team Fortress 2 as almost any class until the combined wisdom of Tim Ferris and PJ Eby has eliminated that addiction :).
How many times have you exercised in the last week?
A better definition of ‘exercise’ would be helpful. For example, does it include long daily walks?
BTW, the issue with driving I outlined above can be generalized and addressed in the test as follows:
Form a (worldwide? weighted?) list of top 3 / 7 / 10 mortality reasons.
Include some questions measuring the person’s awareness and the effort spent on avoidance of these kinds of dangers (measuring the priorities would be good as well).
I also found “people I grew up with” strange. Just who these are? Average classmates?
I’m not still in contact with anyone I “grew up with”, except my significantly younger sister, so these questions threw me off too.
Thanks for telling me. I’ll clarify in the survey.