# 16 types of useful predictions

How of­ten do you make pre­dic­tions (ei­ther about fu­ture events, or about in­for­ma­tion that you don’t yet have)? If you’re a reg­u­lar Less Wrong reader you’re prob­a­bly fa­mil­iar with the idea that you should make your be­liefs pay rent by say­ing, “Here’s what I ex­pect to see if my be­lief is cor­rect, and here’s how con­fi­dent I am,” and that you should then up­date your be­liefs ac­cord­ingly, de­pend­ing on how your pre­dic­tions turn out.

And yet… my im­pres­sion is that few of us ac­tu­ally make pre­dic­tions on a reg­u­lar ba­sis. Cer­tainly, for me, there has always been a gap be­tween how use­ful I think pre­dic­tions are, in the­ory, and how of­ten I make them.

I don’t think this is just laz­i­ness. I think it’s sim­ply not a triv­ial task to find pre­dic­tions to make that will help you im­prove your mod­els of a do­main you care about.

At this point I should clar­ify that there are two main goals pre­dic­tions can help with:

1. Im­proved Cal­ibra­tion (e.g., re­al­iz­ing that I’m only cor­rect about Do­main X 70% of the time, not 90% of the time as I had mis­tak­enly thought).

2. Im­proved Ac­cu­racy (e.g., go­ing from be­ing cor­rect in Do­main X 70% of the time to be­ing cor­rect 90% of the time)

If your goal is just to be­come bet­ter cal­ibrated in gen­eral, it doesn’t much mat­ter what kinds of pre­dic­tions you make. So cal­ibra­tion ex­er­cises typ­i­cally grab ques­tions with eas­ily ob­tain­able an­swers, like “How tall is Mount Ever­est?” or “Will Don Draper die be­fore the end of Mad Men?” See, for ex­am­ple, the Cre­dence Game, Pre­dic­tion Book, and this re­cent post. And cal­ibra­tion train­ing re­ally does work.

But even though mak­ing pre­dic­tions about trivia will im­prove my gen­eral cal­ibra­tion skill, it won’t help me im­prove my mod­els of the world. That is, it won’t help me be­come more ac­cu­rate, at least not in any do­mains I care about. If I an­swer a lot of ques­tions about the heights of moun­tains, I might be­come more ac­cu­rate about that topic, but that’s not very helpful to me.

So I think the difficulty in pre­dic­tion-mak­ing is this: The set {ques­tions whose an­swers you can eas­ily look up, or oth­er­wise ob­tain} is a small sub­set of all pos­si­ble ques­tions. And the set {ques­tions whose an­swers I care about} is also a small sub­set of all pos­si­ble ques­tions. And the in­ter­sec­tion be­tween those two sub­sets is much smaller still, and not eas­ily iden­ti­fi­able. As a re­sult, pre­dic­tion-mak­ing tends to seem too effort­ful, or not fruit­ful enough to jus­tify the effort it re­quires.

But the in­ter­sec­tion’s not empty. It just re­quires some strate­gic thought to de­ter­mine which an­swer­able ques­tions have some bear­ing on is­sues you care about, or—ap­proach­ing the prob­lem from the op­po­site di­rec­tion—how to take is­sues you care about and turn them into an­swer­able ques­tions.

I’ve been mak­ing a con­certed effort to hunt for mem­bers of that in­ter­sec­tion. Here are 16 types of pre­dic­tions that I per­son­ally use to im­prove my judg­ment on is­sues I care about. (I’m sure there are plenty more, though, and hope you’ll share your own as well.)

1. Pre­dict how long a task will take you. This one’s a given, con­sid­er­ing how com­mon and im­pact­ful the plan­ning fal­lacy is.
Ex­am­ples: “How long will it take to write this blog post?” “How long un­til our com­pany’s prof­itable?”

2. Pre­dict how you’ll feel in an up­com­ing situ­a­tion. Affec­tive fore­cast­ing – our abil­ity to pre­dict how we’ll feel – has some well known flaws.
Ex­am­ples: “How much will I en­joy this party?” “Will I feel bet­ter if I leave the house?” “If I don’t get this job, will I still feel bad about it two weeks later?”

One thing this helps me no­tice is when I’ve been try­ing the same kind of ap­proach re­peat­edly with­out suc­cess. Even just the act of mak­ing the pre­dic­tion can spark the re­al­iza­tion that I need a bet­ter game plan.
Ex­am­ples: “Will I stick to my work­out plan for at least a month?” “How well will this event I’m or­ga­niz­ing go?” “How much work will I get done to­day?” “Can I suc­cess­fully con­vince Bob of my opinion on this is­sue?”

4. Pre­dict how your au­di­ence will re­act to a par­tic­u­lar so­cial me­dia post (on Face­book, Twit­ter, Tum­blr, a blog, etc.).
This is a good way to hone your judg­ment about how to cre­ate suc­cess­ful con­tent, as well as your un­der­stand­ing of your friends’ (or read­ers’) per­son­al­ities and wor­ld­views.
Ex­am­ples: “Will this video get an un­usu­ally high num­ber of likes?” “Will link­ing to this ar­ti­cle spark a fight in the com­ments?”

5. When you try a new ac­tivity or tech­nique, pre­dict how much value you’ll get out of it.
I’ve no­ticed I tend to be in­ac­cu­rate in both di­rec­tions in this do­main. There are cer­tain kinds of life hacks I feel sure are go­ing to solve all my prob­lems (and they rarely do). Con­versely, I am overly skep­ti­cal of ac­tivi­ties that are out­side my com­fort zone, and of­ten end up pleas­antly sur­prised once I try them.
Ex­am­ples: “How much will Po­modoros boost my pro­duc­tivity?” “How much will I en­joy swing danc­ing?”

6. When you make a pur­chase, pre­dict how much value you’ll get out of it.
Re­search on money and hap­piness shows two main things: (1) as a gen­eral rule, money doesn’t buy hap­piness, but also that (2) there are a bunch of ex­cep­tions to this rule. So there seems to be lots of po­ten­tial to im­prove your pre­dic­tion skill here, and spend your money more effec­tively than the av­er­age per­son.
Ex­am­ples: “How much will I wear these new shoes?” “How of­ten will I use my club mem­ber­ship?” “In two months, will I think it was worth it to have re­painted the kitchen?” “In two months, will I feel that I’m still get­ting plea­sure from my new car?”

7. Pre­dict how some­one will an­swer a ques­tion about them­selves.
I of­ten no­tice as­sump­tions I’m been mak­ing about other peo­ple, and I like to check those as­sump­tions when I can. Ideally I get in­ter­est­ing feed­back both about the ob­ject-level ques­tion, and about my over­all model of the per­son.
Ex­am­ples: “Does it bother you when our meet­ings run over the sched­uled time?” “Did you con­sider your­self pop­u­lar in high school?” “Do you think it’s okay to lie in or­der to pro­tect some­one’s feel­ings?”

8. Pre­dict how much progress you can make on a prob­lem in five min­utes.
I of­ten have the im­pres­sion that a prob­lem is in­tractable, or that I’ve already worked on it and have con­sid­ered all of the ob­vi­ous solu­tions. But then when I de­cide (or when some­one prompts me) to sit down and brain­storm for five min­utes, I am sur­prised to come away with a promis­ing new ap­proach to the prob­lem.
Ex­am­ple: “I feel like I’ve tried ev­ery­thing to fix my sleep, and noth­ing works. If I sit down now and spend five min­utes think­ing, will I be able to gen­er­ate at least one new idea that’s promis­ing enough to try?”

Me­mory is awfully fal­lible, and I have been sur­prised at how of­ten I am un­able to gen­er­ate spe­cific ex­am­ples to sup­port a con­fi­dent im­pres­sion of mine (or how of­ten the spe­cific ex­am­ples I gen­er­ate ac­tu­ally con­tra­dict my im­pres­sion).
Ex­am­ples: “I have the im­pres­sion that peo­ple who leave academia tend to be glad they did. If I try to list a bunch of the peo­ple I know who left academia, and how happy they are, what will the ap­prox­i­mate ra­tio of happy/​un­happy peo­ple be?”
″It feels like Bob never takes my ad­vice. If I sit down and try to think of ex­am­ples of Bob tak­ing my ad­vice, how many will I be able to come up with?”

10. Pick one ex­pert source and pre­dict how they will an­swer a ques­tion.
This is a quick short­cut to test­ing a claim or set­tling a dis­pute.
Ex­am­ples: “Will Cochrane Med­i­cal sup­port the claim that Vi­tamin D pro­motes hair growth?” “Will Bob, who has run sev­eral com­pa­nies like ours, agree that our start­ing salary is too low?”

11. When you meet some­one new, take note of your first im­pres­sions of him. Pre­dict how likely it is that, once you’ve got­ten to know him bet­ter, you will con­sider your first im­pres­sions of him to have been ac­cu­rate.
A var­i­ant of this one, sug­gested to me by CFAR alum Lau­ren Lee, is to make pre­dic­tions about some­one be­fore you meet him, based on what you know about him ahead of time.
Ex­am­ples: “All I know about this guy I’m about to meet is that he’s a banker; I’m mod­er­ately con­fi­dent that he’ll seem cocky.” “Based on the one con­ver­sa­tion I’ve had with Lisa, she seems re­ally in­sight­ful – I pre­dict that I’ll still have that im­pres­sion of her once I know her bet­ter.”

12. Pre­dict how your Face­book friends will re­spond to a poll.
Ex­am­ples: I of­ten post so­cial eti­quette ques­tions on Face­book. For ex­am­ple, I re­cently did a poll ask­ing, “If a con­ver­sa­tion is go­ing awk­wardly, does it make things bet­ter or worse for the other per­son to com­ment on the awk­ward­ness?” I con­fi­dently pre­dicted most peo­ple would say “worse,” and I was wrong.

13. Pre­dict how well you un­der­stand some­one’s po­si­tion by try­ing to para­phrase it back to him.
The illu­sion of trans­parency is per­ni­cious.
Ex­am­ples: “You said you think run­ning a work­shop next month is a bad idea; I’m guess­ing you think that’s be­cause we don’t have enough time to ad­ver­tise, is that cor­rect?”
″I know you think eat­ing meat is morally un­prob­le­matic; is that be­cause you think that an­i­mals don’t suffer?”

14. When you have a dis­agree­ment with some­one, pre­dict how likely it is that a neu­tral third party will side with you af­ter the is­sue is ex­plained to her.
For best re­sults, don’t re­veal which of you is on which side when you’re ex­plain­ing the is­sue to your ar­biter.
Ex­am­ple: “So, at work to­day, Bob and I dis­agreed about whether it’s ap­pro­pri­ate for in­terns to at­tend hiring meet­ings; what do you think?”

15. Pre­dict whether a sur­pris­ing piece of news will turn out to be true.
This is a good way to hone your bul­lshit de­tec­tor and im­prove your over­all “com­mon sense” mod­els of the world.
Ex­am­ples: “This head­line says some sci­en­tists up­loaded a worm’s brain—af­ter I read the ar­ti­cle, will the head­line seem like an ac­cu­rate rep­re­sen­ta­tion of what re­ally hap­pened?”
″This viral video pur­ports to show strangers be­ing prompted to kiss; will it turn out to have been staged?”

16. Pre­dict whether a quick on­line search will turn up any cred­ible sources sup­port­ing a par­tic­u­lar claim.
Ex­am­ple: “Bob says that watches always stop work­ing shortly af­ter he puts them on – if I spend a few min­utes search­ing on­line, will I be able to find any cred­ible sources say­ing that this is a real phe­nomenon?”

I have one ad­di­tional, gen­eral thought on how to get the most out of pre­dic­tions:

Ra­tion­al­ists tend to fo­cus on the im­por­tance of ob­jec­tive met­rics. And as you may have no­ticed, a lot of the ex­am­ples I listed above fail that crite­rion. For ex­am­ple, “Pre­dict whether a fight will break out in the com­ments? Well, there’s no ob­jec­tive way to say whether some­thing offi­cially counts as a ‘fight’ or not…” Or, “Pre­dict whether I’ll be able to find cred­ible sources sup­port­ing X? Well, who’s to say what a cred­ible source is, and what counts as ‘sup­port­ing’ X?”

And in­deed, ob­jec­tive met­rics are prefer­able, all else equal. But all else isn’t equal. Sub­jec­tive met­rics are much eas­ier to gen­er­ate, and they’re far from use­less. Most of the time it will be clear enough, once you see the re­sults, whether your pre­dic­tion ba­si­cally came true or not—even if you haven’t pinned down a pre­cise, ob­jec­tively mea­surable suc­cess crite­rion ahead of time. Usu­ally the re­sult will be a com­mon sense “yes,” or a com­mon sense “no.” And some­times it’ll be “um...sort of?”, but that can be an in­ter­est­ingly sur­pris­ing re­sult too, if you had strongly pre­dicted the re­sults would point clearly one way or the other.

Along similar lines, I usu­ally don’t as­sign nu­mer­i­cal prob­a­bil­ities to my pre­dic­tions. I just take note of where my con­fi­dence falls on a qual­i­ta­tive “very con­fi­dent,” “pretty con­fi­dent,” “weakly con­fi­dent” scale (which might cor­re­spond to some­thing like 90%/​75%/​60% prob­a­bil­ities, if I had to put num­bers on it).

There’s prob­a­bly some ad­di­tional value you can ex­tract by writ­ing down quan­ti­ta­tive con­fi­dence lev­els, and by de­vis­ing ob­jec­tive met­rics that are im­pos­si­ble to game, rather than just rely­ing on your sub­jec­tive im­pres­sions. But in most cases I don’t think that ad­di­tional value is worth the cost you in­cur from turn­ing pre­dic­tions into an oner­ous task. In other words, don’t let the perfect be the en­emy of the good. Or in other other words: the biggest prob­lem with your pre­dic­tions right now is that they don’t ex­ist.

• I’ve es­tab­lished a habit of putting my money where my mouth is to en­courage my­self to make more firm pre­dic­tions. When I am talk­ing with some­one and we dis­agree, I ask if they want to bet a dol­lar on it. For ex­am­ple, I say, “Roger Ebert di­rected Beyond the Valley of the Dolls”. My wife says, “No, he wrote it.”. Then I offer to bet. We look up the an­swer, and I give her a dol­lar.

This is a good habit for many rea­sons.

1. It is fun to bet. Fun to win. And (kinda) fun to lose.

2. It forces peo­ple to eval­u­ate hon­estly. The same peo­ple that say “I’m sure...” will back off their point when asked to bet a dol­lar on the out­come.

3. It forces peo­ple to ne­go­ti­ate to con­crete terms. For ex­am­ple, I told a friend that a 747 burns 30,000 lbs of fuel an hour. He said no way. We fi­nally set­tled on the bet “Ben thinks that a fully loaded 747 will burn more than 10,000 lbs of fuel per hour un­der stan­dard cruis­ing con­di­tions”. (I won that bet, it burns ~25,000 lbs of fuel/​hour un­der these con­di­tions).

4. A dol­lar feels more im­por­tant than it ac­tu­ally is, so peo­ple treat the bets se­ri­ously even though they are not very se­ri­ous. For this rea­son, I think it is im­por­tant to ac­tu­ally ex­change a dol­lar bill at the end, rather than treat­ing it as just an ab­stract dol­lar.

I’ve learned a lot from this habit.

1. I’m right more of­ten than not (~75%). But I’m wrong a lot too (~25%). This is more wrong than I feel. I feel 95% con­fi­dent. I shouldn’t be so con­fi­dent.

2. The per­son propos­ing the bet is usu­ally right. My wife has got­ten in the habit too. If I pro­pose we bet, I’m usu­ally right. If she pro­poses we bet I’ve learned to usu­ally back down.

3. Peo­ple fre­quently over­state their con­fi­dence. I men­tioned this above, but it bears re­peat­ing. Some peo­ple reg­u­larly will use phrases like “I am sure” or say some­thing em­phat­i­cally. Peo­ple are cal­ibrated to ex­press their be­liefs differ­ently. But when you ask them to bet a dol­lar you get a more con­sis­tent cal­ibra­tion. Peo­ple that are over-con­fi­dent of­ten back away from their po­si­tions. Really in­ter­est­ing con­sid­er­ing that its only a dol­lar on the line.

4. Over time peo­ple learn to cal­ibrate bet­ter. At first my wife would agree to nearly ev­ery bet I pro­posed. Now she usu­ally doesn’t want to. When she agrees to a bet now, I get wor­ried.

• The per­son propos­ing the bet is usu­ally right.

This is a cru­cial ob­ser­va­tion if you are try­ing to use this tech­nique to im­prove your cal­ibra­tion of your own ac­cu­racy! You can’t just start mak­ing bets when no one else you as­so­ci­ate reg­u­larly is challeng­ing you to the bets.

Sev­eral years ago, I started tak­ing note of all of the times I dis­agreed with other peo­ple and look­ing it up, but ini­tially, I only counted my­self as hav­ing “dis­agreed with other peo­ple” if they said some­thing I thought was wrong, and I at­tempted to cor­rect them. Then I soon added in the case when they cor­rected me and I ar­gued back. Dur­ing this pe­riod of time, I went from think­ing I was about 90% ac­cu­rate in my claims to be­liev­ing I was way more ac­cu­rate than that. I would go months with­out be­ing wrong, and this was in col­lege, so I was fre­quently get­ting into dis­agree­ments with peo­ple, prob­a­bly, an av­er­age, three a day dur­ing the school year. Then I started check­ing the times that other peo­ple cor­rected me, just as much as I checked when I cor­rected other peo­ple. (Count­ing even the times that I made no at­tempt to ar­gue.) And my ac­cu­racy rate plum­meted.

Another thing I would recom­mend to peo­ple start­ing out in do­ing this is that you should keep track of your record with in­di­vi­d­ual peo­ple not just your gen­eral over­all record. My ac­cu­racy rate with a few peo­ple is way lower than my over­all ac­cu­racy rate. My over­all rate is higher than it should be be­cause I know a few ar­gu­men­ta­tive peo­ple who are fre­quently wrong. (This would prob­a­bly change if we were ac­tu­ally bet­ting money, and we were only count­ing ar­gu­ments when those peo­ple were will­ing to bet. So you’re ap­proach ad­justs for this bet­ter than mine.) I have sev­eral peo­ple for whom I’m close to 50%, and there are two peo­ple for whom I have sev­eral data points and my over­all ac­cu­racy is be­low 50%.

There’s one other point I think some­body needs to make about cal­ibra­tion. And that’s that 75% ac­cu­racy when you dis­agree with other peo­ple is not the same thing as 75% ac­cu­racy. 75% in­for­ma­tion fidelity is atro­cious; 95% in­for­ma­tion fidelity is not much bet­ter. Hu­man brains are very defec­tive in a lot of ways, but they aren’t that defec­tive! Ex­cept at do­ing math. Brains are ridicu­lously bad at math rel­a­tive to how eas­ily ma­chines can be im­ple­mented to be good at math. For most in­tents and pur­poses, 99% isn’t a very high per­centage. I am not a par­tic­u­lar good driver, but I haven’t got­ten into a col­li­sion with an­other ve­hi­cle in my well over 1000 times driv­ing. Per­centages tend to have an ex­po­nen­tial scale to them (or more ac­cu­rately a lo­gis­tic curve). You don’t have to be a par­tic­u­larly good driver to avoid get­ting into an ac­ci­dent 99.9% of the time you get be­hind the wheel, be­cause that is just a few or­ders of mag­ni­tude im­prove­ment rel­a­tive to 50%.

In­for­ma­tion fidelity differs from in­for­ma­tion re­ten­tion. Dis­card­ing 25% or 95% or more of col­lected in­for­ma­tion is rea­son­able; cor­rupt­ing in­for­ma­tion at that rate is what I’m say­ing would be hor­ren­dous. (Be­cause dis­card­ing in­for­ma­tion con­serves re­sources; whereas cor­rupt­ing in­for­ma­tion does not… ex­cept to the ex­tent that you would con­sider com­press­ing in­for­ma­tion with a lossy (as in “not lossless”) com­pres­sion to be a cor­rupt­ing in­for­ma­tion, but I would still con­sider that to be dis­card­ing in­for­ma­tion. Epi­sodic mem­ory is ei­ther very com­pressed or very cor­rupted de­pend­ing on what you think it should be.)

In my ex­pe­rience, peo­ple are ac­tu­ally more likely to be un­der­con­fi­dent about fac­tual in­for­ma­tion than they are to be over­con­fi­dent, if you mea­sure con­fi­dence on an ab­solute scale in­stead of a rel­a­tive-to-other-peo­ple scale. My fam­ily goes to trivia night, and we al­most always get at least as many cor­rect as we ex­pect to get cor­rect, usu­ally more. How­ever, other teams typ­i­cally score bet­ter than we ex­pect them to score too, and we win the round less of­ten than we ex­pect to.

Think back to grade school when you ac­tu­ally had fill in the blank and mul­ti­ple choice ques­tions on tests. I’m go­ing to guess that you prob­a­bly were an A stu­dent and got around 95% right on your tests… be­cause a) that’s about what I did and I tend to pro­ject, b) you’re on LessWrong so you were prob­a­bly an A stu­dent, and C) you say you feel like you ought to be right about 95% of the time. I’m also go­ing to guess (be­cause I tend to pro­ject my ex­pe­rience onto other peo­ple) that you prob­a­bly felt a lot less than 95% con­fi­dent on av­er­age when you were tak­ing the tests. There were more than a few tests I took in my time in school where I walked out of the test think­ing “I didn’t know any of that; I’ll prob­a­bly get a 70 or bet­ter just be­cause that would be hor­ribly bad com­pared to what I usu­ally do, but I re­ally feel like I failed that”… and it was never 70. (Math was the one ex­cep­tion in which I tended to be over­con­fi­dent, I usu­ally made more mis­takes than I ex­pected to make on my math tests.)

Where cal­ibra­tion is re­ally screwed up is when you deal with sub­jects that are way out­side of the do­main of nor­mal ex­pe­rience, es­pe­cially if you know that you know more than your peer group about this do­main. Peo­ple are not good at think­ing about ab­stract math­e­mat­ics, ar­tifi­cial in­tel­li­gence, physics, evolu­tion, and other sub­jects that hap­pen at a differ­ent scale from nor­mal ev­ery­day life. When I was 17, I thought I un­der­stood Quan­tum Me­chan­ics just be­cause I’d read A Brief His­tory of Time and A Uni­verse in a Nut Shell… Boy was I wrong!

On LessWrong, we are usu­ally dis­cussing sub­jects that are way be­yond the do­main of nor­mal hu­man ex­pe­rience, so we tend to be over­con­fi­dent in our un­der­stand­ing of these sub­jects… but part of the rea­son for this over­con­fi­dence is that we do tend to be cor­rect about most of the things we en­counter within the con­fines of rou­tine ex­pe­rience.

• I was read­ing your com­ment, and when I thought about always bet­ting a dol­lar, my brain went, “That’s a good idea!”

So I asked my brain, “What mem­ory are you ac­cess­ing that makes you think that’s a good idea?”

And my brain replied, “Re­mem­ber that CFAR read­ing list you’re go­ing through? Yeah, that one.”

So I went to my book­shelf, got out Dan Ariely’s Pre­dictably Ir­ra­tional, and started pag­ing through it.

Pro­fes­sor Ariely had sev­eral in­sights that helped me un­der­stand why ac­tu­ally us­ing money seemed like such a good idea:

1. In­ter­act­ing within mar­ket norms makes you do a cost-benefit anal­y­sis. Pro­fes­sor Ariely dis­cusses the differ­ence be­tween so­cial norms and mar­ket norms in chap­ter 4. So­cial norms gov­ern in­ter­ac­tions that don’t in­volve money (fa­vors for a friend), and mar­ket norms gov­ern in­ter­ac­tions that do (costs and benefits). The pro­fes­sor did an ex­per­i­ment in which he had peo­ple drag cir­cles into a square on a com­puter screen (judg­ing their pro­duc­tivity by how many times they did this in a set pe­riod of time). He gave one group of such par­ti­ci­pants an ex­plic­itly stated “50-cent snick­ers bar” and the other a “five-dol­lar box of Go­diva choco­lates.” As it turns out, the re­sults were iden­ti­cal to a pre­vi­ous ex­per­i­ment in which the same amounts of di­rect cash were used. Pro­fes­sor Ariely con­cludes, “Th­ese re­sults show that for mar­ket norms to emerge, it is suffi­cient to men­tion money.” In other words, Pro­fes­sor Ariely’s re­search sup­ports your first (4.) - “A dol­lar feels more im­por­tant than it ac­tu­ally is...” This is the case be­cause as soon as money en­ters the pic­ture, so do mar­ket norms.

2. Money makes us hon­est. In chap­ter 14, aptly ti­tled “Why Deal­ing With Cash Makes Us More Hon­est,” Pro­fes­sor Ariely ex­plains an ex­per­i­ment he con­ducted in the MIT cafe­te­ria. Stu­dents were given a sheet of 20 math prob­lems to solve in five min­utes. The con­trol group was to have their solu­tions checked, and then were given 50 cents per cor­rect an­swer. A sec­ond group was in­structed to tear their pa­per apart, and then tell the ex­per­i­menter how many ques­tions they got cor­rect (al­low­ing them to cheat). They were then paid 50 cents for ev­ery cor­rect an­swer they claimed. Lastly, a third group was al­lowed to cheat similarly to the sec­ond group, ex­cept that when they gave one ex­per­i­menter their score, they were given to­kens, which were traded in im­me­di­ately there­after for cash through a sec­ond ex­per­i­menter. The re­sults: A) The con­trol group solved an av­er­age of 3.5 ques­tions cor­rectly. B) The sec­ond group, who cheated for cash, claimed an av­er­age of 6.2 cor­rect solu­tions. C) The third group, who cheated for to­kens, claimed an av­er­age of 9.4 cor­rect solu­tions. Sim­ply put, when ac­tual, phys­i­cal money was re­moved from the sub­jects’ thought pro­cess by a to­ken and a few sec­onds, the amount of cheat­ing more than dou­bled, from 2.7 to 5.9.

In short, us­ing money to back a pre­dic­tion a) forces us to think an­a­lyt­i­cally, and b) keeps us hon­est.

Thank you for the idea. Now I just need to find an ATM to get some ones...

• This ex­per­i­ment does not prove that money keeps peo­ple more hon­est than ab­sence of money, but more hon­est than to­ken ex­change­able for money. If a con­trol group was al­lowed to cheat with­out re­ceiv­ing money at all they might (my pre­dic­tion and I would bet a dol­lar on it if I didn’t use Euros) cheat even less. Then, the hy­poth­e­sis “money keeps us hon­est” would be dis­proved.

• I think I re­mem­ber de­scribed set of ex­per­i­ments cor­rectly and at least in some of them con­trol group was definitely al­lowed to cheat—there were no differ­ence in the way peo­ple turned in their re­sults (shred­ding ques­tionare and sub­mit­ting only pur­ported re­sult on differ­ent sheet)

• A dol­lar feels more im­por­tant than it ac­tu­ally is, so peo­ple treat the bets se­ri­ously even though they are not very se­ri­ous.

Although there is a weight in the dol­lar, I think there is also an­other rea­son why peo­ple take it more se­ri­ously. Peo­ple ad­just their be­lieve ac­cord­ing to what other peo­ple be­lieve and their con­fi­dence level. There­fore, when you pro­pose a bet, even only for a dol­lar, you are show­ing a high con­fi­dence level and this de­crease their con­fi­dence level. As a re­sult, sys­tem 2 kicks in and they will be > [forced] to eval­u­ate hon­estly.

• I love the list of pre­dic­tions, but I also feel fairly con­fi­dent in pre­dict­ing that this post won’t prompt me to ac­tu­ally make more (or more use­ful) pre­dic­tions. Do you have any tips on build­ing the habit of mak­ing pre­dic­tions?

• My plan is to pick one of these and pre­dict like hell, be­cause six­teen pos­si­bil­ities makes this look more daunt­ing than it is, and then I plan to move on to oth­ers or come up with my own when it’s semi-ha­bit­ual. I did a sort of method of elimi­na­tion to pick my ini­tial habit-former. I’m not go­ing to list my en­tire pro­cess, un­less some­one re­ally wants me to, but I set­tled on #13 be­cause:

1. I already do some­thing similar to this. All I have to do is af­fix a Pre­dict step to the be­gin­ning of my al­gorithm.

2. Suc­cess­ful and un­suc­cess­ful pre­dic­tions are very clear cut, which is good for get­ting Sys­tem 1′s at­ten­tion and suc­cess­fully form­ing the habit.

3. It has few, if any, triv­ial in­con­ve­niences.

4. It’s not as­so­ci­ated with an ac­tivity that has an ugh field around it for me.

I haven’t been here long, but I feel like I’ve seen com­ments like this a lot, of the form, “This is all Well and Good, but how pre­cisely do I re­ally, ac­tu­ally im­ple­ment this?” And some­times that’s a valid point. Some­times what looks like a sug­ges­tion is all noise and no sig­nal (or all sig­nal? Hell, I don’t know).

Other times some­one is apt to say, “I pre­dict that this post will not re­sult in ac­tual im­prove­ment of Prob­lem X,” in Ra­tion­als­peak and all, and they are uni­ver­sally ac­claimed. For you see, in this way, they and all of their fol­low­ers are above the fray.

When the sug­ges­tions are good, I’ve always felt that it’s a bit in­con­sid­er­ate to im­me­di­ately ping the Bur­den of Thought back at some­one who just wrote a blog post on a site as crit­i­cal as LessWrong. I share things when my brain sput­ters out and can’t go any fur­ther on its own; it needs help from the tribe. Did you or the peo­ple who up­voted you think for five min­utes by the clock about how you might form a pre­dict­ing habit be­fore out­sourc­ing the ques­tion to Ju­lia_Galef? It’s awfully easy to never form a pre­dict­ing habit when the Pres­i­dent of CFAR con­ve­niently never gets back to you! You also could have #8-ed and pre­dicted whether or not you would be able to come up with a way to build a pre­dict­ing habit.

I sort of (read: com­pletely) made an ex­am­ple out of you for com­mu­nity’s sake, so sorry if I was hurt­ful or oth­er­wise. This is not all di­rected at you in par­tic­u­lar; it was an op­por­tu­nity to make an ob­ser­va­tion.

• Fair call on my in­tel­lec­tual laz­i­ness in not perform­ing the brain­storm­ing my­self. Point taken. How­ever, if you are notic­ing a pat­tern of many peo­ple do­ing this over time, it seems like this is some­thing ar­ti­cle au­thors could take into ac­count to get more im­pact out of their ar­ti­cles. Un­less the point is to make the per­son read­ing do the brain­storm­ing to build that habit, then the time of many read­ers can be saved by the per­son who wrote the ar­ti­cle, and pre­sum­ably has already passed this point and thus put in the time shar­ing tips or a call to ac­tion on how to get started.

I want to stress that I don’t con­sider this an obli­ga­tion on the ar­ti­cle au­thor. If Ju­lia, or any­one else, doesn’t want to put in that time, then we can be grate­ful (and I am) that they have shared with us the time they have already. How­ever, I do view it as an op­por­tu­nity for au­thors who wish to have a greater im­pact.

On a a more con­crete level, thanks for shar­ing your thought pro­cess on this topic. Very use­ful.

• Ap­par­ently you are putting

2. Pre­dict how you’ll feel in an up­com­ing situ­a­tion. Affec­tive fore­cast­ing – our abil­ity to pre­dict how we’ll feel – has some well known flaws.
Ex­am­ples: “How much will I en­joy this party?” “Will I feel bet­ter if I leave the house?” “If I don’t get this job, will I still feel bad about it two weeks later?”

into your “Easily an­swer­able ques­tions” sub­set. Per­son­ally, I strug­gle to ob­tain a level of in­tro­spec­tion suffi­cient to an­swer ques­tions like these even af­ter the fact.

Does any­one have any tips to help me bet­ter ac­cess my own feel­ings in this way? After I have left the house, how do I de­ter­mine if I feel bet­ter? If I don’t get the job, how do I de­ter­mine if I feel bad about it? Etc.

• Does any­one have any tips to help me bet­ter ac­cess my own feel­ings in this way? After I have left the house, how do I de­ter­mine if I feel bet­ter? If I don’t get the job, how do I de­ter­mine if I feel bad about it? Etc.

Fo­cus­ing. Learn to lo­cate feel­ings in your body and learn to put la­bels on emo­tions.

Hav­ing QS rit­u­als where you put down an es­ta­ma­tion of your in­ter­nal state ev­ery day. I for ex­am­ple write down my dom­i­nat­ing mood for the last 24 hours and a few num­bers.

As a weekly rit­ual it’s also pos­si­ble to fill out longer ques­tion­aires such as http://​​www.con­nec­tions-ther­apy-cen­ter.com/​​up­load/​​burns_anx­iety_in­ven­tory.pdf .

It’s like a mus­cle. If you train to as­sess your in­ter­nal state you get bet­ter.

• I find that play­ing the pi­ano is a par­tic­u­larly use­ful tech­nique for gaug­ing my emo­tions, when they are sup­pressed/​muted. This works bet­ter when I’m just mak­ing stuff up by ear than it does when I’m play­ing some­thing I know or read­ing mu­sic. (And learn­ing to make stuff up is a lot eas­ier than learn­ing to read mu­sic if you don’t already play.) Play­ing the pi­ano does not help me feel the emo­tions any more strongly, but it does let me hear them—I can tell that mu­sic is sad, happy, or an­gry re­gard­less of its im­pact on my af­fect. Most peo­ple can.

Some­thing that I don’t do that I think would work (based par­tially on what Ariely says in The Up­side of Ir­ra­tional­ity, par­tially on what Nor­man says in Emo­tional De­sign, and par­tially on anec­do­tal ex­pe­rience) is to do some­thing challeng­ing/​frus­trat­ing and see how long it takes for you to give up or get an­gry. If you can do it for a while with­out get­ting frus­trated, you’re prob­a­bly in a pos­i­tive state of mind. If you give up feel­ing like it’s fu­tile, you’re sad, and if you start feel­ing an im­pulse to break some­thing, you’re frus­trated/​an­gry. The shorter it takes you to give up or an­gry the stronger that emo­tion is. The huge down­side to this ap­proach is that it re­sults in ex­ac­er­bat­ing nega­tive emo­tions (tem­porar­ily) in or­der to gauge what you were feel­ing and how strongly.

• Hi!

1) If I un­der­stood Ju­lia cor­rectly “eas­ily an­swer­able ques­tions” cor­re­spond not to ar­eas where you are good at pre­dict­ing, but to ar­eas where an­swer space is known to you: “Can I toss the ball through the hoop?”—Yes\No vs. “What is the best pre­sent for a teenage girl?” ??\??\money??\puppy??

2) If you have difficul­ties with as­so­ci­at­ing com­mon groups with your feel­ings or even per­cieve feel­ings that is re­ally con­fus­ing and it would be good not to jump to the con­clu­sions, but to add to other com­menters: you could prob­a­bly start by ask­ing out­side ob­server (i.e body lan­guage “com­fort\defen­sive­ness” “happy\sad”)

• Hm.
Are there any con­texts in which you do have re­li­able in­sight into your own mood?

• Does mak­ing a lot of pre­dic­tions cause any stress? To me it does, al­most like a sense that I’m always “on duty”. Also, I’m a bit eas­ily dis­tracted and it could dis­tract me from the task at hand. So I try to strike some sort of bal­ance. Thoughts?

Also, some­times pre­dic­tions nega­tively af­fect the amount of en­joy­ment I get. Ex. I’m a big bas­ket­ball fan and I do love mak­ing pre­dic­tions in bas­ket­ball, but some­times it isn’t fun to have pre­dicted that the team I like is go­ing to lose.

• I like this post. I lean to­wards skep­ti­cism about the use­ful­ness of cal­ibra­tion or even ac­cu­racy, but I’m glad to find my­self mostly in agree­ment here.

For lots of prac­ti­cal (to me) situ­a­tions, a lit­tle bit of un­cer­tainty goes a long way con­cern­ing how I ac­tu­ally de­cide what to do. It doesn’t re­ally mat­ter how much un­cer­tainty, or how well I can es­ti­mate the un­cer­tainty. It’s bet­ter for me to just be gen­er­ally hum­ble and make con­tin­gency plans. It’s also easy to imag­ine that be­ing well-cal­ibrated (or know­ing that you are) could ac­tu­ally de­mol­ish bi­ases that are ac­tu­ally pro­tec­tive against bad out­comes, if you’re not care­ful. If you are care­ful, sure, there are pos­si­ble benefits, but they seem mod­est.

But mak­ing and test­ing pre­dic­tions seems more than mod­estly use­ful, whether or not you get bet­ter (or bet­ter cal­ibrated) over time. I find I learn bet­ter (test­ing effect!) and I’m more likely to no­tice sur­pris­ing things. And it’s an easy way to lamp­shade cer­tain thoughts/​de­ci­sions so that I put more effort into them. Ba­si­cally, this:

Or in other other words: the biggest prob­lem with your pre­dic­tions right now is that they don’t ex­ist.

To be more con­crete, a while back I ac­tu­ally ran a self-ex­per­i­ment on quan­ti­ta­tive cal­ibra­tion for time-track­ing/​plan­ning (your point #1). The idea was to get a baseline by mak­ing and re­solv­ing pre­dic­tions with­out any feed­back for a few weeks (i.e. I didn’t know how well I was do­ing—I also made pre­dic­tions in batches so I usu­ally couldn’t re­mem­ber them and thus tar­get my pre­dic­tion “dead­lines”). Then I’d start look­ing at cal­ibra­tion curves and so on to see if feed­back might im­prove pre­dic­tions (in gen­eral or in par­tic­u­lar do­mains). It turned out af­ter the first stage that I was already well-cal­ibrated enough that I wouldn’t be able to mea­sure any in­ter­est­ing changes with­out an im­prac­ti­cal num­ber of pre­dic­tions, but while it lasted I got a mod­er­ate boost in pro­duc­tivity just from know­ing I had a clock tick­ing, plus more effec­tive plan­ning from the way pre­dic­tions forced me to think about con­tin­gen­cies. (I stopped the ex­per­i­ment be­cause it was te­dious, but I upped the fre­quency of pre­dic­tions I make ha­bit­u­ally.)

• Perfor­mance pre­dic­tion is a bit tricky. You usu­ally care more about the perfor­mance than be­ing ac­cu­rate at pre­dict­ing. You don’t want to come into a situ­a­tion where you re­duce your perfor­mance to get your pre­dic­tion right.

• This is the rea­son I have mixed feel­ings about mak­ing pre­dic­tions of events that I can in­fluence. I’m cu­ri­ous whether there is any re­search about this ‘jinx­ing’ - does pre­dict­ing low chances of suc­cess at a task make peo­ple less likely to suc­ceed? Or (maybe) the op­po­site?

• re com­part­men­tal­iza­tion ques­tion about ‘jinx­ing’.

I have some ex­pe­rience and knowl­edge in this sub­ject from a sports sci­ence per­spec­tive.

It’s com­monly ac­cepted within sport psy­chol­ogy that first, nega­tivity, is as­so­ci­ated with pre­dict­ing low chances of suc­cess, and sec­ondly that those who do dis­play nega­tivity and pre­dict low chance of suc­cess de­crease their own perfor­mance.

For ex­am­ple, a well coached bas­ket­ball player at the free throw line would be aware that say­ing “I’m go­ing to miss this free throw” in­creases their chances of miss­ing the free throw. Note now that “well coached” im­plies in­clud­ing psy­cholog­i­cal train­ing as a com­po­nent of a wider train­ing pro­gram.

One source for you com­part­men­tal­iza­tion, to dig a lit­tle deeper is...

“Krane and Willi­ams con­cluded that a cer­tain psy­cholog­i­cal pro­file ap­pears to be cor­re­lated with peak perfor­mance for most ath­letes. More speci­fi­cally, this ideal mind/​body state con­sists of the fol­low­ing: (a) feel­ings of high self-con­fi­dence and ex­pec­ta­tions of suc­cess, (b) be­ing en­er­gized yet re­laxed, (c) feel­ing in con­trol,(d) be­ing to­tally con­cen­trated, (e) hav­ing a keen fo­cus on the pre­sent task, (f) hav­ing pos­i­tive at­ti­tudes and thoughts about perfor­mance, and (g) be­ing strongly de­ter­mined and com­mit­ted. Con­versely, the men­tal state typ­i­cally as­so­ci­ated with poorer perfor­mances in sport seems to be marked by feel­ings of self-doubt, lack­ing con­cen­tra­tion, be­ing dis­tracted, be­ing overly fo­cused on the com­pe­ti­tion out­come or score, and feel­ing overly or un­der aroused. While ac­knowl­edg­ing that this ideal mind/​body state is highly idiosyn­cratic, Krane and Willi­ams con­cluded that for most ath­letes, the pres­ence of the right men­tal and emo­tional state just de­scribed is as­so­ci­ated with them perform­ing to their po­ten­tial.” Harmi­son, R. J. (2006). Peak perfor­mance in sport: Iden­ti­fy­ing ideal perfor­mance states and de­vel­op­ing ath­letes’ psy­cholog­i­cal skills. Pro­fes­sional Psy­chol­ogy: Re­search and Prac­tice, 37(3), 233-243. doi: 10.1037/​0735-7028.37.3.233

• Microsoft Out­look Busi­ness Con­tact Man­ager pro­vides ways for­ward to util­is­ing pre­dic­tion. Within its Task schedul­ing one has op­por­tu­nity to es­ti­mate what per­centage of the set task is already com­pleted. Also how long the task will take is es­ti­mated by the user.

I find B.C.M highly use­ful for fo­cus­ing pre­dic­tion and task achieve­ment.

• Cer­tainly, for me, there has always been a gap be­tween how use­ful I think pre­dic­tions are, in the­ory, and how of­ten I make them.

Same here.

There’s prob­a­bly some ad­di­tional value you can ex­tract by writ­ing down quan­ti­ta­tive con­fi­dence lev­els, and by de­vis­ing ob­jec­tive met­rics that are im­pos­si­ble to game, rather than just rely­ing on your sub­jec­tive im­pres­sions.

I agree.

But in most cases I don’t think that ad­di­tional value is worth the cost you in­cur from turn­ing pre­dic­tions into an oner­ous task.

I dis­agree in that (1) I think much of the value of pre­dic­tions would come from the abil­ity to ex­am­ine and an­a­lyze my past pre­dic­tion ac­cu­racy and (2) I don’t think the task of record­ing the pre­dic­tions would nec­es­sar­ily be very oner­ous (e.g. es­pe­cially if there is some re­cur­ring pre­dic­tion which you don’t have to write a new de­scrip­tion for ev­ery time you make it).

I re­ally like Pre­dic­tion Book (which I just checked out for the first time be­fore Googling and find­ing this post), but it doesn’t offer suffi­cient anal­y­sis op­tions to make me want to re­ally be­gin us­ing it yet.

But this could change!

I would pre­dict (75%) that I would be­gin us­ing it on a daily ba­sis (and would con­tinue to do so in­definitely upon re­al­iz­ing that I was in­deed get­ting suffi­cient value out of it to jus­tify the time it takes to record my pre­dic­tions on the site) if it offered not just the sin­gle Ac­cu­racy vs 50-100% Con­fi­dence plot and graph, but the fol­low­ing fea­tures:

• Abil­ity to see con­fi­dence and ac­cu­racy plot­ted ver­sus time. (Use­ful, e.g. to see weekly progress on meet­ing some daily self-im­posed dead­line. Per­haps you used to meet it 60% of days on av­er­age, but now you meet it 80% of days on av­er­age. You could track your progress while see­ing if you ac­cu­rately pre­dict progress as well, or if your pre­dicted val­ues fol­low the im­prove­ment.)

• Abil­ity to see 0-100% con­fi­dence on statis­tics plot, in­stead of just 50-100%. (Maybe it already in­cludes 0-50% and just does the nega­tive of each pre­dic­tion (?). How­ever, if so, this is still a prob­lem since I may have differ­ent bi­ases for 10% pre­dic­tions than 90% pre­dic­tions.)

• Abil­ity to set differ­ent differ­ent pre­dic­tion types and an­a­lyze the data sep­a­rately. (Use­ful for learn­ing how ac­cu­rate one’s pre­dic­tions are in differ­ent do­mains.)

• Abil­ity to down­load all of one’s past pre­dic­tion data. (Use­ful if there is some spe­cial anal­y­sis that one wants to perform.)

• A pub­lic/​pri­vate pre­dic­tion tog­gle but­ton (Use­ful be­cause there may be times when it’s okay for some­one to hide a pre­dic­tion they were em­bar­rass­ingly wrong about or some­one may want to pub­li­cize a pre­vi­ously-pri­vate pre­dic­tion. Forc­ing users to de­cide at the time of the pre­dic­tion whether their pre­dic­tion will for­ever be dis­played pub­li­cly on their ac­count or re­main pri­vate for­ever doesn’t seem very user-friendly.)

• Bonus: An app al­low­ing easy data in­put when not at your com­puter. (Would make it even more con­ve­nient to record pre­dic­tions.)

Some of these fea­tures can be achieved by cre­at­ing mul­ti­ple ac­counts. And I could ac­com­plish all of this in Ex­cel. But us­ing mul­ti­ple ac­counts or Ex­cel would make it too te­dious to be worth it. The value is in hav­ing the graphs and anal­y­sis au­to­mat­i­cally gen­er­ated and pre­sented to you with only a small amount of effort needed to in­put the pre­dic­tions in the first place.

I don’t think any of these ad­di­tional fea­tures would be very difficult to im­ple­ment. How­ever, I’m not a pro­gram­mer, so for me to dive into the Pre­dic­tion Book GitHub and try to figure out how to make these changes would prob­a­bly be quite time-con­sum­ing and not worth it.

Maybe there is some­one else who agrees that these fea­tures would be use­ful to them who is a pro­gram­mer and would like to add some or all of the sug­gested fea­tures I men­tioned? Does any­one know the peo­ple who did most of the work pro­gram­ming the cur­rent web­site?

• You could always play ask a stranger a ques­tion.

• Last year I built a spread­sheet for helping with #1 (Pre­dict how long a task will take you) which I still use daily.

Du­ra­tion Cal­ibra­tion Template

You in­put the name of a task (with cat­e­go­riza­tion if you want) and a lower and up­per bound for how long you ex­pect it to take. Start a timer, do the thing, then put in the ac­tual time. If you were within the bounds, you get a ✓ in the next column, oth­er­wise you get your er­ror, green for over­es­ti­mate and red for un­der­es­ti­mate. I’ll prob­a­bly add some stuff that does ag­gre­gate stats on how of­ten you’re within the bounds (i.e. what ac­tual con­fi­dence in­ter­val those bounds rep­re­sent for you) but for now it’s mostly just the ex­pe­rience of do­ing it that’s helpful.

I think I’ve be­come bet­ter cal­ibrated over months of us­ing this, al­though some days it’s way off be­cause I don’t give it enough at­ten­tion. There’s a pretty big differ­ence be­tween a du­ra­tion es­ti­mate made in 1 sec­ond vs 5 sec­onds.

When I first tried this, I thought that it would be an­noy­ing, but I ac­tu­ally found it to be a re­ally en­joy­able way to do lots of differ­ent kinds of things, be­cause it kept me re­ally fo­cused on the spe­cific thing I was timing, rather than go­ing off on other tan­gents that felt pro­duc­tive but were un­fo­cused, or jump­ing around. In ad­di­tion to track­ing “tasks”, I would also some­times track things like read­ing a chap­ter of hp­mor, or how long it would take me to go get a snack.

(I was plan­ning to write a top-level post about this, but since it’s su­per top­i­cal I figured I might as well post it as a com­ment here. Top level post to fol­low, per­haps.)

• Apart from Pre­dic­tionBook, what’s out there in terms of soft­ware for track­ing your pre­dic­tions?

(Pre­dic­tionBook is nice, but there are pre­dic­tions I make that I don’t want to have on­line, and I find the in­abil­ity to or­ganise pre­dic­tions rather in­con­ve­nient. Ideally, some kind of sys­tem with tags would be nice. I guess I could just set up a SQLite database and write some shell scripts to add/​re­move data, but some­thing neater would be nice.)

• The more time I spend hang­ing out with ra­tio­nal­ists the less com­fortable I am mak­ing pre­dic­tions about any­thing. It’s kind of be­com­ing a real prob­lem?

“Do you think you’ll be hun­gry later?” “Maybe”

: /​

• How are you with de­ci­sions? Which af­ter all are the point of all this ra­tio­nal­ity.

• Sorry to hear that :(

My guess is that a lot of pre­dic­tions → you’re wrong some­times → it feels like you’re wrong a lot. Con­trasted with not mak­ing many pre­dic­tions → not be­ing ac­tu­ally wrong as much. If so:

1) Per­cent in­cor­rect is what mat­ters, not num­ber in­cor­rect. I’m sure you know this, but it could be difficult to get your Sys­tem 1 to know it.

2) If mak­ing a lot of pre­dic­tions does hap­pen to lead to a high per­centage in­cor­rect, that’s valuable in­for­ma­tion to you! It tells you that your mod­els of the world are off and thus pro­vides you with an op­por­tu­nity to im­prove!

• I agree with your clos­ing com­ments, but I think it’s use­ful to make the dis­tinc­tion that it’s helpful to be­gin with pre­dic­tions that are very clearly suc­cess­ful or un­suc­cess­ful, even if this might be con­strued as erring to­ward more ‘ob­jec­tive’ met­rics. This is why I chose #13 to be­gin form­ing my pre­dict­ing habit; my para­phrases of oth­ers’ po­si­tions are usu­ally eas­ily cat­e­go­rized as ei­ther cor­rect or in­cor­rect and salient out­comes are bet­ter for habit for­ma­tion.

Great post!

• To the best of my knowl­edge, hu­man brain is a simu­la­tion ma­chine. It un­con­sciously mak­ing pre­dic­tion about what sen­sory in­put it should ex­pect. This in­clude the higher level in­put, like lan­guage and even con­cepts. This is the ba­sic mechanism un­der­ly­ing sur­prise and similar emo­tion. More­over, it only makes simu­la­tion on the things it cares about and filter the rest.

Given this, I would think that most of your pre­dic­tion is ob­so­lete, be­cause we are do­ing this un­con­sciously. Ex­am­ple:

1. You pre­dict you will finish the task one week early. But you are ended up finish­ing one day early. You are not sur­prised. But if you ended up finish­ing one day late, then you would be sur­prised. When peo­ple are sur­prised by the same trig­ger of­ten enough, most nor­mal peo­ple I pre­sume, will up­date their be­lieve. I know this is re­lated to plan­ning fal­lacy, but I think my ar­gu­ments still hold wa­ter.

2. You post a post on Face­book. You didn’t make any con­scious pre­dic­tion on the re­ac­tion of the au­di­ence. You got one mil­lion likes. I bet you will be sur­prised and scratch­ing your mind about why and how you could get such re­ac­tion.

Other­wise, I still see some value in what you are do­ing, but not be­cause of pre­dic­tion per se, but be­cause you it effec­tively miti­gate bias. For ex­am­ple. “Pre­dict how well you un­der­stand some­one’s po­si­tion by try­ing to para­phrase it back to him.” It ad­dresses illu­sion of trans­parency. But I think there is not much more value in mak­ing pre­dic­tion rather than sim­ply mak­ing a habit to para­phrase more of­ten than oth­er­wise with­out mak­ing pre­dic­tion.

Mak­ing con­scious pre­dic­tion, on top of the un­con­scious one, is cog­ni­tively costly. I do think it might im­prove one’s cal­ibra­tion and ac­cu­racy and is su­pe­rior to the im­prove­ment made by the sur­prise mechanism alone. How­ever, the ques­tion is, is the cal­ibra­tion and ac­cu­racy im­prove­ment worth the ex­tra cog­ni­tive cost?

• I’m not con­vinced that im­prov­ing cal­ibra­tion will not im­prove ac­cu­racy be­cause pre­dic­tions are of­ten nested within other pre­dic­tions. For ex­am­ple, sup­pose we are try­ing to make a pre­dic­tion about P, and the truth or falsity of Q, R and S are rele­vant to the truth of P in some re­spect. We might use as a ba­sis for guess­ing P that we are ninety five per­cent con­fi­dent in our guesses about Q, R & S, (sup­pose the truth of all three would guaran­tee P). Now sup­pose we be­come less con­fi­dent through bet­ter cal­ibra­tion and de­cide there is only a 70% chance that Q, a 70% chance that R and a 70% chance that S, lead­ing to a com­pound prob­a­bil­ity of less <50%. Thus over­all ac­cu­racy can be im­proved by cal­ibra­tion.

• 11 Jun 2015 14:53 UTC
0 points

Pre­dict where this link will hy­per­link to given that I’m tel­ling you it’s a link to a page with pho­tos of the Span­ish rev­olu­tion and my post his­tory.

edit 1: no peak­ing by hov­er­ing over the link!

• My hunch is that en­courag­ing peo­ple that have to man­age an un­pre­dictable or tricky health con­di­tion to pre­dict and note their pre­dic­tion of how good or bad an ac­tivity will be for their pain /​ en­ergy /​ /​ mood /​ what­ever else would be a very use­ful habit that both frees peo­ple up to do things and pre­vents them from do­ing too much of what hurts. Ju­lia, have you or any­one from CFAR looked at part­ner­ing with a pain man­age­ment or other type of dis­ease man­age­ment team or set­ting to see how many of the ra­tio­nal­ity skills would be helpful?

• I m new here, it is very helpful wrote for who started to think­ing. But think­ing is very com­plex and has very im­proved ver­sions. I wont com­ment about my fur­ther thouths be­fore read­ing other ar­ti­cles. Maybe there are some im­proved ideas.

• Cards Against Hu­man­ity is a party game that can help de­velop the abil­ity to pre­dict what each per­son in the group finds funny, or which an­swer they will choose. If you get bet­ter at pre­dict­ing, you get bet­ter at the game. I men­tion it as a fun way to get bet­ter at pre­dict­ing.

• 14 Apr 2015 11:30 UTC
0 points

Main­tain­ing a veg­etable gar­den gives plenty of op­por­tu­ni­ties for such quick pre­dic­tions, even for se­quen­tially up­dated ones. ‘I have to weed the car­rot rows. But I have only done them last week! Surely there will be less than a hun­dred weed seedlings?’ → ‘Oh, this is how it looks to­day! Okay, rais­ing the es­ti­mate to 150.’ → ‘My back! Why do I grow car­rots, any­way? It’s a car­pet of weeds!’:)

• There the pos­si­bil­ity to write down pre­dic­tions whether you make an agree­ment with an­other per­son. If I sched­ule a date then I can pre­dict the chance of flak­ing.

That means that I stop fully trust­ing the per­son to be there. I treat them more like a num­ber than as a per­son.

• I liked (and up­voted) this post and the list is use­ful.

The use of “male pro­noun as de­fault” was a bit jar­ring :(

• I usu­ally try to mix it up. A quick count shows 6 male ex­am­ples and 2 fe­male ex­am­ples, which was not a de­liber­ate choice, but I guess I can be more in­ten­tional about a more even split in fu­ture?

• I think Eliezer men­tioned once that he flips a coin for such cases. I think it’s a pretty good policy.

• Note that a 62 split (as in this post) would hap­pen fairly fre­quently when toss­ing coins (~20% of the time, un­less I miss my guess). Since al­icey found that jar­ring, I ex­pect that pur­pose­ful al­ter­na­tion would be a bet­ter strat­egy.

• Maybe make 4 ♂ sym­bols and 4 ♀ sym­bols (if you know there will be 8 cases to­gether) and shuffle them? Oh, I guess even in that case some­one would com­plain if one set of sym­bols hap­pened to get pre­dom­i­nantly at the top of the ar­ti­cle, and other set at the bot­tom of the ar­ti­cle...

So prob­a­bly it is best to al­ter them like this: ABABABAB… or maybe like this: ABBAABBA… with a coin flip de­cid­ing who is A and who is B.

• You can also sim­ply use stan­dard names:

Alice, Bob, Carol, Dave, Erin, Frank...

• More jar­ring than that is if one set of gen­der pro­nouns gets used pre­dom­i­nantly in nega­tive ex­am­ples, and the other set gets used pre­dom­i­nantly in pos­i­tive ex­am­ples.

I try to de­liber­ately switch based on con­text. If I wrote an ex­am­ple of some­one be­ing wrong and then some­one be­ing right. I will stick with the same gen­der for both cases, and then switch to the other gen­der when I move to the next ex­am­ple of some­one be­ing wrong, right, or in­differ­ent.

Oc­ca­sion­ally, some­thing will be so in­her­ently gen­dered that I can­not use the non-de­fault gen­der and feel rea­son­able do­ing it. In these cases, I ac­tu­ally don’t think I should. (Trig­gers: sex­ual vi­o­lence. I was re­cently writ­ing about vi­o­lence, in­clud­ing rape, and I don’t think I could rea­son­able al­ter­nate pro­nouns for refer­ring to the rapist be­cause, while not all per­pe­tra­tors are male, they are so over­whelm­ingly male that it would be un­rea­son­able to use “she” in iso­la­tion. I mixed “he” with an oc­ca­sional “he or she” for the ex­tremely nega­tive ex­am­ples in those few para­graphs.)

1. That seems like it’d in­ter­rupt the flow of writ­ing.

2. It’d be in­ter­est­ing if there was some sort of com­piler that did this for you :)

• Pre­cisely for this rea­son, there was a time when I wrote in Elver­son pro­nouns (ba­si­cally, Spi­vak pro­nouns) for gen­der am­bigu­ous cases. So, if I was writ­ing about Bill Clin­ton, I would use “he,” and if I was writ­ing about Grace Hop­per, I would use “she,” but if I was writ­ing about some­body/​any­body in would use, I would use “ey” in­stead. This al­lows one to eas­ily com­pile the pro­nouns ac­cord­ing to prefer­ence with­out mis-at­tribut­ing pro­nouns to ac­tual peo­ple… I’ve always planned on get­ting around to host­ing my own blog run­ning on my own code which would in­clude an op­tion to let peo­ple set a cookie to store their gen­der prefer­ence so they could get “she by de­fault”, “he by de­fault”, “Spi­vak by de­fault”, or ran­dom­iza­tion be­tween he and she—with a gim­mick op­tion for switch­ing be­tween differ­ent sets of gen­der neu­tral pro­nouns at ran­dom. The de­fault de­fault be­ing ran­dom­iza­tion be­tween he and she. But I haven’t got­ten around to writ­ing the web­site to host my stuff yet, and I just use un­mod­ified blog­ger, so for now I’m do­ing de­liber­ate switch­ing by hand as de­scribed above.

(I think I could write a script like that for blog­ger too, but I haven’t both­ered look­ing into how to cus­tomize blog­ger be­cause I keep plan­ning to write my own web­site any­ways be­cause there are a lot of things I want to differ­ently, and that’s not nec­es­sar­ily the one that’s at the top of my list.)

• Oddly, I also came away with an im­pres­sion of ‘male pro­noun as de­fault’, and on reread­ing it seems that e.g. I strongly no­ticed the male pro­noun in 13, but did not no­tice the fe­male pro­noun in 14. I guess I’ve just been trained to no­tice de­fault-male-pro­noun us­ages. (You did also use ‘sin­gu­lar they’ in ex­am­ple 7, which to me reads much more nat­u­rally than pro­noun al­ter­na­tion.)

• Huh, I missed the pro­noun in 14 too. I sus­pect the 6:2 ra­tio is less of an is­sue than three male refer­ences be­ing be­fore the first fe­male one. I no­ticed “Lisa”, but at that point I already had the idea that the ar­ti­cle was gen­der-bi­ased. Con­fir­ma­tion bias I sup­pose.

• nod. Sounds rea­son­able!

It might help to be more in­ten­tional, to pre­vent peo­ple from hav­ing jar­ring ex­pe­riences like that.

• For ev­ery per­son that finds “male pro­noun as de­fault” jar­ring, I’d ex­pect there to be two who find con­sciously al­ter­nat­ing be­tween gen­ders jar­ring, and five for most of the more ex­otic al­ter­na­tives. I’m not say­ing it’s a bad idea af­ter tak­ing ev­ery­thing into ac­count, but if all you care about is ease of read­ing, you’d have to have a very spe­cific au­di­ence in mind for this solu­tion to make sense to me.

(Not my down­vote, by the way.)

• IME I’ve mostly found that us­ing plu­ral pro­nouns with­out call­ing at­ten­tion to them works well enough, ex­cept in cases where there’s an­other plu­ral pro­noun in the same phrase. That is, “Sam didn’t much care for corn, be­cause it got stuck in their teeth” rarely causes com­ment (though I ex­pect it to cause com­ment now, be­cause I’ve called at­ten­tion to it), but “Sam didn’t much care for corn ker­nels, be­cause they got stuck in their teeth” makes peo­ple blink.

(Of course, this is no differ­ent from any other shared-pro­noun situ­a­tion. “Sam didn’t much care for kiss­ing her boyfriend, be­cause her tongue kept get­ting caught in his braces” is clear enough, but “Sam didn’t much care for kiss­ing her girlfriend, be­cause her tongue kept get­ting caught in her braces” is de­cid­edly un­clear.)

• After a bit of thought, I be­lieve I’ve found a ba­si­cally per­ma­nent solu­tion for this. I use word re­placer (not sure how to add links with­out just post­ing them, you can google it, it is in the chrome web store) with a bunch of rules to en­force ‘they’ as de­fault. If you put rules for longer strings at the top they match first (‘he is’ to ‘they are’ at the top with ‘he’ to ‘they’ lower down, for ex­am­ple)

You will have to put up with some num­ber mis­match un­less you want to add a rule for ev­ery verb in English (‘they puts’), but I feel that that is an ac­cept­able sac­ri­fice.

EDIT: an­other is­sue: If you are ac­tu­ally talk­ing about pro­nouns, you will have to tem­porar­ily dis­able it for things to make any sense what­so­ever, and it doesn’t seem to have a way to dis­able it on a spe­cific page un­like the ser­vice I was us­ing it to re­place, so you have to use the ex­ten­sions screen in set­tings.

EDEDIT: Also, and this is both­er­ing me enough that I might ac­tu­ally stop us­ing this, is ‘her’ as a pro­noun ver­sus ‘her’ as a pos­ses­sive. for ex­am­ple in ‘Get to know her’ ver­sus ‘I found her wallet’. The first should be ‘Get to know them’ wheras the sec­ond one should be ‘I found their wallet’, and I’m not sure what to do about that. If I find/​build an ex­ten­sion which can in­ter­face with a list of en­glish words with part-of-speech tag­ging and have rules like ‘her’->‘them’, ‘her ’->‘their ’, then that’d work, but as is it is bug­ging me.

1. My im­pres­sion is that switch­ing it up would be a bit con­fus­ing to the reader. In the spirit of mak­ing pre­dic­tions, I’ll say that I’m 70% con­fi­dent that switch­ing it up would cause con­fu­sion in read­ers (not sure how I’d define con­fu­sion :/​ ). It’d be in­ter­est­ing to see re­search on this. Maybe how switch­ing it up af­fects read­ing com­pre­hen­sion or some­thing.

2. For bet­ter or for worse, con­ven­tion seems to be to use male pro­nouns, and I sense that de­vi­a­tion from this draws the read­ers at­ten­tion to­wards pro­noun use and away from the con­tent. You may ar­gue that this is an ex­am­ple of the legacy prob­lem. Again, it’d be in­ter­est­ing to see if there was any similar re­search into this.

• Data point: As­sum­ing there are any gen­dered pro­nouns in the ex­am­ples, I find it weirder when the same one is used con­sis­tently for the en­tire ar­ti­cle.