How You Make Judgments: The Elephant and its Rider

Part of the se­quence: Ra­tion­al­ity and Philosophy

Whether you’re do­ing sci­ence or philos­o­phy, flirt­ing or play­ing mu­sic, the first and most im­por­tant tool you are us­ing is your mind. To use your tool well, it will help to know how it works. To­day we ex­plore how your mind makes judg­ments.

From Plato to Freud, many have re­marked that hu­mans seem to have more than one mind.1 To­day, de­tailed ‘dual-pro­cess’ mod­els are be­ing tested by psy­chol­o­gists and neu­ro­scien­tists:

Since the 1970s dual-pro­cess the­o­ries have been de­vel­oped [to ex­plain] var­i­ous as­pects of hu­man psy­chol­ogy… Typ­i­cally, one of the pro­cesses is char­ac­ter­ized as fast, effortless, au­to­matic, non­con­scious, in­flex­ible, heav­ily con­tex­tu­al­ized, and un­de­mand­ing of work­ing mem­ory, and the other as slow, effort­ful, con­trol­led, con­scious, flex­ible, de­con­tex­tu­al­ized, and de­mand­ing of work­ing mem­ory.2

Dual-pro­cess the­o­ries for rea­son­ing,3 learn­ing and mem­ory,4 de­ci­sion-mak­ing,5 be­lief,6 and so­cial cog­ni­tion7 are now widely ac­cepted to be cor­rect to some de­gree,8 with re­searchers cur­rently work­ing out the de­tails.9 Dual-pro­cess the­o­ries even seem to be ap­pro­pri­ate for some non­hu­man pri­mates.10

Nat­u­rally, some have won­dered if there might be a “grand unify­ing dual-pro­cess the­ory that can in­cor­po­rate them all.”11 We might call such the­o­ries dual-sys­tem the­o­ries of mind,12 and sev­eral have been pro­posed.13 Such unified the­o­ries face prob­lems, though. ‘Type 1’ (fast, non­con­scious) pro­cesses prob­a­bly in­volve many non­con­scious ar­chi­tec­tures,14 and brain imag­ing stud­ies show a wide va­ri­ety of brain sys­tems at work at differ­ent times when sub­jects en­gage in ‘type 2’ (slow, con­scious) pro­cesses.15

Still, per­haps there is a sense in which one ‘mind’ re­lies mostly on type 1 pro­cesses, and a sec­ond ‘mind’ re­lies mostly on type 2 pro­cesses. One sug­ges­tion is that Mind 1 is evolu­tion­ar­ily old and thus shared with other an­i­mals, while Mind 2 is re­cently evolved and par­tic­u­larly de­vel­oped in hu­mans. (But not fully unique to hu­mans, be­cause some an­i­mals do seem to ex­hibit a dis­tinc­tion be­tween stim­u­lus-con­trol­led and higher-or­der con­trol­led be­hav­ior.16) But this the­ory faces prob­lems. A stan­dard mo­ti­va­tion for dual-pro­cess the­o­ries of rea­son­ing is the con­flict be­tween cog­ni­tive bi­ases (from type 1 pro­cesses) and log­i­cal rea­son­ing (type 2 pro­cesses).17 For ex­am­ple, logic and be­lief bias of­ten con­flict.18 But both logic and be­lief bias can be lo­cated in the pre-frontal cor­tex, an evolu­tion­ar­ily new sys­tem.19 So ei­ther Mind 1 is not en­tirely old, or Mind 2 is not en­tirely com­posed of type 2 pro­cesses.

We won’t try to un­tan­gle these mys­ter­ies here. In­stead, we’ll fo­cus on one of the most suc­cess­ful dual-pro­cess the­o­ries: Kah­ne­man and Fred­er­ick’s dual-pro­cess the­ory of judg­ment.20

At­tribute substitution

Kah­ne­man and Fred­er­ick pro­pose an “at­tribute-sub­sti­tu­tion model of heuris­tic judg­ment” which claims that judg­ments re­sult from both type 1 and type 2 pro­cesses.21 The au­thors ex­plain:

The early re­search on judg­ment heuris­tics was guided by a sim­ple and gen­eral hy­poth­e­sis: When con­fronted with a difficult ques­tion, peo­ple may an­swer an eas­ier one in­stead and are of­ten un­aware of the sub­sti­tu­tion. A per­son who is asked “What pro­por­tion of long-dis­tance re­la­tion­ships break up within a year?” may an­swer as if she had been asked “Do in­stances of failed long-dis­tance re­la­tion­ships come read­ily to mind?” This would be an ap­pli­ca­tion of the availa­bil­ity heuris­tic. A pro­fes­sor who has heard a can­di­date’s job talk and now con­sid­ers the ques­tion “How likely is it that this can­di­date could be tenured in our de­part­ment?” may an­swer the much eas­ier ques­tion: “How im­pres­sive was the talk?” This would be an ex­am­ple of one form of the rep­re­sen­ta­tive­ness heuris­tic.22

Next: what is at­tribute sub­sti­tu­tion?

...when­ever the as­pect of the judg­men­tal ob­ject that one in­tends to judge (the tar­get at­tribute) is less read­ily as­sessed than a re­lated prop­erty that yields a plau­si­ble an­swer (the heuris­tic at­tribute), in­di­vi­d­u­als may un­wit­tingly sub­sti­tute the sim­pler as­sess­ment.22

For ex­am­ple, one study23 asked sub­jects two ques­tions among many oth­ers: “How happy are you with your life in gen­eral?” and “How many dates did you have last month?” In this or­der, the cor­re­la­tion be­tween the two ques­tions was neg­ligible. If the dat­ing ques­tion was asked first, how­ever, the cor­re­la­tion was .66. The ques­tion about dat­ing fre­quency seems to evoke “an eval­u­a­tion of one’s ro­man­tic satis­fac­tion” that “lingers to be­come the heuris­tic at­tribute when the global hap­piness ques­tion is sub­se­quently en­coun­tered.”22

Or, con­sider a ques­tion in an­other study: “If a sphere were dropped into an open cube, such that it just fit (the di­ame­ter of the sphere is the same as the in­te­rior width of the cube), what pro­por­tion of the vol­ume of the cube would the sphere oc­cupy?”24 The tar­get at­tribute (the vol­u­met­ric re­la­tion be­tween cube and sphere) is difficult to as­sess in­tu­itively, and it ap­pears that sub­jects sought out an eas­ier-to-as­sess heuris­tic at­tribute in­stead, sub­sti­tut­ing the ques­tion “If a cir­cle were drawn in­side a square, what pro­por­tion of the area of the square does the cir­cle oc­cupy?” The mean es­ti­mate for the ‘sphere in­side cube’ prob­lem was 74%, al­most iden­ti­cal to the mean es­ti­mate of the ‘cir­cle in­side square’ prob­lem (77%) but far larger than the cor­rect an­swer for the ‘sphere in­side cube’ prob­lem (52%).

At­tri­bu­tion sub­sti­tu­tions like this save on pro­cess­ing power but in­tro­duce sys­tem­atic bi­ases into our judg­ment.25

Some at­tributes are always can­di­dates for the heuris­tic role in at­tribute sub­sti­tu­tion be­cause they play roles in daily per­cep­tion and cog­ni­tion and are thus always ac­cessible: cog­ni­tive fluency, causal propen­sity, sur­pris­ing­ness, mood, and af­fec­tive valence.26 Less preva­lent at­tributes can be­come ac­cessible for sub­sti­tu­tion if re­cently evoked or primed.27

Su­per­vi­sion of in­tu­itive judgments

In­tu­itive judg­ments, say Kah­ne­man and Fred­er­ick, arise from pro­cesses like at­tribute sub­sti­tu­tion, of which we are un­aware. They “bub­ble up” from the un­con­scious, af­ter which many of them are eval­u­ated and ei­ther en­dorsed or re­jected by type 2 pro­cesses.

You can feel the ten­sion28 be­tween type 1 and type 2 pro­cesses in your own judg­ment when you try the Stroop task. Name the color of a list of col­ored words and you will find that you pause a bit when the word you see names a differ­ent color than the color it is writ­ten in, like this: green. Your un­con­scious, in­tu­itive judg­ment uses an availa­bil­ity heuris­tic to sug­gest the word ‘green’ is shown in green, but your con­scious type 2 pro­cesses quickly cor­rect the un­con­scious judg­ment and con­clude that it is writ­ten in red. You have no such mo­men­tary difficulty nam­ing the color of this word: blue.

In many cases, type 2 pro­cesses have no trou­ble cor­rect­ing the judg­ments of type 1 pro­cesses.29 But be­cause type 2 pro­cesses are slow, they can be in­ter­rupted by time pres­sure.30 On the other hand, bi­ased at­tribute sub­sti­tu­tion can some­times be pre­vented if sub­jects are alerted to the pos­si­ble eval­u­a­tion con­tam­i­na­tion in ad­vance.31 (This find­ing jus­tifies a great deal of ma­te­rial on Less Wrong, which alerts you to many cog­ni­tive bi­ases—that is, pos­si­ble sources of eval­u­a­tion con­tam­i­na­tion.)

Often, type 2 pro­cesses fail to cor­rect in­tu­itive judg­ments, as demon­strated time and again in the heuris­tics and bi­ases liter­a­ture.32 And even when type 2 pro­cesses cor­rect in­tu­itive judg­ments, the feel­ing that the in­tu­itive judg­ments is cor­rect may re­main. Con­sider the fa­mous Linda prob­lem. Knowl­edge of prob­a­bil­ity the­ory does not ex­tin­guish the feel­ing (from type 1 pro­cesses) that Linda must be a fem­i­nist bank tel­ler. As Stephen Jay Gould put it:

I know [the right an­swer], yet a lit­tle ho­muncu­lus in my head con­tinues to jump up and down, shout­ing at me, “But she can’t just be a bank tel­ler; read the de­scrip­tion!”33

Conclusion

Kah­ne­man and Fred­er­ick’s dual-pro­cess the­ory ap­pears to be suc­cess­ful in ex­plain­ing a wide range of oth­er­wise puz­zling phe­nom­ena in hu­man judg­ment.34 The big pic­ture of all this is de­scribed well by Jonathan Haidt, who imag­ines his con­scious mind as a rider upon an elephant:

I’m hold­ing the reins in my hands, and by pul­ling one way or the other I can tell the elephant to turn, to stop, or to go. I can di­rect things, but only when the elephant doesn’t have de­sires of his own. When the elephant re­ally wants to do some­thing, I’m no match for him.

...The con­trol­led sys­tem [can be] seen as an ad­vi­sor. It’s a rider placed on the elephant’s back to help the elephant make bet­ter choices. The rider can see farther into the fu­ture, and the rider can learn valuable in­for­ma­tion by talk­ing to other rid­ers or by read­ing maps, but the rider can­not or­der the elephant around against its will...

...The elephant, in con­trast, is ev­ery­thing else. The elephant in­cludes gut feel­ings, visceral re­ac­tions, emo­tions, and in­tu­itions that com­prise much of the au­to­matic sys­tem. The elephant and the rider each have their own in­tel­li­gence, and when they work to­gether well they en­able the unique brilli­ance of hu­man be­ings. But they don’t always work to­gether well.35

Next post: Your Evolved Intuitions

Pre­vi­ous post: Philos­o­phy: A Diseased Discipline

Notes

1 Plato di­vided the soul into three parts: rea­son, spirit, and ap­petite (An­nas 1981, ch. 5). Descartes held that hu­mans op­er­ate on un­con­scious me­chan­i­cal pro­cesses we share with an­i­mals, but that hu­mans’ ad­di­tional ca­pac­i­ties for ra­tio­nal thought sep­a­rate us from the an­i­mals (Cot­ting­ham 1992). Leib­niz said that an­i­mals are guided by in­duc­tive rea­son­ing, which also guides ‘three-fourths’ of hu­man rea­son­ing, but that hu­mans can also par­take in ‘true rea­son­ing’ logic and math­e­mat­ics (Leib­niz 1714/​1989, p. 208; Leib­niz 1702/​1989, pp. 188-191). Many thinkers, most fa­mously Freud, have drawn a di­vi­sion be­tween con­scious and un­con­scious think­ing (Whyte 1978). For a more de­tailed sur­vey, see Frank­ish & Evans (2009). Mul­ti­ple-pro­cess the­o­ries of mind stand in con­trast to monis­tic the­o­ries of mind, for ex­am­ple: John­son-Laird (1983); Braine (1990); Rips (1994). Note that dual-pro­cess the­o­ries of mind need not con­flict with mas­sively mod­u­lar view of hu­man cog­ni­tion like Bar­rett & Kurzban (2006) or Tooby & Cos­mides (1992): see Mercier & Sper­ber (2009). Fi­nally, note that dual-pro­cess the­o­ries sit com­fortably with cur­rent re­search on situ­ated cog­ni­tion: Smith & Semin (2004).

2 Frank­ish & Evans (2009).

3 Evans (1989, 2006, 2007); Evans & Over (1996); Slo­man (1996, 2002); Stanovich (1999, 2004, 2009); Smolen­sky (1988); Car­ruthers (2002, 2006, 2009); Lie­ber­man (2003; 2009); Gilbert (1999).

4 Sun et al. (2009); Eichen­baum & Co­hen (2001); Car­ruthers (2006); Sherry & Schac­ter (1987); Berry & Dienes (1993); Re­ber (1993); Sun (2001).

5 Kah­ne­man & Fred­er­ick (2002, 2005).

6 Den­nett (1978, ch. 16; 1991); Co­hen (1992); Frank­ish (2004); Ver­scheuren et al. (2005).

7 Smith & Col­lins (2009); Bargh (2006); Strack & Deutsch (2004).

8 Evans (2008); Evans & Frank­ish (2009). Or as Car­ruthers (2009) puts it, “Dual-sys­tem the­o­ries of hu­man rea­son­ing are now quite widely ac­cepted, at least in out­line.”

9 One such de­tail is: When and to what ex­tent does Sys­tem 2 in­ter­vene in Sys­tem 1 pro­cesses? See: Evans (2006); Stanovich (1999); De Neys (2006); Evans & Cur­tis-Holmes (2005); Finu­cane et al. (2000); New­stead et al. (1992); Evans et al. (1994); Daniel & Klaczyn­ski (2006); Vade­non­coeur & Markovits (1999); Thomp­son (2009). Other im­por­tant open ques­tions are ex­plored in Fazio & Ol­son (2003); Nosek (2007); Saun­ders (2009). For an ac­cessible overview of the field, see Evans (2010).

10 Call & To­masello (2005).

11 Evans (2009).

12 Dual-pro­cess and dual-sys­tem the­o­ries of the mind sug­gest mul­ti­ple cog­ni­tive ar­chi­tec­tures, and should not be con­fused with the­o­ries of mul­ti­ple modes of pro­cess­ing, or two kinds of cog­ni­tive style. One ex­am­ple of the lat­ter is the sup­posed dis­tinc­tion be­tween Eastern and Western think­ing (Nis­bett et al. 2001). Dual-pro­cess and dual-sys­tem the­o­ries of the mind should also be dis­t­in­guished from the­o­ries that posit a con­tinuum be­tween one form of think­ing and an­other (e.g. Ham­mond 1996; New­stead 2000; Os­man 2004), since this sug­gests there are not sep­a­rate cog­ni­tive ar­chi­tec­tures at work.

13 Evans (2003); Stanovich (1999, 2009); Evans & Over (1996); Smith & DeCoster (2000); Wil­son (2002).

14 Evans (2008, 2009); Stanovich (2004); Wil­son (2002).

15 Goel (2007).

16 Toates (2004, 2006).

17 Evans (1989); Evans & Over (1996); Kah­ne­man & Fred­er­ick (2002); Klaczyn­ski & Cot­trell (2004); Slo­man (1996); Stanovich (2004).

18 Evans et al. (1983); Klauer et al. (2000).

19 Evans (2009); Goel & Dolan (2003).

20 Those who pre­fer video lec­tures to read­ing may en­joy a 2008 lec­ture on judg­ment and in­tu­ition, to which Kah­ne­man con­tributed: 1, 2, 3, 4.

21 They use the terms ‘sys­tem 1’ and ‘sys­tem 2’ in­stead of ‘type 1’ and ‘type 2’. Their the­ory is out­lined in Kah­ne­man & Fred­er­ick (2002, 2005).

22 Kah­ne­man & Fred­er­ick (2005).

23 Strack et al. (1988).

24 Fred­er­ick & Nel­son (2007).

25 Cog­ni­tive bi­ases par­tic­u­larly in­volved in at­tribute sub­sti­tu­tion in­clude the availa­bil­ity heuris­tic (Licht­en­stein et al. 1978; Sch­warz et al. 1991; Sch­warz & Vaughn 2002), the rep­re­sen­ta­tive­ness heuris­tic (Kah­ne­man & Tver­sky 1973; Tver­sky & Kah­ne­man 1982; Bar-Hillel & Neter 1993; Ag­no­lia 1991), and the af­fect heuris­tic (Slovic et al. 2002; Finu­cane et al. 2000).

26 Cog­ni­tive fluency: Ja­coby & Dal­las (1981); Sch­warz & Vaughn (2002); Tver­sky & Kah­ne­man (1973). Causal propen­sity: Mi­chotte (1963); Kah­ne­man & Varey (1990). Sur­pris­ing­ness: Kah­ne­man & Miller (1986). Mood: Sch­warz & Clore (1983). Affec­tive valence: Bargh (1997); Ca­cioppo et al. (1993); Kah­ne­man et al. (1999); Slovic et al. (2002); Za­jonc (1980, 1997).

27 Bargh et al. (1986); Hig­gins & Brendl (1995). Note also that at­tributes must be mapped across di­men­sions on a com­mon scale, and we un­der­stand to some de­gree the mechanism that does this: Kah­ne­man & Fred­er­ick (2005); Gan­zach and Krantz (1990); Stevens (1975).

28 Also see De Neys et al. (2010).

29 Gilbert (1989).

30 Finu­cane et al. (2000).

31 Sch­warz & Clore (1983); Sch­warz (1996).

32 Gilovich et al. (2002); Kah­ne­man et al. (1982); Pohl (2005); Gilovich (1993); Hastie & Dawes (2009).

33 Gould (1991), p. 469.

34 See the overview in Kah­ne­man & Fred­er­ick (2005).

35 Haidt (2006), pp. 4, 17.

References

Ag­no­lia (1991). Devel­op­ment of judg­men­tal heuris­tics and log­i­cal rea­son­ing: Train­ing coun­ter­acts the rep­re­sen­ta­tive­ness heuris­tic. Cog­ni­tive Devel­op­ment, 6: 195–217.

An­nas (1981). An in­tro­duc­tion to Plato’s re­pub­lic. Oxford Univer­sity Press.

Bar-Hillel & Neter (1993). How al­ike is it ver­sus how likely is it: A dis­junc­tion fal­lacy in prob­a­bil­ity judg­ments. Jour­nal of Per­son­al­ity and So­cial Psy­chol­ogy, 41: 671–680.

Bargh (1997). The au­to­mat­ic­ity of ev­ery­day life. Ad­vances in so­cial cog­ni­tion, 10. Er­lbaum.

Bargh, Bond, Lom­bardi, & Tota (1986). The ad­di­tive na­ture of chronic and tem­po­rary sources of con­struct ac­cessibil­ity. Jour­nal of Per­son­al­ity and So­cial Psy­chol­ogy, 50(5): 869–878.

Bargh (2006). So­cial psy­chol­ogy and the un­con­scious. Psy­chol­ogy Press.

Bar­rett & Kurzban (2006). Mo­du­lar­ity in cog­ni­tion: Fram­ing the de­bate. Psy­cholog­i­cal Re­view, 113: 628-647.

Berry & Dienes (1993). Im­plicit learn­ing. Er­lbaum.

Braine (1990). The ‘nat­u­ral logic’ ap­proach to rea­son­ing. In Over­ton (ed.), Rea­son­ing, ne­ces­sity and logic: Devel­op­men­tal per­spec­tives. Psy­chol­ogy Press.

Ca­cioppo, Priester, & Berntson (1993). Rudi­men­tary de­ter­mi­nants of at­ti­tudes: II. Arm flex­ion and ex­ten­sion have differ­en­tial effects on at­ti­tudes. Jour­nal of Per­son­al­ity and So­cial Psy­chol­ogy, 65: 5–17.

Call & To­masello (2005). Rea­son­ing and think­ing in non­hu­man pri­mates. In Holyoak & Mor­ri­son (eds.), The Cam­bridge Hand­book of Think­ing and Rea­son­ing (pp. 607-632). Cam­bridge Univer­sity Press.

Car­ruthers (2002). The cog­ni­tive func­tions of lan­guage. Be­hav­ioral and Brain Sciences, 25: 657-719.

Car­ruthers (2006). The ar­chi­tec­ture of the mind. Oxford Univer­sity Press.

Car­ruthers (2009). An ar­chi­tec­ture for dual rea­son­ing. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 109-128). Oxford Univer­sity Press.

Co­hen (1992). An es­say on be­lief and ac­cep­tance. Oxford Univer­sity Press.

Cot­ting­ham (1992). Carte­sian du­al­ism: The­ol­ogy, meta­physics, and sci­ence. In Cot­ting­ham (ed.), The Cam­bridge com­pan­ion to Descartes (pp. 236-257). Cam­bridge Univer­sity Press.

Daniel & Klaczyn­ski (2006). Devel­op­men­tal and in­di­vi­d­ual differ­ences in con­di­tional rea­son­ing: Effects of logic in­struc­tions and al­ter­na­tive an­tecedents. Child Devel­op­ment, 77: 339-354.

De Neys, Moyens, & Vansteen­we­gen (2010). Feel­ing we’re bi­ased: Au­to­nomic arousal and rea­son­ing con­flict. Cog­ni­tive, af­fec­tive, and be­hav­ioral neu­ro­science, 10(2): 208-216.

Den­nett (1978). Brain­storms: Philo­soph­i­cal es­says on mind and psy­chol­ogy. MIT Press.

Den­nett (1991). Two con­trasts: Folk craft ver­sus folk sci­ence and be­lief ver­sus opinion. In Green­wood (ed.), The fu­ture of folk psy­chol­ogy: In­ten­tion­al­ity and cog­ni­tive sci­ence (pp. 135-148). Cam­bridge Univer­sity Press.

De Neys (2006). Dual pro­cess­ing in rea­son­ing: Two sys­tems but one rea­soner. Psy­cholog­i­cal Science, 17: 428-433.

Eichen­baum & Co­hen (2001). From con­di­tion­ing to con­scious re­flec­tion: Me­mory sys­tems of the brain. Oxford Univer­sity Press.

Evans (1989). Bias in hu­man rea­son­ing: Causes and con­se­quences. Er­lbaum.

Evans (2003). In two minds: Dual-pro­cess ac­counts of rea­son­ing. Trends in Cog­ni­tive Sciences, 7: 454-459.

Evans (2006). The heuris­tic-an­a­lytic the­ory of rea­son­ing: Ex­ten­sion and eval­u­a­tion. Psy­cho­nomic Bul­letin and Re­view, 13: 378-395.

Evans (2007). Hy­po­thet­i­cal Think­ing: Dual pro­cesses in rea­son­ing and judg­ment. Psy­chol­ogy Press.

Evans (2008). Dual-pro­cess­ing ac­counts of rea­son­ing, judg­ment and so­cial cog­ni­tion. An­nual Re­view of Psy­chol­ogy, 59: 255-278.

Evans (2009). How many dual-pro­cess the­o­ries do we need? One, two, or many? In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 33-54). Oxford Univer­sity Press.

Evans (2010). Think­ing Twice: Two minds in one brain. Oxford Univer­sity Press.

Evans & Over (1996). Ra­tion­al­ity and Rea­son­ing. Psy­chol­ogy Press.

Evans & Frank­ish, eds. (2009). In Two Minds: Dual Pro­cesses and Beyond. Oxford Univer­sity Press.

Evans & Cur­tis-Holmes (2005). Rapid re­spond­ing in­creases be­lief bias: Ev­i­dence for the dual-pro­cess the­ory of rea­son­ing. Think­ing & Rea­son­ing, 11: 382-389.

Evans, Barston, & Pol­lard (1983). On the con­flict be­tween logic and be­lief in syl­l­o­gis­tic rea­son­ing. Me­mory & Cog­ni­tion, 11: 295-306.

Evans, New­stead, Allen, & Pol­lard (1994). De­bi­as­ing by in­struc­tion: The case of be­lief bias. Euro­pean Jour­nal of Cog­ni­tive Psy­chol­ogy, 6: 263-285.

Fazio & Ol­son (2003). Im­plicit mea­sures in so­cial cog­ni­tion re­search: Their mean­ing and uses. An­nual Re­view of Psy­chol­ogy, 54: 297-327.

Finu­cane, Alhakami, Slovic, & John­son (2000). The af­fect heuris­tic in judg­ments of risks and benefits. Jour­nal of Be­hav­ioral De­ci­sion Mak­ing, 13: 1-17.

Frank­ish (2004). Mind and Su­per­mind. Cam­bridge Univer­sity Press.

Frank­ish & Evans (2009). The du­al­ity of mind: An his­tor­i­cal per­spec­tive. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 1-29). Oxford Univer­sity Press.

Fred­er­ick & Nel­son (2007). At­tribute sub­sti­tu­tion in the es­ti­ma­tion of vol­u­met­ric re­la­tion­ships: Psy­chophys­i­cal phe­nom­ena un­der­score judg­men­tal heuris­tics. Manuscript in prepa­ra­tion. Mas­sachusetts In­sti­tute of Tech­nol­ogy.

Gan­zach and Krantz (1990). The psy­chol­ogy of mod­er­ate pre­dic­tion: I. Ex­pe­rience with mul­ti­ple de­ter­mi­na­tion. Or­ga­ni­za­tional Be­hav­ior and Hu­man De­ci­sion Pro­cesses, 47: 177–204.

Gilbert (1989). Think­ing lightly about oth­ers: Au­to­matic com­po­nents of the so­cial in­fer­ence pro­cess. In Ule­man & Bargh (eds.), Un­in­tended thought (pp. 189–211). Guilford Press.

Gilbert (1999). What the mind’s not. In Chaiken & Trope (eds.), Dual-pro­cess the­o­ries in so­cial psy­chol­ogy (pp. 3–11 ). Guilford Press.

Gilovich (1993). How we know what isn’t so. Free Press.

Gilovich, Griffin, & Kah­ne­man, eds. (2002). Heuris­tics and bi­ases: the psy­chol­ogy of in­tu­itive judg­ment. Cam­bridge Univer­sity Press.

Goel (2007). Cog­ni­tive neu­ro­science of de­duc­tive rea­son­ing. In Holyoak & Mor­ri­son (eds.), The Cam­bridge Hand­book of Think­ing and Rea­son­ing (pp. 475-492). Cam­bridge Univer­sity Press.

Goel & Dolan (2003). Ex­plain­ing mod­u­la­tion of rea­son­ing by be­lief. Cog­ni­tion, 87: B11-B22.

Gould (1991). Bully for bron­tosaurus. Reflec­tions in nat­u­ral his­tory. Nor­ton.

Ham­mond (1996). Hu­man judg­ment and so­cial policy. Oxford Univer­sity Press.

Haidt (2006). The hap­piness hy­poth­e­sis: Find­ing mod­ern truth in an­cient wis­dom. Ba­sic Books.

Hastie & Dawes, eds. (2009). Ra­tional Choice in an Uncer­tain World, 2nd ed. Sage.

Hig­gins & Brendl (1995). Ac­cessibil­ity and ap­pli­ca­bil­ity: Some ‘ac­ti­va­tion rules’ in­fluenc­ing judg­ment. Jour­nal of Ex­per­i­men­tal So­cial Psy­chol­ogy, 31: 218–243.

Ja­coby & Dal­las (1981). On the re­la­tion­ship be­tween au­to­bi­o­graph­i­cal mem­ory and per­cep­tual learn­ing. Jour­nal of Ex­per­i­men­tal Psy­chol­ogy: Gen­eral, 3: 306–340.

John­son-Laird (1983). Men­tal Models. Cam­bridge Univer­sity Press.

Kah­ne­man & Tver­sky (1973). On the psy­chol­ogy of pre­dic­tion. Psy­cholog­i­cal Re­view, 80: 237–251.

Kah­ne­man et al. (1999). Eco­nomic prefer­ences or at­ti­tude ex­pres­sions? An anal­y­sis of dol­lar re­sponses to pub­lic is­sues. Jour­nal of Risk and Uncer­tainty, 19: 203–235.

Kah­ne­man & Fred­er­ick (2002). Rep­re­sen­ta­tive­ness re­vis­ited: At­tribute sub­sti­tu­tion in in­tu­itive judg­ment. In Gilovich, Griffin, & Kah­ne­man (eds.), Heuris­tics and bi­ases: The psy­chol­ogy of in­tu­itive judg­ment (pp. 49-81). Cam­bridge Univer­sity Press.

Kah­ne­man & Fred­er­ick (2005). A model of heuris­tic judg­ment. In Holyoak & Mor­ri­son (eds.), The Cam­bridge Hand­book of Think­ing and Rea­son­ing (pp. 267-294). Cam­bridge Univer­sity Press.

Kah­ne­man & Miller (1986). Norm the­ory: Com­par­ing re­al­ity with its al­ter­na­tives. Psy­cholog­i­cal Re­view, 93: 136–153.

Kah­ne­man & Varey (1990). Propen­si­ties and coun­ter­fac­tu­als: The loser that al­most won. Jour­nal of Per­son­al­ity and So­cial Psy­chol­ogy, 59(6): 1101–1110.

Klaczyn­ski & Cot­trell (2004). A dual-pro­cess ap­proach to cog­ni­tive de­vel­op­ment: The case of chil­dren’s un­der­stand­ing of sunk cost de­ci­sions. Think­ing and Rea­son­ing, 10: 147-174.

Kah­ne­man, Slovic, & Tver­sky, eds. (1982). Judg­ment un­der un­cer­tainty: Heuris­tics and bi­ases. Cam­bridge Univer­sity Press.

Klauer, Musch, & Naumer (2000). On be­lief bias in syl­l­o­gis­tic rea­son­ing. Psy­cholog­i­cal Re­view, 107: 852-884.

Leib­niz (1702/​1989). Let­ter to Queen So­phie Char­lotte of Prus­sia, on what is in­de­pen­dent of sense and mat­ter. In Leib­niz, Philo­soph­i­cal es­says (pp. 186-192). Hack­ett.

Leib­niz (1714/​1989). Prin­ci­ples of na­ture and grace, based on rea­son. In Leib­niz, Philo­soph­i­cal es­says (pp. 206-213). Hack­ett.

Licht­en­stein, Slovic, Fischhoff, Lay­man, & Combs (1978). Judged Fre­quency of Lethal Events. Jour­nal of Ex­per­i­men­tal Psy­chol­ogy: Hu­man Learn­ing and Me­mory, 4(6): 551-578.

Lie­ber­man (2003). Reflec­tive and re­flex­ive judg­ment pro­cesses: A so­cial cog­ni­tive neu­ro­science ap­proach. In For­gas, Willi­ams, & von Hip­pel (eds.), So­cial judg­ments: Im­plicit and ex­plicit pro­cesses (pp. 44-67). Cam­bridge Univer­sity Press.

Lie­ber­man (2009). What zom­bies can’t do: A so­cial cog­ni­tive neu­ro­science ap­proach to the ir­re­ducibil­ity of re­flec­tive con­scious­ness. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 293-316). Oxford Univer­sity Press.

Mercier & Sper­ber (2009). In­tu­itive and re­flec­tive in­fer­ences. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 149-170). Oxford Univer­sity Press.

Mi­chotte (1963). The per­cep­tion of causal­ity. Ba­sic Books.

New­stead (2000). Are there two differ­ent types of think­ing? Be­hav­ioral and Brain Sciences, 23: 690-691.

New­stead, Pol­lard, & Evans (1992). The source of be­lief bias effects in syl­l­o­gis­tic rea­son­ing. Cog­ni­tion, 45: 257-284.

Nis­bett, Peng, Choi, & Noren­za­yan (2001). Cul­ture and sys­tems of thought: Holis­tic vs an­a­lytic cog­ni­tion. Psy­cholog­i­cal Re­view, 108: 291-310.

Nosek (2007). Im­plicit-ex­plicit re­la­tions. Cur­rent Direc­tions in Psy­cholog­i­cal Science, 16: 65-69.

Os­man (2004). An eval­u­a­tion of dual-pro­cess the­o­ries of rea­son­ing. Psy­cho­nomic Bul­letin and Re­view, 11: 988-1010.

Pohl, ed. (2005). Cog­ni­tive illu­sions: a hand­book on fal­la­cies and bi­ases in think­ing, judg­ment and mem­ory. Psy­chol­ogy Press.

Re­ber (1993). Im­plicit learn­ing and tacit knowl­edge. Oxford Univer­sity Press.

Rips (1994). The psy­chol­ogy of proof: De­duc­tive rea­son­ing in hu­man think­ing. MIT Press.

Saun­ders (2009). Rea­son and in­tu­ition in the moral life: A dual-pro­cess ac­count of moral jus­tifi­ca­tion. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 335-354). Oxford Univer­sity Press.

Sch­warz, Bless, Strack, Klumpp, Rit­te­nauer-Schatka, & Si­mons (1991). Ease of re­trieval as in­for­ma­tion: Another look at the availa­bil­ity heuris­tic. Jour­nal of Per­son­al­ity and So­cial Psy­chol­ogy, 61: 195–202.

Sch­warz & Clore (1983). Mood, mis­at­tri­bu­tion, and judg­ments of well-be­ing: In­for­ma­tive and di­rec­tive func­tions of af­fec­tive states. Jour­nal of Per­son­al­ity and So­cial Psy­chol­ogy, 45(3): 513–523.

Sch­warz (1996). Cog­ni­tion and com­mu­ni­ca­tion: Judg­men­tal bi­ases, re­search meth­ods, and the logic of con­ver­sa­tion. Er­lbaum.

Sch­warz & Vaughn (2002). The availa­bil­ity heuris­tic re­vis­ited: Ease of re­call and con­tent of re­call as dis­tinct sources of in­for­ma­tion. In Gilovich, Griffin, & Kah­ne­man (eds.), Heuris­tics & bi­ases: The psy­chol­ogy of in­tu­itive judg­ment (pp. 103–119). Cam­bridge Univer­sity Press.

Sherry & Schac­ter (1987). The evolu­tion of mul­ti­ple mem­ory sys­tems. Psy­cholog­i­cal Re­view, 94: 439-454.

Slo­man (1996). The em­piri­cal case for two sys­tems of rea­son­ing. Psy­cholog­i­cal Bul­letin, 119: 1-23.

Slo­man (2002). Two sys­tems of rea­son­ing. In Gilovich, Griffin, & Kah­ne­man (eds.), Heuris­tics and Bi­ases: The Psy­chol­ogy of In­tu­itive Judg­ment. Cam­bridge Univer­sity Press.

Slovic, Finu­cane, Peters, & MacGre­gor (2002). Ra­tional Ac­tors or Ra­tional Fools: Im­pli­ca­tions of the Affect Heuris­tic for Be­hav­ioral Eco­nomics. Jour­nal of So­cio-Eco­nomics, 31: 329-342.

Smith & Col­lins (2009). Dual-pro­cess mod­els: A so­cial psy­cholog­i­cal per­spec­tive. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 197-216). Oxford Univer­sity Press.

Smith & DeCoster (2000). Dual-pro­cess mod­els in so­cial and cog­ni­tive psy­chol­ogy: Con­cep­tual in­te­gra­tion and links to un­der­ly­ing mem­ory sys­tems. Per­son­al­ity and So­cial Psy­chol­ogy Re­view, 4: 108-131.

Smith & Semin (2004). So­cially situ­ated cog­ni­tion: Cog­ni­tion in its so­cial con­text. Ad­vances in ex­per­i­men­tal so­cial psy­chol­ogy, 36: 53-117.

Smolen­sky (1988). On the proper treat­ment of con­nec­tion­ism. Be­hav­ioral and Brain Sciences, 11: 1-23.

Stanovich (1999). Who is ra­tio­nal? Stud­ies of in­di­vi­d­ual differ­ences in rea­son­ing. Psy­chol­ogy Press.

Stanovich (2004). The robot’s re­bel­lion: Find­ing mean­ing in the age of Dar­win. Chicago Univer­sity Press.

Stanovich (2009). Dist­in­guish­ing the re­flec­tive, al­gorith­mic, and au­tonomous minds: Is it time for a tri-pro­cess the­ory? In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 55-88). Oxford Univer­sity Press.

Strack & Deutsch (2004). Reflec­tive and im­pul­sive de­ter­mi­nants of so­cial be­hav­ior. Per­son­al­ity and So­cial Psy­chol­ogy Re­view, 8: 220-247.

Strack, Martin, & Sch­warz (1988). Prim­ing and com­mu­ni­ca­tion: The so­cial de­ter­mi­nants of in­for­ma­tion use in judg­ments of life satis­fac­tion. Euro­pean Jour­nal of So­cial Psy­chol­ogy, 1: 429–442.

Stevens (1975). Psy­chophysics: In­tro­duc­tion to its per­cep­tual, neu­ral, and so­cial prospects. Wiley.

Sun (2001). Dual­ity of mind: A bot­tom-up ap­proach to­wards cog­ni­tion. Psy­chol­ogy Press.

Sun, Lane, & Mathews (2009). The two sys­tems of learn­ing: An ar­chi­tec­tural per­spec­tive. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 239-262). Oxford Univer­sity Press.

Toates (2004). ‘In two minds’ - con­sid­er­a­tion of evolu­tion­ary pre­cur­sors per­mits a more in­te­gra­tive the­ory. Trends in Cog­ni­tive Sciences, 8: 57.

Toates (2006). A model of the heirar­chy of be­havi­our, cog­ni­tion, and con­scious­ness. Con­scious­ness and Cog­ni­tion, 15: 75-118.

Thomp­son (2009). Dual-pro­cess the­o­ries: A metacog­ni­tive per­spec­tive. In Evans & Fran­klin (eds.), In Two Minds: Dual Pro­cesses and Beyond (pp. 171-195). Oxford Univer­sity Press.

Tooby & Cos­mides (1992). The psy­cholog­i­cal foun­da­tions of cul­ture. In Barkow, Cos­mides, & Tooby (eds.), The Adapted Mind (pp. 19-136). Oxford Univer­sity Press.

Tver­sky & Kah­ne­man (1973). Availa­bil­ity: a heuris­tic for judg­ing fre­quency and prob­a­bil­ity. Cog­ni­tive Psy­chol­ogy, 5(2): 207–232.

Tver­sky & Kah­ne­man (1982). Judg­ments of and by rep­re­sen­ta­tive­ness. In Kah­ne­man, Slovic, & Tver­sky (eds.), Judg­ment un­der un­cer­tainty: Heuris­tics and bi­ases (pp. 84–98). Cam­bridge Univer­sity Press.

Vade­non­coeur & Markovits (1999). The effect of in­struc­tions and in­for­ma­tion re­trieval on ac­cept­ing the premises in a con­di­tional rea­son­ing task. Think­ing & Rea­son­ing, 5: 97-113.

Ver­scheuren, Schaeken, & d’Yde­walle (2005). A dual-pro­cess speci­fi­ca­tion of causal con­di­tional rea­son­ing. Think­ing & Rea­son­ing, 11: 239-278.

Whyte (1978). The un­con­scious be­fore Freud. St. Martin’s Press.

Wil­son (2002). Strangers to our­selves: Dis­cov­er­ing the adap­tive un­con­scious. Belk­nap Press.

Za­jonc (1980). Feel­ing and think­ing: Prefer­ences need no in­fer­ences. Amer­i­can Psy­chol­o­gist, 35(2): 151–175.

Za­jonc (1997). Emo­tions. In Gilbert, Fiske, & Lindzey (eds.), Hand­book of so­cial psy­chol­ogy (4th ed., pp. 591–632). Oxford Univer­sity Press.