Motivated Stopping and Motivated Continuation
While I disagree with some views of the Fast and Frugal crowd—in my opinion they make a few too many lemons into lemonade—it also seems to me that they tend to develop the most psychologically realistic models of any school of decision theory. Most experiments present the subjects with options, and the subject chooses an option, and that’s the experimental result. The frugalists realized that in real life, you have to generate your options, and they studied how subjects did that.
Likewise, although many experiments present evidence on a silver platter, in real life you have to gather evidence, which may be costly, and at some point decide that you have enough evidence to stop and choose. When you’re buying a house, you don’t get exactly ten houses to choose from, and you aren’t led on a guided tour of all of them before you’re allowed to decide anything. You look at one house, and another, and compare them to each other; you adjust your aspirations—reconsider how much you really need to be close to your workplace and how much you’re really willing to pay; you decide which house to look at next; and at some point you decide that you’ve seen enough houses, and choose.
Gilovich’s distinction between motivated skepticism and motivated credulity highlights how conclusions a person does not want to believe are held to a higher standard than conclusions a person wants to believe. A motivated skeptic asks if the evidence compels them to accept the conclusion; a motivated credulist asks if the evidence allows them to accept the conclusion.
I suggest that an analogous bias in psychologically realistic search is motivated stopping and motivated continuation: when we have a hidden motive for choosing the “best” current option, we have a hidden motive to stop, and choose, and reject consideration of any more options. When we have a hidden motive to reject the current best option, we have a hidden motive to suspend judgment pending additional evidence, to generate more options—to find something, anything, to do instead of coming to a conclusion.
A major historical scandal in statistics was R. A. Fisher, an eminent founder of the field, insisting that no causal link had been established between smoking and lung cancer. “Correlation is not causation,” he testified to Congress. Perhaps smokers had a gene which both predisposed them to smoke and predisposed them to lung cancer.
Or maybe Fisher’s being employed as a consultant for tobacco firms gave him a hidden motive to decide that the evidence already gathered was insufficient to come to a conclusion, and it was better to keep looking. Fisher was also a smoker himself, and died of colon cancer in 1962.1
Like many other forms of motivated skepticism, motivated continuation can try to disguise itself as virtuous rationality. Who can argue against gathering more evidence?2
I can. Evidence is often costly, and worse, slow, and there is certainly nothing virtuous about refusing to integrate the evidence you already have. You can always change your mind later.3
As for motivated stopping, it appears in every place a third alternative is feared, and wherever you have an argument whose obvious counterargument you would rather not see, and in other places as well. It appears when you pursue a course of action that makes you feel good just for acting, and so you’d rather not investigate how well your plan really worked, for fear of destroying the warm glow of moral satisfaction you paid good money to purchase.4 It appears wherever your beliefs and anticipations get out of sync, so you have a reason to fear any new evidence gathered.5
The moral is that the decision to terminate a search procedure (temporarily or permanently) is, like the search procedure itself, subject to bias and hidden motives. You should suspect motivated stopping when you close off search, after coming to a comfortable conclusion, and yet there’s a lot of fast cheap evidence you haven’t gathered yet—there are websites you could visit, there are counter-counter arguments you could consider, or you haven’t closed your eyes for five minutes by the clock trying to think of a better option. You should suspect motivated continuation when some evidence is leaning in a way you don’t like, but you decide that more evidence is needed—expensive evidence that you know you can’t gather anytime soon, as opposed to something you’re going to look up on Google in thirty minutes—before you’ll have to do anything uncomfortable.
1Ad hominem note: Fisher was a frequentist. Bayesians are more reasonable about inferring probable causality; see Judea Pearl’s Causality: Models, Reasoning, and Inference.
2Compare Robin Hanson, “Cut Medicine In Half,” Overcoming Bias (blog), September 10, 2007, http://www.overcomingbias.com/2007/09/cut-medicine-in.html.
3Apparent contradiction resolved as follows: Spending one hour discussing the problem, with your mind carefully cleared of all conclusions, is different from waiting ten years on another $20 million study.
4See “‘Can’t Say No’ Spending.” http://lesswrong.com/lw/kb/cant_say_no_spending.
5See “Belief in Belief” in Map and Territory.
- Bad Omens in Current Community Building by 12 May 2022 15:41 UTC; 561 points) (EA Forum;
- The motivated reasoning critique of effective altruism by 14 Sep 2021 20:43 UTC; 276 points) (EA Forum;
- Eliezer’s Sequences and Mainstream Academia by 15 Sep 2012 0:32 UTC; 221 points) (
- The Intelligent Social Web by 22 Feb 2018 18:55 UTC; 211 points) (
- Beware surprising and suspicious convergence by 24 Jan 2016 19:11 UTC; 187 points) (EA Forum;
- The inordinately slow spread of good AGI conversations in ML by 21 Jun 2022 16:09 UTC; 173 points) (
- Compartmentalization in epistemic and instrumental rationality by 17 Sep 2010 7:02 UTC; 116 points) (
- Disputing Definitions by 12 Feb 2008 0:15 UTC; 106 points) (
- The Tragedy of Group Selectionism by 7 Nov 2007 7:47 UTC; 98 points) (
- Of Two Minds by 17 May 2018 4:34 UTC; 92 points) (
- Taking Ideas Seriously by 13 Aug 2010 16:50 UTC; 81 points) (
- Optimal Employment by 31 Jan 2011 12:50 UTC; 77 points) (
- Can Humanism Match Religion’s Output? by 27 Mar 2009 11:32 UTC; 74 points) (
- Ethical Injunctions by 20 Oct 2008 23:00 UTC; 72 points) (
- Reasoning isn’t about logic (it’s about arguing) by 14 Mar 2010 4:42 UTC; 66 points) (
- Fake Utility Functions by 6 Dec 2007 16:55 UTC; 60 points) (
- The inordinately slow spread of good AGI conversations in ML by 29 Jun 2022 4:02 UTC; 59 points) (EA Forum;
- Inherited Improbabilities: Transferring the Burden of Proof by 24 Nov 2010 3:40 UTC; 46 points) (
- Let Your Mind Be Not Fixed by 31 Jul 2020 17:54 UTC; 46 points) (
- Politics as Charity by 23 Sep 2010 5:33 UTC; 37 points) (
- A Case Study of Motivated Continuation by 31 Oct 2007 1:27 UTC; 35 points) (
- 13 Aug 2012 5:18 UTC; 32 points)'s comment on What are your questions about making a difference? by (
- Self-experiment Protocol: Effect of Chocolate on Sleep by 22 Jul 2019 3:32 UTC; 29 points) (
- Probability and Politics by 24 Nov 2010 17:02 UTC; 28 points) (
- Rational vs Reasonable by 11 Jul 2015 3:31 UTC; 27 points) (
- Dreams of AI Design by 27 Aug 2008 4:04 UTC; 27 points) (
- The Motivated Reasoning Critique of Effective Altruism by 15 Sep 2021 1:43 UTC; 26 points) (
- Trust and The Small World Fallacy by 4 Oct 2021 0:38 UTC; 26 points) (
- A LessWrong “rationality workbook” idea by 9 Jan 2011 17:52 UTC; 26 points) (
- Some of the best rationality essays by 19 Oct 2021 22:57 UTC; 25 points) (
- Beware surprising and suspicious convergence by 24 Jan 2016 19:13 UTC; 23 points) (
- A Genius for Destruction by 1 Aug 2008 19:25 UTC; 22 points) (
- Total Nano Domination by 27 Nov 2008 9:54 UTC; 19 points) (
- Contaminated by Optimism by 6 Aug 2008 0:26 UTC; 19 points) (
- 15 Jan 2015 9:29 UTC; 18 points)'s comment on Je suis Charlie by (
- That Crisis thing seems pretty useful by 10 Apr 2009 17:10 UTC; 18 points) (
- 1 Mar 2018 19:32 UTC; 18 points)'s comment on Ms. Blue, meet Mr. Green by (
- HPMoR: What do you think you know? by 23 Oct 2011 4:17 UTC; 17 points) (
- Does cognitive therapy encourage bias? by 22 Nov 2010 11:31 UTC; 16 points) (
- 18 Feb 2010 0:28 UTC; 15 points)'s comment on Open Thread: February 2010, part 2 by (
- Some Remarks on the Nature of Political Conflict by 4 Jul 2018 12:31 UTC; 15 points) (
- Concept Safety: World-models as tools by 9 May 2015 12:07 UTC; 14 points) (
- 21 Oct 2011 15:35 UTC; 14 points)'s comment on Amanda Knox: post mortem by (
- 9 Jun 2019 6:48 UTC; 14 points)'s comment on Drowning children are rare by (
- Guidelines for Mad Entrepreneurs by 16 Sep 2022 6:33 UTC; 12 points) (
- 2 Jun 2020 20:45 UTC; 12 points)'s comment on Book of Mormon Discussion by (
- [SEQ RERUN] Motivated Stopping and Motivated Continuation by 10 Oct 2011 14:59 UTC; 11 points) (
- 18 Sep 2013 11:51 UTC; 11 points)'s comment on Welcome to Less Wrong! (5th thread, March 2013) by (
- 28 Jul 2011 20:01 UTC; 9 points)'s comment on A Fable of Science and Politics by (
- A Different Prisoner’s Dilemma by 14 Apr 2018 15:54 UTC; 9 points) (
- 23 Nov 2010 16:08 UTC; 9 points)'s comment on Existential Risk and Public Relations by (
- 30 Oct 2007 16:51 UTC; 8 points)'s comment on Torture vs. Dust Specks by (
- 11 Oct 2011 3:15 UTC; 8 points)'s comment on Rationality Lessons Learned from Irrational Adventures in Romance by (
- Against the Bottom Line by 21 Apr 2012 10:20 UTC; 7 points) (
- 23 Oct 2011 23:26 UTC; 7 points)'s comment on Amanda Knox: post mortem by (
- 18 May 2010 5:00 UTC; 7 points)'s comment on Preface to a Proposal for a New Mode of Inquiry by (
- Rationality Reading Group: Part G: Against Rationalization by 12 Aug 2015 22:09 UTC; 7 points) (
- A Test of Faith by 6 Apr 2018 4:36 UTC; 6 points) (
- 2 Feb 2015 21:20 UTC; 6 points)'s comment on Open Thread, Feb. 2 - Feb 8, 2015 by (
- 24 Oct 2011 20:39 UTC; 6 points)'s comment on Amanda Knox: post mortem by (
- 24 Jul 2014 13:56 UTC; 6 points)'s comment on Alpha Mail by (
- 1 Mar 2010 19:21 UTC; 5 points)'s comment on Open Thread: March 2010 by (
- 15 May 2018 4:01 UTC; 5 points)'s comment on Mental Illness Is Not Evidence Against Abuse Allegations by (
- A Ketogenic Diet as an Effective Cancer Treatment? by 26 Jun 2013 22:40 UTC; 5 points) (
- 14 Aug 2009 5:14 UTC; 4 points)'s comment on It’s all in your head-land by (
- 23 Jan 2021 4:18 UTC; 4 points)'s comment on AllAmericanBreakfast’s Shortform by (
- 15 Nov 2012 15:58 UTC; 4 points)'s comment on Rationality Quotes November 2012 by (
- Does cognitive therapy encourage bias? by 22 Nov 2010 9:52 UTC; 4 points) (
- 14 Nov 2016 17:23 UTC; 3 points)'s comment on Yudkowsky vs Trump: the nuclear showdown. by (
- 27 Jun 2011 17:33 UTC; 3 points)'s comment on What can we gain from rationality? by (
- 12 Oct 2011 7:37 UTC; 3 points)'s comment on Rationality Lessons Learned from Irrational Adventures in Romance by (
- 16 Dec 2013 16:31 UTC; 2 points)'s comment on Luck II: Expecting White Swans by (
- 26 Sep 2012 5:04 UTC; 2 points)'s comment on [SEQ RERUN] Traditional Capitalist Values by (
- 11 Sep 2014 18:28 UTC; 2 points)'s comment on Rationality Quotes September 2014 by (
- 18 Jul 2011 17:11 UTC; 2 points)'s comment on The limits of introspection by (
- 18 Mar 2009 6:28 UTC; 2 points)'s comment on The Pascal’s Wager Fallacy Fallacy by (
- Does cognitive therapy encourage bias? by 22 Nov 2010 11:11 UTC; 2 points) (
- 11 Feb 2010 5:34 UTC; 1 point)'s comment on Open Thread: February 2010 by (
- 1 May 2018 22:43 UTC; 1 point)'s comment on Sentience by (
- 2 May 2018 22:24 UTC; 1 point)'s comment on Sentience by (
- 12 May 2009 11:44 UTC; 1 point)'s comment on Open Thread: May 2009 by (
- 17 Feb 2016 13:52 UTC; 1 point)'s comment on Should we admit it when a person/group is “better” than another person/group? by (
- 28 Oct 2011 8:32 UTC; 1 point)'s comment on The Pleasures of Rationality by (
- 27 Oct 2010 9:09 UTC; 0 points)'s comment on Pathological utilitometer thought experiment by (
- 8 Jan 2016 12:19 UTC; 0 points)'s comment on Open Thread, January 4-10, 2016 by (
- 5 Apr 2012 14:26 UTC; 0 points)'s comment on Bayesianism and use of Evidence in Social Deduction Games by (
- Do the people behind the veil of ignorance vote for “specks”? by 11 Nov 2011 1:26 UTC; -1 points) (
- 22 Mar 2012 14:01 UTC; -4 points)'s comment on Modest Superintelligences by (
- 28 Jul 2011 3:02 UTC; -8 points)'s comment on New Post version 1 (please read this ONLY if your last name beings with a–k) by (
Eliezer, are you familiar with Russell and Wefald’s book “Do the Right Thing”?
It’s fairly old (1991), but it’s a good example of how people in AI view limited rationality.
Maybe you could exploit this, if the question you’re gathering evidence for is important enough to warrant all that costly searching. Spending hours digging through obscure journals is not something most people do for fun, but if you can come up with a pet theory which needs reinforcing, most people would rather do the evidence-gathering than be forced to give it up.
‘Motivated stopping’? What springs to my mind is psi tests. If you regard psi tests as a possibly infinite series, then when you cut off testing and start analysing can produce any result you want.
‘Lucky streaks’ can occur at any any time in a string of random numbers.
That’s why in psi testing you must calculate the exact number of tests required to show an effect of the size you expect and do precisely that number of tests, no more and no less. And you are not allowed to throw away the tests that resulted in average or negative results either.
My favourite example of motivated stopping is Lazzarini’s experimental “verification” of the Buffon needle formula.
(Drop toothpicks at random on a plane ruled with evenly spaced parallel lines. The average number of line-crossings per toothpick is related to pi. Lazzarini did the experiment and got pi to 6 decimal places. It seems clear that he did this by doing trials in batches whose size made it likely that he’d get an estimate equivalent to pi = 355⁄113, which happens to be very close, and then did one batch at a time until he happened to hit it on the nose.
Completely off-topic, here’s a beautiful derivation of the formula: Expectations are additive, so the expected number of line-crossings is proportional to the length of the toothpick and doesn’t depend on what shape it actually is. So consider a circular “toothpick” whose diameter equals the spacing between the lines. No matter how you drop this, you get 2 crossings. Therefore the constant of proportionality is 2/pi. Therefore the expected number of crossings for any toothpick of length L, in units where the line-spacing is 1, is 2L/pi. If L<1 then this is also the probability of getting a crossing at all, since you can’t get more than one.)
To put it differently, motivated stopping is a problem in pi tests just like it is in psi tests. :-)
I find this article relevant to the whole series Amanda Knox posts/comments.
There was, as far as I know, no observed evidence of a causal link between smoking and lung cancer.
The correlation (specifically, the relative risk ratio) was roughly 19,000% (~190x).
There have been some “natural experiments” that suggest a causal link since, but I don’t think there were any available at that time.