# Proving Too Much (w/​ exercises)

This is the first post in the Ar­gu­ing Well se­quence. This post is in­fluenced by Scott Alexan­der’s write up on Prov­ing too Much.

[edit: Re­for­mat­ted the post as a Prob­lem/​Solu­tion to clar­ify what I’m try­ing to claim]

# The Problem

One of the pur­poses of ar­gu­ing well is to figure out what is true. A very com­mon type of bad ar­gu­ment claims some­thing like this:

Be­cause of rea­son X, I am 100% con­fi­dent in be­lief Y

I don’t know of any rea­son that leads to 100% truth all the time (and if you do, please let me know!), and it’s usu­ally hard to rea­son with the per­son un­til this faulty logic is dealt with first. This is the pur­pose of this post.

As­sum­ing the con­text of all ex­er­cises is with some­one claiming 100% be­lief for that one rea­son, what’s wrong with the fol­low­ing:

Ex. 1: I be­lieve that Cthulhu ex­ists be­cause that’s just how I was raised.

How some­one was raised doesn’t make some­thing true or not. In fact, I could be raised to be­lieve that Cthulhu doesn’t ex­ist. We can’t both be right.

Ex. 2: I be­lieve that a god­dess is watch­ing over me be­cause it makes me feel bet­ter and helps me get through the day.

Just be­cause be­liev­ing it makes you feel bet­ter doesn’t make it true. Kids might feel bet­ter be­liev­ing in Santa Claus, but that doesn’t make him ac­tu­ally ex­ist.

# Gen­er­al­ized Model

How would you gen­er­al­ize the com­mon prob­lem in the above ar­gu­ments? You have 2 minutes

The com­mon theme that I see is that same logic that proves the origi­nal claim, also proves some­thing false. It “Proves too much” be­cause it also proves false things. I like to think of this logic as “Qual­ifi­ca­tions for 100% truth”, and what­ever qual­ifi­ca­tions proves the origi­nal claim can also prove a false claim.

Truth Qual­ifi­ca­tions → Claim

Same Truth Qual­ifi­ca­tions → Ab­surd Claim

Im­por­tant Note: the pur­pose of this frame isn’t to win an ar­gu­ment/​ prove any­thing. It’s to differ­en­ti­ate be­tween heuris­tics that claim 100% suc­cess rates vs ones that claim a more ac­cu­rate es­ti­mates. Imag­ine “I’m 100% con­fi­dent I’ll roll 7′s with my two die cause of my luck!” vs “There’s a 636 chance I’ll roll 7′s be­cause I’m as­sum­ing two fair die”

Let’s work a cou­ple more ex­am­ples with this model.

Ex. 3: My startup is guaran­teed to suc­ceed be­cause it uses quan­tum ma­chine learn­ing on a blockchain!

A startup us­ing buz­zwords doesn’t make it suc­ceed. In fact, there are sev­eral star­tups that use those terms and failed.

Ex. 4: Of course I be­lieve in evolu­tion! Stephen Hawk­ing be­lieves it, and he’s re­ally smart.

A smart per­son be­liev­ing some­thing doesn’t make it true. In fact, smart peo­ple of­ten dis­agree and I bet there’s a per­son with Mensa-level IQ that doesn’t be­lieve in evolu­tion.

Ex. 5: This pa­per’s re­sult has to be true since it has p < 0.05!

A pa­per hav­ing a p value less than 0.05 doesn’t mean it’s true. In fact, there are sev­eral pa­pers that dis­agree with each other with p < 0.05. Also, home­opa­thy has been shown to have a p value < 0.005!

# Ideal Algorithm

What al­gorithm were you run­ning when you solved the above prob­lems? Is there a more ideal/​gen­eral al­gorithm? You have 3 min­utes.

1. What does this per­son be­lieve?

2. Why do they be­lieve it?

3. Gen­er­al­ize that reasoning

4. What’s some­thing crazy I can prove with this rea­son­ing?

The al­gorithm I ac­tu­ally ran felt like a mix of 1 & 2 & 3, and then 4, but with­out liter­ally think­ing those words in my head.

Now to prac­tice that new, ideal al­gorithm you made.

# Fi­nal Prob­lem Sets

Ex. 6: I be­lieve in my re­li­gion be­cause of faith (defined as hope)

Hop­ing in some­thing doesn’t make it true. I can hope to make a good grade on a test, but that doesn’t mean that I will make a good grade. Study­ing would prob­a­bly help more than hop­ing. (Here I pro­vided a counter-ex­am­ple as re­quired and an ad­di­tional counter-rea­son)

Ex. 7: I be­lieve in my re­li­gion be­cause of faith (defined as trust)

Trust­ing in some­thing doesn’t make it true. I can trust that my dog won’t bite peo­ple, but then some­one steps on her paw and she bites them. Trust­ing that my dog won’t bite peo­ple doesn’t make my dog not bite peo­ple.

Ex. 8: I be­lieve in a soul be­cause I have a re­ally strong gut feel­ing.

Hav­ing a strong gut feel­ing doesn’t make it true. In ju­ries, peo­ple can even have con­flict­ing gut feel­ings about a crime. If a jury was try­ing to de­ter­mine if I was guilty, I would want them to use the ev­i­dence available and not their gut feel­ing. (Again, I added an ad­di­tional counter-rea­son)

Ex. 9: I be­lieve in my re­li­gion be­cause I had a re­ally amaz­ing, trans­for­ma­tive ex­pe­rience.

There are sev­eral re­li­gions that claim con­tra­dic­tory be­liefs, and also have sev­eral peo­ple who have had re­ally great, trans­for­ma­tive ex­pe­riences.

Ex. 10: I be­lieve in my re­li­gion be­cause there are sev­eral ac­counts of peo­ple see­ing heaven when they died and came back.

There are sev­eral ac­counts of peo­ple see­ing their re­li­gion’s ver­sion of heaven or nir­vana in death-to-life ex­pe­riences. You would have to be­lieve Chris­ti­an­ity, Mor­monism, Is­lam, Hindu, … too!

Ex. 11: You get an email ask­ing to be wired money, which you’ll be paid back hand­somely for. The email con­cludes “I, prince Nubadola, as­sure you that this is my mes­sage, and it is le­gi­t­i­mate. You can trust this email and any oth­ers that come from me.”

The email say­ing the email is le­gi­t­i­mate doesn’t make it true. I could even write a new email say­ing “Prince Nubadola is a fraud, and I as­sure you that this is true”. (This is cir­cu­lar rea­son­ing/​ beg­ging the ques­tion)

# Conclusion

In or­der to ar­gue well, it’s im­por­tant to iden­tify and work through ar­gu­ments that prove too much. In prac­tice, this tech­nique has the po­ten­tial to lower some­one’s con­fi­dence… in a be­lief, or help clar­ify that “No, I don’t think this leads to 100% true things all the time, just most of the time”. Either way, com­mu­ni­ca­tion is bet­ter and progress is made.

In the next post, I will be gen­er­al­iz­ing Prov­ing too much. In the mean­time, what’s wrong with this ques­tion:

If a tree falls in the woods, but no one is around to hear it, does it make a sound? (note: you shouldn’t be able to frame it as Prov­ing too much)

[Feel free to com­ment if you got differ­ent an­swers/​ gen­er­al­iza­tions/​ al­gorithms than I did. Same if you feel like you hit on some­thing in­ter­est­ing or that there’s a con­cept I missed. Ad­ding your own ex­am­ples with the Spoiler tag >! is en­couraged]

• Your Prov­ing Too Much dis­proves too much: If we only al­low rea­son­ing steps that always work, we never get real-world knowl­edge be­yond “I think, there­fore I am.”. Some of these rea­sons for be­lief make their be­lief more likely to be true, and qual­i­ta­tively that’s the best we can get.

• “I think, there­fore I am.”

(This is also in­cor­rect, be­cause con­sid­er­ing a think­ing you in a coun­ter­fac­tual makes sense. Many UDTish ex­am­ples demon­strate that this prin­ci­ple doesn’t hold.)

• Great cri­tique! I’ve up­dated the post to show the pur­pose of this frame. We already have Bayesian up­dat­ing for achiev­ing be­liefs to be more likely true. This fram­ing of Prov­ing Too Much is about differ­en­ti­at­ing use­ful rea­sons and those that claim”This rea­son leads to truth 100% of the time”.

There are lots of be­liefs that peo­ple hold with 100% con­fi­dence for only 1 rea­son, and this is sim­ply a way of jar­ring them out of that con­fi­dence. En­coun­ter­ing heuris­tics that ac­count for less than 100% con­fi­dence will be cov­ered in Find­ing Cruxes.

• I did all the ex­er­cises above. Here’s what I wrote down dur­ing the timed sec­tions. (It’s a stream of con­scious­ness ac­count, it may not be very clear/​un­der­stand­able.)

How would you gen­er­al­ize the com­mon prob­lem in the above ar­gu­ments? You have 2 minutes

The struc­ture of the rea­son­ing does not nec­es­sar­ily cor­re­late with one out­come more than oth­ers. You say A be­cause X, but I can ar­gue that B be­cause X.

But I’m con­fused, be­cause I can do this for any ar­gu­ment that’s not max­i­mally well-speci­fied though. Like, there’s always a gotcha. If I ar­gue for the struc­ture of ge­net­ics due to the pat­tern of chil­dren born with cer­tain fea­tures, I could also use that ev­i­dence com­bined with an anti-in­duc­tive prior to ar­gue the op­po­site. I’m not quite sure what the rea­son is that some things feel like they prove too much and some don’t. I sup­pose it’s just “in the con­text of my ac­tual un­der­stand­ing of the situ­a­tion, do I feel like this ar­gu­ment pins down a world-state pos­i­tively cor­re­lated with the be­lief or not?” and if it doesn’t, then I can neatly ex­press this by show­ing it can prove any­thing, be­cause it’s not ac­tu­ally real ev­i­dence.

Oh huh, maybe that’s wrong. It’s not that it isn’t ev­i­dence for any­thing, it’s that if it were ev­i­dence for this it would be ev­i­dence for many in­con­sis­tent things. (Though I think those two are the same.)

What al­gorithm were you run­ning when you solved the above prob­lems? Is there a more ideal/​gen­eral al­gorithm? You have 3 min­utes.

Hmm, I did like the thing that hap­pened ac­tu­ally. Nor­mally in such a dis­agree­ment with a per­son, I would ex­plain the struc­ture of my be­liefs around the thing they called a ‘rea­son’. I’d do lots of in­ter­pre­tive work like that. “Let me ex­plain the pro­cess by which smart peo­ple get their be­liefs and when those pro­cesses are/​aren’t truth-track­ing” or “Let me ex­plain what heuris­tics help pre­dict whether a star­tups is suc­cess­ful” or “let me ex­plain what p-hack­ing is”. But in all of them the new men­tal mo­tion was much cleaner/​cheaper, which was pro­duc­ing a small im­pos­si­bil­ity proof.

I think I nor­mally avoid such proofs be­cause they’re non-con­struc­tive—they don’t tell you where the mis­take was or how that part of the world works, and I’m of­ten wor­ried this will feel like a de­mo­ti­vat­ing thing or con­ver­sa­tion kil­ler for the other per­son I’m talk­ing with. But I think it’s worth think­ing this way for my­self more. I do want to prac­tice it, cer­tainly. I should be able to use all tools of proof and dis­proof, not just those that make con­ver­sa­tions go smoothly.

Some gen­eral thoughts

• I found do­ing the ex­er­cises very en­joy­able.

• I think that the an­swers here could’ve been more to-a-for­mat. Th­ese aren’t very open-ended ques­tions, and I think that if I’d prac­ticed match­ing a for­mat that would’ve drilled a more spe­cific tool bet­ter. But not clear that’s ap­pro­pri­ate.

• I didn’t like how all the ex­am­ples were of the “don’t be­lieve a dumb low-sta­tus thing”. Like I think peo­ple of­ten build episte­molo­gies around mak­ing sure to never be re­li­gious, en­dorse a failed startup idea, or be­lieve home­opa­thy, but I think that you should mostly build it around mak­ing sure you will make suc­cess­ful in­sights in physics, or build­ing a suc­cess­ful startup, which is a differ­ent frame. I would’ve liked much more difficult ex­am­ples in ar­eas where it’s not clear what the right choice is purely based on pat­tern-match­ing to low-sta­tus be­liefs.

• The post tells peo­ple to sit by a clock. I think at the start I would’ve told peo­ple to find a timer by googling ‘timer’ (when you do that, one just ap­pears on google) else I ex­pect most folks to have bounced off and not done those ex­er­cises.

• I re­ally liked the ‘re­flect on the gen­eral tech­nique’ sec­tions, they were ex­cel­lent and well-placed.

• Wow, this is ex­actly the type of feed­back I wanted, thank you!

I’ve changed my view on this, and my cur­rent model is the frame “I can prove any­thing in the set A be­cause of rea­son X”

Like I can prove a cer­tain set of facts about Nat­u­ral num­bers us­ing in­duc­tion, but to claim that in­duc­tion proves all things about Real num­bers or moral­ity or… is prov­ing too much.

I would rewrite the post to fo­cus on ques­tions re­gard­ing that such as:

1. What set of claims do you think rea­son X proves?

2. How do you know that rea­son X proves those types of claims? (And of course figure out how to phrase these things more tact­fully)

Also, I’ve also en­joyed Think­ing Physics and TurnTrout’s AU se­quence type ques­tions over my “pat­tern match to low-sta­tus be­lief” ones (I do like my gen­er­al­iza­tion and al­gorithm ques­tion though), so I think I un­der­stand your point there.

• Ex. 8: I be­lieve in a soul be­cause I have a re­ally strong gut feel­ing.

OTOH, think­ing with­out any in­tu­itions or as­sump­tions at all re­mains an un­solved prob­lem.

• I’ve up­dated the post! The pur­pose of my fram­ing isn’t to deny use­ful heuris­tics always, it’s to deny heuris­tics that claim to be 100% cor­rect always.

• Re­gard­ing ex­am­ple 4. Believ­ing some­thing be­cause a re­ally smart per­son be­lieves it is not a bad heuris­tic, as long as you aren’t cherry-pick­ing the re­ally smart per­son. prepri­ors, then ev­ery­one says, you had the mis­for­tune of be­ing born wrong, I’m lucky enough to be born right. If you were trans­ported to an al­ter­nate re­al­ity, where half the pop­u­la­tion thought 2+2=4, and half thought 2+2=7, would you be­come un­cer­tain, or would you just think that the 2+2=7 pop­u­la­tion were wrong?

The ar­gu­ment about be­liev­ing in Cthulhu be­cause that was how you were raised prov­ing too much, it­self proves too much.

Re­gard­ing ex­am­ple 4. Believ­ing some­thing be­cause a re­ally smart per­son be­lieves it is not a bad heuris­tic, as long as you aren’t cherry-pick­ing the re­ally smart per­son. If you have data about many smart peo­ple, tak­ing the av­er­age is an even bet­ter heuris­tic. As is fo­cus­ing on the smart peo­ple that are ex­perts in some rele­vant field. The use­ful­ness of this heuris­tic also de­pends on your defi­ni­tion of ‘smart’. There are a few peo­ple with a high IQ, a pow­er­ful brain ca­pa­ble of think­ing and re­mem­ber­ing well, but who have very poor episte­mol­ogy, and are cre­ation­ists or Scien­tol­o­gists. Many defi­ni­tions of smart would rule out these peo­ple, re­quiring some ra­tio­nal­ity skills of some sort. This makes the smart peo­ple heuris­tic even bet­ter.

• Thanks for the spe­cific ex­am­ples on why this is wrong! I’ve up­dated the post to state the use­ful­ness of this tech­nique. So you don’t have to search for it:

Im­por­tant Note: the pur­pose of this frame isn’t to win an ar­gu­ment/​ prove any­thing. It’s to differ­en­ti­ate be­tween heuris­tics that claim 100% suc­cess rates vs ones that claim a more ac­cu­rate es­ti­mates. Imag­ine “I’m 100% con­fi­dent I’ll roll 7′s with my two die cause of my luck!” vs “There’s a 636 chance I’ll roll 7′s be­cause I’m as­sum­ing two fair die”

So for ex­am­ple 4, ap­peal to au­thor­ity may be a use­ful heuris­tic, but if that’s the only rea­son they be­lieve in evolu­tion with 100% con­fi­dence, then show­ing it Proves Too Much is use­ful. Does this satisfy your cri­tique?

• Fair enough, I think that satis­fies my cri­tique.

A full con­sid­er­a­tion of prov­ing too much re­quires that we have un­cer­tainty both over what ar­gu­ments are valid, and over the real world. The un­cer­tainty about what ar­gu­ments are valid, along with our in­abil­ity to con­sider all pos­si­ble ar­gu­ments makes this type of rea­son­ing work. If you see a par­tic­u­lar type of ar­gu­ment in fa­vor of con­clu­sion X, and you dis­agree with con­clu­sion X, then that gives you ev­i­dence against that type of ar­gu­ment.

This is used in moral ar­gu­ments too. Con­sider the ar­gu­ment that touch­ing some­one re­ally gen­tly isn’t wrong. And if it isn’t wrong to touch some­one with force F, then it isn’t wrong to touch them with force F+0.001 New­tons. There­fore, by in­duc­tion, it isn’t wrong to punch peo­ple as hard as you like.

Now con­sider the ar­gu­ment that 1 grain of sand isn’t a heap. If you put a grain of sand down some­where that there isn’t already a heap of sand, you don’t get a heap. There­fore by in­duc­tion, no amount of sand is a heap.

If you were un­sure about the moral­ity of punch­ing peo­ple, but knew that heaps of sand ex­isted, then see­ing the first ar­gu­ment would make you up­date to­wards “punch­ing peo­ple is ok”. When you then see the sec­ond ar­gu­ment, you up­date to “in­duc­tive ar­gu­ments don’t work in the real world.” and re­verse the pre­vi­ous up­date about punch­ing peo­ple.

See­ing an ar­gu­ment for a con­clu­sion that you don’t be­lieve can make you re­duce your cre­dence on other state­ments sup­ported by similar ar­gu­ments.

• I re­ally like this! I think my model is now:

If a heuris­tic claims 100% suc­cess rate in a spe­cific con­text, one can show it proves too much by prov­ing a counterexample

In­spired by your in­duc­tion ex­am­ple, in­duc­tion is very use­ful for proofs re­gard­ing Nat­u­ral num­bers, but it breaks down in the con­text of moral rea­son­ing (or even just the con­text of Real num­bers).

This is a bet­ter frame of Prov­ing Too Much than I have framed it in this post. I will need to ei­ther edit the post or make a new one and link it at the top of this ar­ti­cle. Either way thanks!

With that said, I don’t think this cap­tures your point of un­cer­tainty over both valid ar­gu­ments and pos­si­ble wor­lds. Would you elab­o­rate on how your point re­lates to the above, up­dated model?

• “If a tree falls in the woods, but no one is around to hear it, does it make a sound?” doesn’t sound like an ar­gu­ment, but a ques­tion. “Yes, be­cause the pres­ence of a per­son with ears doesn’t af­fect the phys­i­cal be­hav­ior of the air” or “No, be­cause air waves shouldn’t be con­sid­ered sound un­til they in­ter­act with a mind” are ar­gu­ments.

Or do you mean “ar­gu­ment” in the sense of a de­bate or dis­cus­sion (as in “we’re hav­ing an ar­gu­ment about X”)?

• You’re right! I could con­strue it to mean “it gen­er­ally leads to ar­gu­ments”, but I just ed­ited it to “ques­tion” to avoid fu­ture con­fu­sion.