Knowledge value = knowledge quality × domain importance

Months ago, my room­mate and I were dis­cussing some­one who had tried to repli­cate Seth Roberts’ but­ter mind self-ex­per­i­ment. My room­mate seemed to be mak­ing al­most no in­fer­ence from the per­son’s self-re­ports, be­cause they weren’t part of a sci­en­tific study.

But knowl­edge does not come in two grades, “sci­en­tific” and “use­less”. Anec­dotes do count as ev­i­dence, they are just weak ev­i­dence. And well de­signed sci­en­tific stud­ies con­sti­tute stronger ev­i­dence then poorly de­signed stud­ies. There’s a con­tinuum for knowl­edge qual­ity.

Know­ing that hu­mans are bi­ased should make us take their sto­ries and ad hoc in­fer­ences less se­ri­ously, but not dis­card them al­to­gether.


There ex­ists some do­mains where most of our knowl­edge is fairly low-qual­ity. But that doesn’t mean they’re not worth study, if the value of in­for­ma­tion in the do­main is high.

For ex­am­ple, a friend of mine read a bunch of books on ne­go­ti­a­tion and says this is the best one. Flip­ping through my copy, it looks like the au­thor is mostly just enu­mer­at­ing his own thoughts, sto­ries, and the­o­ries. So one might be tempted to dis­card the book en­tirely be­cause it isn’t very sci­en­tific.

But that would be a mis­take. If a smart per­son thinks about some­thing for a while and comes to a con­clu­sion, that’s de­cent-qual­ity ev­i­dence that the con­clu­sion is cor­rect. (If you dis­agree with me on this point, why do you think about things?)

And the value of in­for­ma­tion in the do­main of ne­go­ti­a­tion can be very high: If you’re a pro­fes­sional, be­ing able to ne­go­ti­ate your salary bet­ter can net you hun­dreds of thou­sands over the course of a ca­reer. (An­chor­ing means your salary next year will prob­a­bly just be an in­cre­men­tal raise from your salary last year, so start­ing salary is very im­por­tant.)

Similarly, this self-help book is about as dopey and un­scien­tific as they come. But do­ing one of the ex­er­cises from it years ago de­stroyed a large in­se­cu­rity of mine that I was only periph­er­ally aware of. So I prob­a­bly got more out of it in in­stru­men­tal terms than I would’ve got­ten out of a chem­istry text­book.

In gen­eral, self-im­prove­ment seems like a do­main of re­ally high im­por­tance that’s un­for­tu­nately flooded with low-qual­ity knowl­edge. If you in­vest two hours im­ple­ment­ing some self-im­prove­ment scheme and find your­self op­er­at­ing 10% more effec­tively, you’ll dou­ble your in­vest­ment in just a week, as­sum­ing a 40 hour work week. (ALERT: this seems like a re­ally im­por­tant point! I’d write an en­tire post about it, but I’m not sure what else there is to say.)

Here are some free self-im­prove­ment re­sources where the knowl­edge qual­ity seems at least mid­dling: For peo­ple who feel like failures. For stu­dents. For math­e­mat­i­ci­ans. Pro­duc­tivity and gen­eral ass kick­ing (web im­ple­men­ta­tion for that last idea). Even more ass kick­ing ideas that you might have seen already.