How’s that Epistemic Spot Check Project Coming?

Link post

Quick con­text: Epistemic spot checks started as a pro­cess in which I did quick in­ves­ti­ga­tions a few of a book’s early claims to see if it was trust­wor­thy be­fore con­tin­u­ing to read it, in or­der to avoid wast­ing time on books that would teach me wrong things. Epistemic spot checks worked well enough for catch­ing ob­vi­ous flaws (*cou*Carol Dweck*ugh*), but have a num­ber of prob­lems. They em­pha­size a trust/​don’t trust bi­nary over model build­ing, and prov­abil­ity over im­por­tance. They don’t han­dle “severely flawed but deeply in­sight­ful” well at all. So I started try­ing to cre­ate some­thing bet­ter.

Below are some scat­tered ideas I’m play­ing with that re­late to this pro­ject. They’re by no means fully baked, but it seemed like it might be helpful to share them. This kind of as­sumes you’ve been fol­low­ing my jour­ney with epistemic spot checks at least a lit­tle. If you haven’t that’s fine, a more pol­ished ver­sion of these ideas will come out even­tu­ally.

A parable in Three Books.

I’m cur­rently at­tempt­ing to write up an in­ves­ti­ga­tion of Chil­dren and Child­hood in Ro­man Italy (Beryl Raw­son) (af­fili­ate link) (Roam notes). This is very slow go­ing, be­cause CaCiRI doesn’t seem to have a the­sis. At least, I haven’t found one, and I’ve read al­most half of the con­tent. It’s just a bunch of facts. Often not even syn­the­ses, just “Here is one par­tic­u­lar statue and some things about it.” I rec­og­nize that this is im­por­tant work, even the kind of work I’d use to ver­ify an­other book’s claims. But as a fo­cal source, it’s deadly bor­ing to take notes on and very hard to write any­thing in­ter­est­ing about. What am I sup­posed to say? “Yes, that 11 year old did do well (with­out win­ning) in a po­etry com­pe­ti­tion and it was men­tioned on his funeral al­tar, good job re­port­ing that.” I want to la­bel this sin “weed based pub­lish­ing” (as in, “lost in the weeds”, al­though the fact that I have to ex­plain that is a ter­rible sign for it as a name).

One par­tic­u­lar bad sign for Chil­dren and Child­hood in Ro­man Italy was that I found my­self copy­ing mul­ti­ple sen­tences at once into my notes. Direct quot­ing can some­times mean “there’s only so many ways to ar­range these words and the au­thor did a perfectly good job so why bother”, but when it’s fre­quent, and long, it of­ten means “I can’t sum­ma­rize or dis­till what the au­thor is say­ing”, which can mean the au­thor is be­ing vague, elid­ing over im­por­tant points, or let­ting im­pli­ca­tions do work that should be made ex­plicit. This was eas­ier to no­tice when I was tak­ing notes in Roam (a work­flowy/​wiki hy­brid) be­cause Roam pushes me to make my bul­let points as self-con­tained as pos­si­ble (so when you re­fer them in iso­la­tion noth­ing is lost), so it be­came ob­vi­ous and un­pleas­ant when I couldn’t split a para­graph into self con­tained as­ser­tions. Ob­vi­ously real life is con­text-de­pen­dent and you shouldn’t try to make things more self-con­tained than they are, but I’m com­fortable say­ing fre­quent long quotes are a bad sign about a book.

On the other side you have The Un­bound Prometheus (David S. Lan­des) (af­fili­ate link) (Roam notes), which made sev­eral big, in­ter­est­ing, im­por­tant, sys­temic claims (e.g., “Bri­tain had a le­gal sys­tem more fa­vor­able to in­dus­tri­al­iza­tion than con­ti­nen­tal Europe’s”, “Europe had a more fa­vor­able cli­mate for sci­ence than Is­lamic re­gions”), none of which it pro­vided sup­port for (in the sec­tions I read- a friend tells me he gets more spe­cific later). I tried to in­ves­ti­gate these my­self and ended up even more con­fused- schol­ars can’t even agree on whether Bri­tain’s patent pro­tec­tions were strong or weak. I want to la­bel this sin “mak­ing me make your case for you”.

A Goldilocks book is The Fate of Rome (Kyle Harper) (af­fili­ate link) (Roam notes). Fate of Rome’s the­sis is that the peak of the Ro­man em­pire cor­re­sponds with un­usu­ally fa­vor­able weather con­di­tions in the med­iter­anean. It backs this up with claims about cli­mate arche­ol­ogy, e.g., ice core data (claim 1, 2). This prompted nat­u­ral and re­ward­ing fol­low up ques­tions like “What is ice core ca­pa­ble of prov­ing?” and “What does it ac­tu­ally show?”. My note tak­ing sys­tem in Roam was su­perb at en­abling in­ves­ti­ga­tions of ques­tions like these (my an­swer).

Based on claims cre­ation, Against the Grain (James Scott) (af­fili­ate link) (Roam notes) is even bet­ter. It has both in­ter­est­ing high level mod­els (“set­tle­ment and states are differ­ent thing that came very far apart”, “states are en­tan­gled with grains in par­tic­u­lar”) and very spe­cific claims to back them up (“X was per­ma­nently set­tled in year Y but didn’t de­velop state­hood hal­l­marks A, B, and C un­til year Z”). It is very easy to see how that claim sup­ports that model, and the claim is about as easy to in­ves­ti­gate as it can be. It is still quite pos­si­ble that the claim is wrong or more con­tro­ver­sial than the au­thor is ad­mit­ting, but it’s some­thing I’ll be able to de­ter­mine in a rea­son­able amount of time. As op­posed to Un­bound Prometheus, where I still worry there’s a trove of data some­where that an­swers all of the ques­tions con­clu­sively and I just failed to find it.

[Against the Grain was started as part of the Fore­cast­ing pro­ject, which is cur­rently be­ing re­worked. I can’t re­search its claims be­cause that would ruin our abil­ity to use it for the next round, should we choose to do so, so eval­u­a­tion is on hold.]

If you asked me to rate these books purely on ease-of-read­ing, the or­der­ing (start­ing with the eas­iest) would be:

  • Against the Grain

  • The Fate of Rome

  • Chil­dren and Child­hood in Ro­man Italy

  • The Un­bound Prometheus

Which is also very nearly the or­der they were pub­lished in (Against the Grain came out six weeks be­fore Fate of Rome; the oth­ers are sep­a­rated by decades). It’s pos­si­ble that the two mod­ern books were no bet­ter epistem­i­cally but felt so be­cause they were eas­ier to read. It’s also pos­si­ble it’s a co­in­ci­dence, or that epistemics have got­ten bet­ter in the last 50 years.

Model Based Reading

As is kind of im­plied in the parable above, one shift in Epistemic Spot Checks is a new em­pha­sis on ex­tract­ing and eval­u­at­ing the au­thor’s mod­els, which in­cludes an em­pha­sis on find­ing load bear­ing facts. I feel dumb for not em­pha­siz­ing this sooner, but bet­ter late than never. I think the real trick here is not iden­ti­fy­ing that know­ing a book’s mod­els are good, but cre­at­ing tech­niques for how to do that.

How do we Know This?

The other con­cept I’m play­ing with is that “what we know” is in­ex­tri­ca­ble from “how we know it”. This is dan­ger­ously close to log­i­cal pos­i­tivism, which I dis­agree with my limited un­der­stand­ing of. And yet it’s re­ally im­proved my think­ing when do­ing his­tor­i­cal re­search.

This is a pretty strong re­ver­sal for me. I re­mem­ber strongly want­ing to just be told what we knew in my sci­ence classes in col­lege, not the ex­per­i­ments that re­vealed it. I’m now pretty sure that’s sci­en­tism, not sci­ence.

How’s it Go­ing with Roam?

When I first started tak­ing notes with Roam (note spel­ling), I was pretty high on it. Two months later, I’m pre­dictably lov­ing it less than I did (it no longer drives me to do real life chores), but still find it in­dis­pens­able. The big dis­cov­ery is that the delight it brings me is some­what book de­pen­dent- it’s great for Against the Grain or The Fate of Rome, but didn’t help nearly so much with Chil­dren and Child­hood in Ro­man Italy, be­cause it was most very on-the-ground facts that didn’t benefit from my ver­ifi­ca­tion sys­tem and long para­graphs that couldn’t be dis­am­biguated.

I was run­ning into a ton of prob­lems with Roam’s search not han­dling non-se­quen­tial words, but they seem to have fixed that. Search is still not ideal, but it’s at least usable

Roam is pretty slow. It’s cur­rently a race be­tween their perfor­mance im­prove­ments and my in­creas­ing hoard of #Claims.