I like this object level review. I think that epistemic spot checks are a good idea overall, and like the focus on extracting the underlying model. However, I am worried that just checking the citations that are actually provided in the book makes this sort of check non-robust against the type of selection bias that Scott describes in part 4 of this post. Do you have any idea’s for dealing with that? I’ll describe mine after a line break for people who would like to consider the problem.
I think it is worth looking at existing reviews from experts, as well as choosing a small but representative sample of new to you concepts from the book to research. Or possiblity to pick a few claims and look specifically for counter examples.
I agree that “your own citations support your statements” is a low bar. In general I consider a book’s own citations being good and saying what they claim to be the bare minimum, and if something fails that I won’t go farther. I made an exception in this case because it came so well recommended. For the second chapter I reviewed I happened to have a lot of implicit understanding that would have taken a long time to formalize, via my neurologist and research I did years ago verifying things my neurologist told me.
What you suggest about looking at experts seems like an excellent way to hone in on concepts; I think I’m approaching the same goal, segmented in a different way. I read multiple books in the same vein and run these checks on all of them, and synthesize understanding from there. Segmenting it by book frees up a lot of RAM for me.
I’m torn on looking for counterexamples. On one hand, it’s an easy way to disprove wrong things. On the other hand, even the best theory is fairly easy to poke holes in, especially if you consider claims in isolation rather than the system as a whole. I find if I focus on disproving things, it inhibits understanding and learning, and leads to rejecting things that are imperfect but useful.
Possible solutions to both of these are to involve other people. For synthesizing concepts, you can have each person read on related topics and come together to discuss interactions. For looking for counterexamples, one person can focus on that while the other focuses on understanding in a generous way, without worrying about looking stupid for believing something wrong.
I like this object level review. I think that epistemic spot checks are a good idea overall, and like the focus on extracting the underlying model. However, I am worried that just checking the citations that are actually provided in the book makes this sort of check non-robust against the type of selection bias that Scott describes in part 4 of this post. Do you have any idea’s for dealing with that? I’ll describe mine after a line break for people who would like to consider the problem.
I think it is worth looking at existing reviews from experts, as well as choosing a small but representative sample of new to you concepts from the book to research. Or possiblity to pick a few claims and look specifically for counter examples.
I agree that “your own citations support your statements” is a low bar. In general I consider a book’s own citations being good and saying what they claim to be the bare minimum, and if something fails that I won’t go farther. I made an exception in this case because it came so well recommended. For the second chapter I reviewed I happened to have a lot of implicit understanding that would have taken a long time to formalize, via my neurologist and research I did years ago verifying things my neurologist told me.
What you suggest about looking at experts seems like an excellent way to hone in on concepts; I think I’m approaching the same goal, segmented in a different way. I read multiple books in the same vein and run these checks on all of them, and synthesize understanding from there. Segmenting it by book frees up a lot of RAM for me.
I’m torn on looking for counterexamples. On one hand, it’s an easy way to disprove wrong things. On the other hand, even the best theory is fairly easy to poke holes in, especially if you consider claims in isolation rather than the system as a whole. I find if I focus on disproving things, it inhibits understanding and learning, and leads to rejecting things that are imperfect but useful.
Possible solutions to both of these are to involve other people. For synthesizing concepts, you can have each person read on related topics and come together to discuss interactions. For looking for counterexamples, one person can focus on that while the other focuses on understanding in a generous way, without worrying about looking stupid for believing something wrong.