A “spot check” of a few of a book’s claims is supposed to a proxy for the accuracy of the rest of the claims, right?
Of course there are issues to work through. For example, you’d probably want to have a training set and a test set like people always do in machine learning to see if it’s just “what sticks” or whether you’ve actually found a signal that generalizes. And if you published your reasoning then people might game whatever indicators you discovered. (Should still work for older books though.) You might also find that most of the variability in accuracy is per-book rather than per-author or anything like that. (Alternatively, you might find that a book’s accuracy can be predicted better based on external characteristics than doing a few spot checks, if individual spot checks are comparatively noisy.) But the potential upside is much larger because it could help you save time deciding what to read on any subject.
A “spot check” of a few of a book’s claims is supposed to a proxy for the accuracy of the rest of the claims, right?
Of course there are issues to work through. For example, you’d probably want to have a training set and a test set like people always do in machine learning to see if it’s just “what sticks” or whether you’ve actually found a signal that generalizes. And if you published your reasoning then people might game whatever indicators you discovered. (Should still work for older books though.) You might also find that most of the variability in accuracy is per-book rather than per-author or anything like that. (Alternatively, you might find that a book’s accuracy can be predicted better based on external characteristics than doing a few spot checks, if individual spot checks are comparatively noisy.) But the potential upside is much larger because it could help you save time deciding what to read on any subject.
Anyway, just an idea.