Scholarship: how to tell good advice from bad advice?

Lukeprog has done some great posts talking about how scholarship has benefited him personally and how to do it. However, he hasn’t talked much about how to tell the good from the bad, and I think this is an especially big problem with it comes to “how to”s, rather than standard academic subjects.

The thing is this: writing on academic subjects tends to be dominated by academics seeking to gain status from their work. Not all fields have a strong correlation between status and making actual discoveries, but many (perhaps most) do. So for academic subjects, the big challenge when it comes to making sure you’re not reading nonsense is to try to evaluate the soundness of the field in question.

When it comes to seeking out “how to” advice, though, writing on the subjects seems to be dominated by people looking to make money selling advice. That means that if bad advice will sell better than good advice, they’ll give the bad advice. Even if their writing includes legitimate advice, they have incentives to give a distorted picture of the subject they’re writing on if it will sell better (for example, making whatever they’re talking about sound easier than it really is). In some cases, advice books end up trying to nudge you towards buying some more expensive thing the author is selling.

So: when you’re researching a “how to” topic, how do you tell the good advice from the bad advice?