That strikes me as really … odd.
To whom is the advice addressed? If something is actually untrue, and one has determined it to be untrue, then the task of being skeptical about it is finished.
I could probably find a loophole in the preceding statement, but it couldn’t possibly be what Bill James was referring to.
As for directing skepticism at [claims depending upon] things that are difficult to measure, well that seems like one step away from directing skepticism at claims depending on little evidence. Which is surely what we want to do. Again, there’s a loophole, but clearly not something Bill James was trying to point out.
I took the survey. Skipped out at the “unreasonably long” section. Will it handle things properly if I return to it another day?
Note, if you ask me question that I can look up in two seconds flat, and the next question is “without checking sources, assess the probability of the last answer being correct” then I’m not sure you’re going to get the results you’re looking for. I consider the Internet as part of my partly trustable memory that I reference when I want to achieve success in the world I.e. all the time—but its not clear that’s a commonly held opinion.