How to be skeptical
Community
The Center For Applied Rationality (CFAR) checklist is a heuristic for assessing the admissibility of one’s own testimony.
What of the challenge of evaluating the testimony of others?
Slapping the label of a bias on a situation?
Arguing at the object level by provision of evidence to the contrary?
This risks Gish Gallop. For those who prefer to pick their battles, I commisioned this post of my time, a structural intervention into the information ecosystem.
We need not event the wheel, for legal theorists have researched this issue for years, while practitioners and courts have identified heuristics useful to lay people interested in this field.
Precedent
The Daubert standard provides a rule of evidence regarding the admissibility of expert witnesses’ testimony during United States federal legal proceedings. Pursuant to this standard, a party may raise a Daubert motion, which is a special case of motion in limine raised before or during trial to exclude the presentation of unqualified evidence to the jury. The Daubert trilogy refers to the three United States Supreme Court cases that articulated the Daubert standard:
-https://en.wikipedia.org/wiki/Daubert_standard
Further reading on the case is available here on Google Scholar
Practice
How can this be applied in practice?
What is the first principle of skepticism. It’s effectively synonymous: ‘question’
What question? This isn’t the 5 W’s of primary school, after all.
I have summarized critical questions to a reading here to get the ball rolling:
Issues to consider when contesting and evaluating expert opinion evidence
A. Relevance (on the voir dire)
I accept that you are highly qualified and have extensive experience, but how do we know that your level of performance regarding . . . [the task at hand — eg, voice comparison] is actually better than that of a lay person (or the jury)?
What independent evidence… [such as published studies of your technique and its accuracy] can you direct us to that would allow us to answer this question?
What independent evidence confirms that your technique works?
Do you participate in a blind proficiency testing program?
Given that you undertake blind proficiency exercises, are these exercises also given to lay persons to determine if there are significant differences in results, such that your asserted expertise can be supported?
B. Validation
Do you accept that techniques should be validated?
Can you direct us to specific studies that have validated the technique that you used?
What precisely did these studies assess (and is the technique being used in the same way in this case)?
Have you ever had your ability formally tested in conditions where the correct answer was known? (ie, not a previous investigation or trial)
Might different analysts using your technique produce different answers?
Has there been any variation in the result on any of the validation or proficiency tests you know of or participated in?
Can you direct us to the written standard or protocol used in your analysis?
Was it followed?
C. Limitations and errors
Could you explain the limitations of this technique?
Can you tell us about the error rate or potential sources of error associated with this technique?
Can you point to specific studies that provide an error rate or an estimation of an error rate for your technique?
How did you select what to examine?
Were there any differences observed when making your comparison . . . [eg, between two fingerprints], but which you ultimately discounted? On what basis were these discounted?
Could there be differences between the samples that you are unable to observe?
Might someone using the same technique come to a different conclusion?
Might someone using a different technique come to a different conclusion?
Did any of your colleagues disagree with you?
Did any express concerns about the quality of the sample, the results, or your interpretation?
Would some analysts be unwilling to analyse this sample (or produce such a confident opinion)?
...
D Personal proficiency
...
Have you ever had your own ability… [doing the specific task/using the technique] tested in conditions where the correct answer was known?
If not, how can we be confident that you are proficient?
If so, can you provide independent empirical evidence of your performance?
E Expressions of opinion
...
Can you explain how you selected the terminology used to express your opinion? Is it based on a scale or some calculation?
If so, how was the expression selected?
Would others analyzing the same material produce similar conclusions, and a similar strength of opinion? How do you know?
Is the use of this terminology derived from validation studies?
Did you report all of your results?
You would accept that forensic science results should generally be expressed in non-absolute terms?
-http://netk.net.au/Forensic/UNSW1.pdf
More
For further reading, I recommend the seminal text in cross-examination which is the 1903 The Art of Cross Examination.
The Full Text is available free here on Project Gutenberg.
Other countries use different standards, such as the Opinion Rule in Australia.
Did something get jumbled here? This isn’t right at all.
Grammar aside, the standard legal process and courts are really bad at reaching true conclusions. Taking their practices as wisdom seems likely to be quite bad.
The rules of evidence used by courts are not necessarily intended to establish the truth. There are sometimes other motives which might well be in clear opposition to that. For example, the exclusion of evidence from illegal searches is intended to discourage illegal searches. It is not intended to establish the truth, and in fact the rule obviously impedes the discovery of truth.
‘What evidence would convince me I am mistaken?’ If the answer is ‘no evidence’ I am probably not being skeptical.
The above based on the works of Karl Popper.
How has the checklist that you provide been validated?