Epistemic status: Exploratory, speculative, half-baked thought
It’s a worldwide optimization problem. What content to consume under what conditions to reach a particular goal? Taking a step back and looking only at academia, and constraining ourselves to only academic papers (and preprints for simplicity), the question boils down to What papers should I read in what sequence to understand this field/problem, to figure out what the relevant problems are, etc.? (slight abuse of the semantics of “boils down to”, but what can you do).
One clear metric is to look at the citations of a paper, but that will be of no use when the field in question is evolving at a rapid pace (e.g. AI) and most of the relevant papers are recent preprints on arXiv. There are of course other metrics (number of retweets on Twitter, endorsements from the community on discussion forums, etc.), but as far as I know, there isn’t a widely accepted measure of the epistemic status of scientific papers. I saw many posts on LessWrong declaring their epistemic status explicitly before the main content. This seems like a great proxy for figuring out whether you want to read something or not. For instance, if you’re looking for a tutorial on training sparse autoencoders on transformer activations, you’re not likely going to find what you’re looking for in a 20 minute read-long post with Epistemic status: philosophical, speculative, even if it mentions sparse autoencoders in the title. It’s fair to say that the epistemic status of a particular post is implicit in the platform it’s published on, i.e. every platform attracts a particular type of content, but I’d argue that this is not the case for preprints on arXiv, where you find both ML papers of the type “we tried a new architecture and it performed X% better on this benchmark” as well as those of the type “we propose this framework for thinking about consciousness”. And the greater issue at hand, which is why I think an explicit measure of epistemic status might be useful, is that you often can’t tell the epistemic status of the paper simply by reading the title, abstract, and perhaps not even after you skim through it (though I leave open the possibility that this is not as common as I make it to be). Further, given the large quantity of preprints, it’s highly likely that certain works will overlap without each other’s knowledge, which is another aspect that could be captured by an externally imposed epistemic status.
Here, I propose the ESI (Epistemic Status Index), a measure of cumulative epistemic status as perceived by people who read the work in question and contributed their own subjective opinion of its epistemic status. The precise calculation of the index is yet to be determined (as is its usefulness), but the core idea is to categorize preprints into different categories describing their epistemic status. Below, I propose a few categories (adapted and extended from https://www.lesswrong.com/posts/Hrm59GdN2yDPWbtrd/feature-idea-epistemic-status).
Epistemic Status Categories
exploratory
authoritative
speculative
theoretical
empirical
preliminary
esoteric
methodological
review
philosophical
For now the metric would just work by displaying the categories with the largest number of votes (along with the actual number of votes as presumably more votes will give a more faithful distribution. Also, ESI is not an index of quality, but rather the type of epistemic uncertainty you’re likely to resolve by reading the paper.) A viable way to introduce this metric into arXiv could be through a browser extension.
Measuring pre-peer-review epistemic status
Epistemic status: Exploratory, speculative, half-baked thought
It’s a worldwide optimization problem. What content to consume under what conditions to reach a particular goal? Taking a step back and looking only at academia, and constraining ourselves to only academic papers (and preprints for simplicity), the question boils down to What papers should I read in what sequence to understand this field/problem, to figure out what the relevant problems are, etc.? (slight abuse of the semantics of “boils down to”, but what can you do).
One clear metric is to look at the citations of a paper, but that will be of no use when the field in question is evolving at a rapid pace (e.g. AI) and most of the relevant papers are recent preprints on arXiv. There are of course other metrics (number of retweets on Twitter, endorsements from the community on discussion forums, etc.), but as far as I know, there isn’t a widely accepted measure of the epistemic status of scientific papers. I saw many posts on LessWrong declaring their epistemic status explicitly before the main content. This seems like a great proxy for figuring out whether you want to read something or not. For instance, if you’re looking for a tutorial on training sparse autoencoders on transformer activations, you’re not likely going to find what you’re looking for in a 20 minute read-long post with Epistemic status: philosophical, speculative, even if it mentions sparse autoencoders in the title. It’s fair to say that the epistemic status of a particular post is implicit in the platform it’s published on, i.e. every platform attracts a particular type of content, but I’d argue that this is not the case for preprints on arXiv, where you find both ML papers of the type “we tried a new architecture and it performed X% better on this benchmark” as well as those of the type “we propose this framework for thinking about consciousness”. And the greater issue at hand, which is why I think an explicit measure of epistemic status might be useful, is that you often can’t tell the epistemic status of the paper simply by reading the title, abstract, and perhaps not even after you skim through it (though I leave open the possibility that this is not as common as I make it to be). Further, given the large quantity of preprints, it’s highly likely that certain works will overlap without each other’s knowledge, which is another aspect that could be captured by an externally imposed epistemic status.
Here, I propose the ESI (Epistemic Status Index), a measure of cumulative epistemic status as perceived by people who read the work in question and contributed their own subjective opinion of its epistemic status. The precise calculation of the index is yet to be determined (as is its usefulness), but the core idea is to categorize preprints into different categories describing their epistemic status. Below, I propose a few categories (adapted and extended from https://www.lesswrong.com/posts/Hrm59GdN2yDPWbtrd/feature-idea-epistemic-status).
Epistemic Status Categories
exploratory
authoritative
speculative
theoretical
empirical
preliminary
esoteric
methodological
review
philosophical
For now the metric would just work by displaying the categories with the largest number of votes (along with the actual number of votes as presumably more votes will give a more faithful distribution. Also, ESI is not an index of quality, but rather the type of epistemic uncertainty you’re likely to resolve by reading the paper.) A viable way to introduce this metric into arXiv could be through a browser extension.