I’m pretty confused by your numbered list, because they seem directly in contradiction with how scientific journals have worked historically. Here’s a quote from an earlier post of mine:
I looked through a volume of the London Mathematical Society, in particular, the volume where Turing published his groundbreaking paper proving that not all mathematical propositions are decidable (thanks to sci-hub for making it possible for me to read the papers!). My eyes looked at about 60% of the pages in the journal (about 12 papers), and not one of them disagreed with any prior work. There was :
A footnote that thanked an advisor for finding a flaw in a proof
An addendum page (to the whole volume) that consisted of a single sentence thanking someone for showing one of their theorems was a special case of someone else’s theorem
One person who was skeptical of another person’s theorem. But that theorem by Ramanujan (who was famous for stating theorems without proofs), and the whole paper primarily found proofs of his other theorems.
There were lots of discussions of people’s work but always building, or extending, or finding a neater way of achieving the same results. Never disagreement, correction, or the finding of errors.
I think that many-to-most papers published in scientific journals are basically on unhelpful questions and add little to the field, and I’d bet some of the proofs are false. And yet it seems to be very rarely published that they’re unhelpful or wrong or even criticized in the journals. People build on the good content, and forget about the rest. And journals provide a publishing house for the best ideas at a given time. (Not too dissimilar to the annual LW Review.)
It seems to me that low-quality content is indeed pretty low cost if you have a good filtering mechanism for the best content, and an incentive for people to produce great content. I think on the margin I am interested in creating more of both — better filtering and stronger incentives. That is where my mind currently goes when I think of ways to improve LessWrong.
This seems like a very odd response given the fact of the replication crisis, the many cases of scientific knowledge being forgotten for decades after being discovered, the rise of false or inaccurate (and sometimes quite harmful!) models, etc.
I think that (in many, or most, scientific fields) people often don’t build on the good content, and don’t forget about the bad content; often, the reverse happens. It’s true that it’s “very rarely published that [bad papers/proofs are] unhelpful or wrong or even criticized in the journals”! But this is, actually, very bad, and is a huge reason why more and more scientific fields are being revealed to be full of un-replicable nonsense, egregious mistakes, and even outright fraud! The filtering mechanisms we have are actually quite poor.
The “papers in scientific journals” example / case study seems to me to yield clear, strong support for my view.
It’s very worthwhile to understand the ways in which academia has died over the last 60 years or so, and part of it definitely involves failures in the journal system. But the axis of public criticism in journals doesn’t seem at all to have been what changed in the last 60 years? Insofar as you think that’s a primary reason, you seem to be explaining a change by pointing to a variable that has not changed.
In replying to your proposed norms, it’s not odd to point out that the very mechanism of labeling everything that’s bad as bad and ensuring we have common knowledge of it, was not remotely present when science was at its most productive — when Turing was inventing his machines, or when Crick & Watson were discovering the structure of DNA. In fact it seems to have been actively opposed in the journal system, because you do not get zero criticism without active optimization for it. That is why it seems to me to be strong evidence against the system you propose.
There may be a system that works via criticizing everything bad in public, but when science was most successful it did not, and instead seems to me to be based around the system I describe (a lot of submissions and high reward for success, little punishment for failure).
I’m pretty confused by your numbered list, because they seem directly in contradiction with how scientific journals have worked historically. Here’s a quote from an earlier post of mine:
I think that many-to-most papers published in scientific journals are basically on unhelpful questions and add little to the field, and I’d bet some of the proofs are false. And yet it seems to be very rarely published that they’re unhelpful or wrong or even criticized in the journals. People build on the good content, and forget about the rest. And journals provide a publishing house for the best ideas at a given time. (Not too dissimilar to the annual LW Review.)
It seems to me that low-quality content is indeed pretty low cost if you have a good filtering mechanism for the best content, and an incentive for people to produce great content. I think on the margin I am interested in creating more of both — better filtering and stronger incentives. That is where my mind currently goes when I think of ways to improve LessWrong.
This seems like a very odd response given the fact of the replication crisis, the many cases of scientific knowledge being forgotten for decades after being discovered, the rise of false or inaccurate (and sometimes quite harmful!) models, etc.
I think that (in many, or most, scientific fields) people often don’t build on the good content, and don’t forget about the bad content; often, the reverse happens. It’s true that it’s “very rarely published that [bad papers/proofs are] unhelpful or wrong or even criticized in the journals”! But this is, actually, very bad, and is a huge reason why more and more scientific fields are being revealed to be full of un-replicable nonsense, egregious mistakes, and even outright fraud! The filtering mechanisms we have are actually quite poor.
The “papers in scientific journals” example / case study seems to me to yield clear, strong support for my view.
It’s very worthwhile to understand the ways in which academia has died over the last 60 years or so, and part of it definitely involves failures in the journal system. But the axis of public criticism in journals doesn’t seem at all to have been what changed in the last 60 years? Insofar as you think that’s a primary reason, you seem to be explaining a change by pointing to a variable that has not changed.
In replying to your proposed norms, it’s not odd to point out that the very mechanism of labeling everything that’s bad as bad and ensuring we have common knowledge of it, was not remotely present when science was at its most productive — when Turing was inventing his machines, or when Crick & Watson were discovering the structure of DNA. In fact it seems to have been actively opposed in the journal system, because you do not get zero criticism without active optimization for it. That is why it seems to me to be strong evidence against the system you propose.
There may be a system that works via criticizing everything bad in public, but when science was most successful it did not, and instead seems to me to be based around the system I describe (a lot of submissions and high reward for success, little punishment for failure).