In the post ‘Four layers of Intellectual Conversation’, Eliezer says that both the writer of an idea, and the person writing a critique of that idea, need to expect to have to publicly defend what they say at least one time. Otherwise they can write something stupid and never lose status because they don’t have to respond to the criticism.

I was wondering about where this sort of dialogue happens in academia. I have been told by many people that current journals are quite terrible, but I’ve also heard a romantic notion that science (especially physics and math) used to be more effectively pursued in the early 20th century (Einstein, Turing, Shannon, etc). So Oliver and I thought we’d look at the journals to see if they had real conversations.

We looked at two data points, and didn’t find any.

First, Oliver looked through Einstein’s publication history (Oli is German and could read it). Einstein has lots of ‘reviews’ of others’ work in his list of publications, sometimes multiple of the same person, which seemed like a promising example of conversation. Alas, it turned out that Einstein had merely helped German journals write *summaries* of papers that had been written in English, and there was no real dialogue.

Second, I looked through a volume of the London Mathematical Society, in particular, the volume where Turing published his groundbreaking paper proving that not all mathematical propositions are decidable (thanks to sci-hub for making it possible for me to read the papers!). My eyes looked at about 60% of the pages in the journal (about 12 papers), and *not one of them* disagreed with any prior work. There was :

A footnote that thanked an advisor for finding a flaw in a proof

An addendum page (to the whole volume) that consisted of a single sentence thanking someone for showing one of their theorems was a special case of someone else’s theorem

One person who was skeptical of another person’s theorem. But that theorem by Ramanujan (who was famous for stating theorems without proofs), and the whole paper primarily found proofs of his other theorems.

There were lots of discussions of people’s work but always building, or extending, or finding a neater way of achieving the same results. Never disagreement, correction, or the finding of errors.

One thing that really confuses me about this is that *it’s really hard to get all the details right*. Lots of great works are filled with tiny flaws (e.g. Donald Knuth reliably has people find errors in his texts). So I’d expect any discussion of old papers to bring up flaws, or that journals would require a section at the end for corrections of the previous volume. There were of course reviewers, but they can’t be experts in all the areas.

But more importantly *where did/does the dialogue happen* *if not in the journals?*

If I try to be concrete about what I’m curious about:

As people go about the craft of doing science, they will make errors (conceptual mistakes, false proofs, and so on). One of the main pieces of infrastructure in academia are journals, where work gets published and can become common knowledge.

Two places to fix errors are pre-publication and post-publication. I don’t know much about the pre-publication process, but if it is strong enough to ensure no errors got published, I’d like some insight into what that process was like. Alternatively, if course-correction happened post-publication, I’m interested to know how and where, because when I looked (see above) I couldn’t find it.

There’s also the third alternative, that no progress was made. And there’s the fourth alternative, that most papers were bad and the thing scientists did was to just never read or build on them. I’m happy to get evidence for any of these, or a fifth alternative.

**Added: **Maybe this is a more crisp statement:

Why do (old) journals not claim to have errors in any of the papers? Is it because they’re (implicitly) lying about the quality of the papers? Or if there’s a reliable process that removed errors from 100% of papers, can someone tell me what that process was?

This post:

Tackles an important question. In particular, it seems quite valuable to me that someone who tries to build a platform for intellectual progress attempts to build their own concrete models of the domain and try to test those against history

It also has a spirit of empiricism and figuring things out yourself, rather than assuming that you can’t learning anything from something that isn’t an academic paper

Those are positive attributes and contribute to good epistemic norms on the margin. Yet at the same time, a culture of unchecked amateur research could end up in bad states, and reviews seem like a useful mechanism to protect against that

This makes this suitable for a nomination.

It’s a really important question, and the answers actually helped me answer it (though they were far from comprehensive).

You beat me too it!

I seem to have liked this question at the time.