The core idea of the book is that science makes progress by forbidding non-empirical evaluation of hypotheses from publications, focusing on predictions and careful measurements while excluding philosophical interpretations (like Newton’s “I have not as yet been able to deduce from phenomena the reason for these properties of gravity, and I do not feign hypotheses. […] It is enough that gravity really exists and acts according to the laws that we have set forth.”).
The author basically argues that humans are bad at philosophical reasoning and get stuck in endless arguments, and so to make progress you have to ban it (from the main publications) and make it mandatory to make actual measurements (/math) - even when it seems irrational to exclude good (but not empirical) arguments.
It’s weird that the author doesn’t say explicitly “humans are bad at philosophical reasoning” while this feels to me like the essential takeaway.
I’m unsure to what extent this is true, but it’s an interesting claim.
The author doesn’t deny the importance of coming up with good hypotheses, and the role of philosophical reasoning for this part of the process, but he would say that there is clear progress decade by decade only because people did not argue with Einstein by commenting on how crazy the theory was, but instead by they tested the predictions Einstein’s theories made—because that’s the main kind of refutation allowed in scientific venues [Edit: That specific example is wrong and is not in the book, see the comments below.]. Same for evolution, it makes a ton of predictions (though at the time what theory the evidence favored was ambiguous). Before the scientific revolution, lots of people had good ideas, but 1. they had little data to use in their hypotheses’ generation process, and 2. the best ideas had a hard time rising to the top because people argued using arguments instead of collecting data.
(The book also has whole chapters on objectivity, subjectivity, “credibility rankings”, etc. where Bayes and priors aren’t mentioned once. It’s quite sad the extent to which you have to go when you don’t want to scare people with math / when you don’t know math)
Application to AI safety research:
The endless arguments and different schools of thought around the likelihood of scheming and the difficulty of alignment look similar to the historical depictions of people who didn’t know what was going on and should have focused on making experiments.
This makes me more sympathetic to the “just do some experiments” vibe some people, even when it seems like reasoning should be enough if only people understood each other’s arguments.
This makes me more sympathetic towards reviewers/conference organizers rejecting AI safety papers that are mostly about making philosophical points (the rejection may make sense even if the arguments look valid to them).
IIRC Einstein’s theory had a pretty immediate impact on publication on a lot of top physicists even before more empirical evidence came in. Wikipedia on the history of relativity says:
Walter Kaufmann (1905, 1906) was probably the first who referred to Einstein’s work. He compared the theories of Lorentz and Einstein and, although he said Einstein’s method is to be preferred, he argued that both theories are observationally equivalent. Therefore, he spoke of the relativity principle as the “Lorentz–Einsteinian” basic assumption.[76] Shortly afterwards, Max Planck (1906a) was the first who publicly defended the theory and interested his students, Max von Laue and Kurd von Mosengeil, in this formulation. He described Einstein’s theory as a “generalization” of Lorentz’s theory and, to this “Lorentz–Einstein Theory”, he gave the name “relative theory”; while Alfred Bucherer changed Planck’s nomenclature into the now common “theory of relativity” (“Einsteinsche Relativitätstheorie”). On the other hand, Einstein himself and many others continued to refer simply to the new method as the “relativity principle”. And in an important overview article on the relativity principle (1908a), Einstein described SR as a “union of Lorentz’s theory and the relativity principle”, including the fundamental assumption that Lorentz’s local time can be described as real time. (Yet, Poincaré′s contributions were rarely mentioned in the first years after 1905.) All of those expressions, (Lorentz–Einstein theory, relativity principle, relativity theory) were used by different physicists alternately in the next years.[77]
Following Planck, other German physicists quickly became interested in relativity, including Arnold Sommerfeld, Wilhelm Wien, Max Born, Paul Ehrenfest, and Alfred Bucherer.[78] von Laue, who learned about the theory from Planck,[78] published the first definitive monograph on relativity in 1911.[79] By 1911, Sommerfeld altered his plan to speak about relativity at the Solvay Congress because the theory was already considered well established.[78]
Overall I don’t think Einstein’s theories seemed particularly crazy. I think they seemed quite good almost immediately after publication, without the need for additional experiments.
Tiny review of The Knowledge Machine (a book I listened to recently)
The core idea of the book is that science makes progress by forbidding non-empirical evaluation of hypotheses from publications, focusing on predictions and careful measurements while excluding philosophical interpretations (like Newton’s “I have not as yet been able to deduce from phenomena the reason for these properties of gravity, and I do not feign hypotheses. […] It is enough that gravity really exists and acts according to the laws that we have set forth.”).
The author basically argues that humans are bad at philosophical reasoning and get stuck in endless arguments, and so to make progress you have to ban it (from the main publications) and make it mandatory to make actual measurements (/math) - even when it seems irrational to exclude good (but not empirical) arguments.
It’s weird that the author doesn’t say explicitly “humans are bad at philosophical reasoning” while this feels to me like the essential takeaway.
I’m unsure to what extent this is true, but it’s an interesting claim.
The author doesn’t deny the importance of coming up with good hypotheses, and the role of philosophical reasoning for this part of the process, but he would say that there is clear progress decade by decade only because people did not argue with Einstein by commenting on how crazy the theory was, but instead by they tested the predictions Einstein’s theories made—because that’s the main kind of refutation allowed in scientific venues [Edit: That specific example is wrong and is not in the book, see the comments below.]. Same for evolution, it makes a ton of predictions (though at the time what theory the evidence favored was ambiguous). Before the scientific revolution, lots of people had good ideas, but 1. they had little data to use in their hypotheses’ generation process, and 2. the best ideas had a hard time rising to the top because people argued using arguments instead of collecting data.
(The book also has whole chapters on objectivity, subjectivity, “credibility rankings”, etc. where Bayes and priors aren’t mentioned once. It’s quite sad the extent to which you have to go when you don’t want to scare people with math / when you don’t know math)
Application to AI safety research:
The endless arguments and different schools of thought around the likelihood of scheming and the difficulty of alignment look similar to the historical depictions of people who didn’t know what was going on and should have focused on making experiments.
This makes me more sympathetic to the “just do some experiments” vibe some people, even when it seems like reasoning should be enough if only people understood each other’s arguments.
This makes me more sympathetic towards reviewers/conference organizers rejecting AI safety papers that are mostly about making philosophical points (the rejection may make sense even if the arguments look valid to them).
Did Einstein’s theory seem crazy to people at the time?
IIRC Einstein’s theory had a pretty immediate impact on publication on a lot of top physicists even before more empirical evidence came in. Wikipedia on the history of relativity says:
Overall I don’t think Einstein’s theories seemed particularly crazy. I think they seemed quite good almost immediately after publication, without the need for additional experiments.
Thanks for the fact check. I was trying to convey the vibe the book gave me, but I think this specific example was not in the book, my bad!
Thanks! And makes sense, you did convey the vibe. And good to know it isn’t in the book.