Thanks for your well-considered post. Your departure will be a loss for the community, and sorry to see you go.
I also feel that some of the criticism you’re posting here might be due to a misunderstanding, mainly regarding the validity of thought experiments, and of reasoning by analogy. I think both of these have a valid place in rational thought, and have generally been used appropriately in the material you’re referring to. I’ll make an attempt below to elaborate.
Reasoning by analogy, or, the outside view
What you call “reasoning by analogy” is well described in the sequence on the outside view. However, as you say,
The fundamental mistake here is that reasoning by analogy is not in itself a sufficient explanation for a natural phenomenon, because it says nothing about the context sensitivity or insensitivity of the original example and under what conditions it may or may not hold true in a different situation.
This is exactly the same criticism that Eliezer has of outside-view thinking, detailed in the sequences!
Of course Robin Hanson has a different idea of what constitutes the reference class and so makes a rather different prediction—a problem I refer to as “reference class tennis”[...] But mostly I would simply decline to reason by analogy, preferring to drop back into causal reasoning in order to make weak, vague predictions.
You’re very right that the uncertainty in the AI field is very high. I hope that work is being done to get a few data points and narrow down the uncertainty, but don’t think that you’re the first to object to an over-reliance on “reasoning by analogy”. It’s just that when faced with a new problem with no clear reference class, it’s very hard to use the outside vew, but unfortunately also hard to trust predictions from a model which has sensitive parameters with high uncertainties.
Thought experiments are a tool of deduction, not evidence
We get instead definitive conclusions drawn from thought experiments only.
This is similar to complaining about people arriving at definitive conclusions drawn from mathematical derivation only.
I want to stress that this is not a problem in most cases, especially not in physics. Physics is a field in which models are very general and held with high confidence, but often hard to use to handle complicated cases. We have a number of “laws” in physics that we have fairly high certainty of; nonetheless, the implications of these laws are not clear, and even if we believe them we may be unsure of whether certain phenomena are permitted by these laws or not. Of course we also do have to test our basic laws, which is why we have CERN and such, especially because we suspect they are incomplete (thanks in part to thought experiments!).
A thought experiment is not data, and you do not use conclusions from thought experiments to update your beliefs as though the thought experiment were producing data. Instead, you use thought experiments to update your knowledge of the predictions of the beliefs you already have. You can’t just give an ordinary human the laws of physics written down on a piece of paper and expect them to immediately and fully understand the implications of the truth of those laws, or even to verify that the laws are not contradictory.
Thus, Einstein was able to use thought experiments very profitably to identify that the laws of classical mechanics (as formulated at the time) led to a contradiction with the laws of electrodynamics. No experimental evidence was needed; the thought experiment is a logical inference procedure which identifies one consequence of Maxwell’s equations being that light travels at speed ‘c’ in all reference frames, and shows that to be incompatible with Galilean relativity. A thought experiment, just like a mathematical proof-by-contradiction, can be used to show that certain beliefs are mutually inconsistent and one must be changed or discarded.
Thus, I take issue with this statement:
(thought experiments favored over real world experiments)
Thought experiments are not experiments at all, and cannot even be compared to experiments. They are a powerful tool for exploring theory, and should be compared to other tools of theory such as mathematics. Experiments are a powerful tool for checking your theory, but experiments alone are just data; they won’t tell you what your theory predicted, or whether your theory is supported or refuted by the data. Theory is a powerful tool for exploring the spaces of mutually compatible beliefs, but without data you cannot tell whether a theory has relevance to reality or not.
It would make sense to protest that thought experiments are being used instead of math, which some think is a more powerful tool for logical inference. On the other hand, math fails at being accessible to a wide audience, while thought experiments are. But the important thing is that thought experiments are similar to math in their purpose. They are not at all like experiments; don’t get their purposes confused!
Within Less Wrong, I have only ever seen thought experiments used for illustrating the consequences of beliefs, not for being taken as evidence. For example, the belief that “humans have self-sabotaging cognitive flaws, and a wide variation of talents” and the belief that “humans are about as intelligent as intelligent things can get” would appear to be mutually incompatible, but it’s not entirely obvious and a valid space to explore with thought experiments.
Hi Mark,
Thanks for your well-considered post. Your departure will be a loss for the community, and sorry to see you go.
I also feel that some of the criticism you’re posting here might be due to a misunderstanding, mainly regarding the validity of thought experiments, and of reasoning by analogy. I think both of these have a valid place in rational thought, and have generally been used appropriately in the material you’re referring to. I’ll make an attempt below to elaborate.
Reasoning by analogy, or, the outside view
What you call “reasoning by analogy” is well described in the sequence on the outside view. However, as you say,
This is exactly the same criticism that Eliezer has of outside-view thinking, detailed in the sequences!
In outside view as a conversation halter:
You’re very right that the uncertainty in the AI field is very high. I hope that work is being done to get a few data points and narrow down the uncertainty, but don’t think that you’re the first to object to an over-reliance on “reasoning by analogy”. It’s just that when faced with a new problem with no clear reference class, it’s very hard to use the outside vew, but unfortunately also hard to trust predictions from a model which has sensitive parameters with high uncertainties.
Thought experiments are a tool of deduction, not evidence
This is similar to complaining about people arriving at definitive conclusions drawn from mathematical derivation only.
I want to stress that this is not a problem in most cases, especially not in physics. Physics is a field in which models are very general and held with high confidence, but often hard to use to handle complicated cases. We have a number of “laws” in physics that we have fairly high certainty of; nonetheless, the implications of these laws are not clear, and even if we believe them we may be unsure of whether certain phenomena are permitted by these laws or not. Of course we also do have to test our basic laws, which is why we have CERN and such, especially because we suspect they are incomplete (thanks in part to thought experiments!).
A thought experiment is not data, and you do not use conclusions from thought experiments to update your beliefs as though the thought experiment were producing data. Instead, you use thought experiments to update your knowledge of the predictions of the beliefs you already have. You can’t just give an ordinary human the laws of physics written down on a piece of paper and expect them to immediately and fully understand the implications of the truth of those laws, or even to verify that the laws are not contradictory.
Thus, Einstein was able to use thought experiments very profitably to identify that the laws of classical mechanics (as formulated at the time) led to a contradiction with the laws of electrodynamics. No experimental evidence was needed; the thought experiment is a logical inference procedure which identifies one consequence of Maxwell’s equations being that light travels at speed ‘c’ in all reference frames, and shows that to be incompatible with Galilean relativity. A thought experiment, just like a mathematical proof-by-contradiction, can be used to show that certain beliefs are mutually inconsistent and one must be changed or discarded.
Thus, I take issue with this statement:
Thought experiments are not experiments at all, and cannot even be compared to experiments. They are a powerful tool for exploring theory, and should be compared to other tools of theory such as mathematics. Experiments are a powerful tool for checking your theory, but experiments alone are just data; they won’t tell you what your theory predicted, or whether your theory is supported or refuted by the data. Theory is a powerful tool for exploring the spaces of mutually compatible beliefs, but without data you cannot tell whether a theory has relevance to reality or not.
It would make sense to protest that thought experiments are being used instead of math, which some think is a more powerful tool for logical inference. On the other hand, math fails at being accessible to a wide audience, while thought experiments are. But the important thing is that thought experiments are similar to math in their purpose. They are not at all like experiments; don’t get their purposes confused!
Within Less Wrong, I have only ever seen thought experiments used for illustrating the consequences of beliefs, not for being taken as evidence. For example, the belief that “humans have self-sabotaging cognitive flaws, and a wide variation of talents” and the belief that “humans are about as intelligent as intelligent things can get” would appear to be mutually incompatible, but it’s not entirely obvious and a valid space to explore with thought experiments.