Your description of Occam/Ockham’s razor is wrong - “entities must not be multiplied beyond necessity” is one common statement. This would give equal chances to both storms and sea monsters (barring, e.g. the separate observation of storms and the lack of observation of sea monsters), though it gives a greater chance to sea monsters than green scaly sea monsters.
Modern science uses a few variations on Occam’s razor that add the requirement that you don’t pull any information out of thin air, mostly captured by the Einstein quote “Make everything as simple as possible, but not simpler.”
And here at LW we often use a quantitative measurement of simplicity called Kolmogorov complexity, which is how complicated a computer has to be before it can output your hypothesis. Not in natural language, but in terms of actual properties.
The reason it makes sense to act as if natural language is how we should describe things is because when natural language reflects things we’ve already seen, it’s simpler (in terms of properties) to make hypotheses about the whole universe that reuse parts, rather than hypotheses that have lots of new parts all the time—each of your mini-hypotheses is really part of the bigger hypothesis “what is the universe like?”
But since the correspondence between natural language and “stuff we’ve already seen” isn’t perfect, this breaks down in places. For example, in natural language, the hypothesis “god did it” is almost unsurpassed in simplicity. The fossil record, rainbows, why light things fall as fast as heavy things in a vacuum. The reason Occam’s razor does not suggest that “god did it” is the best explanation for everything is because god is a very complicated concept despite being a short word. So when you use something like Kolmogorov complexity that measures the size of concepts rather than the number of letters, you get evolution, diffraction, and gravity.
The reason Occam’s razor does not suggest that “god did it” is the best explanation for everything is because god is a very complicated concept despite being a short word.
It’s more because “it” is a very complicated concept.
Kolmogorov complexity is not used at LessWrong; it is not used anywhere because it is uncomputable. Approximations of Kolmogorov complexity (replacing the Turing machine in the definition with something weaker) do not have the same nigh-magical properties that Kolmogorov complexity would have, if it were available.
Kolmogorov complexity is computable for some hypotheses, just not all (for each formal axiomatic system, there is an upper bound to the complexity of hypotheses that can have their complexity determined by the system). Anyways, while we can never use Kolmogorov complexity to analyze all hypotheses, I believe that Manfred merely meant that we use it as an object of study, rather than to implement full Solomonoff induction.
I am aware that my definition of Occam’s razor is not the “official” definition. However, it is the definition which I see used most often in discussions and arguments, which is why I chose it. The fact that this definition of Occam’s razor is common supports my claim that humans consider it a good heuristic.
Forgive me for my ignorance, as I have not studied Kolmogorov complexity in detail. As you suggest, it seems that human understanding of a “simple description” is not in line with Kolmogorov complexity.
I think the intention of my post may have been unclear. I am not trying to argue that natural language is a good way of measuring the complexity of statements. (I’m also not trying to argue that it’s bad.) My intention was merely to explore how humans understand the complexity of ideas, and to investigate how such judgements of complexity influence the way typical humans build models of the world.
The fact that human understanding of complexity is so far from Kolmogorov complexity indicates to me that if an AI were to model its environment using Kolmogorov complexity as a criterion for selecting models, the model it developed would be different from the models developed by typical humans. My concern is that this disparity in understanding of the world would make it difficult for most humans to communicate with the AI.
As you suggest, it seems that human understanding of a “simple description” is not in line with Kolmogorov complexity.
Rather than this, I’m suggesting that natural language is not in line with complexity of the “minimum description length” sort. Human understanding in general is pretty good at it, actually—it’s good enough to intuit, with a little work, that gravity really is a simpler explanation than “intelligent falling, ” and that the world is simpler than solipsism that just happens to replicate the world. Although humans may consider verbal complexity “a good heuristic,” humans can still reason well about complexity even when the heuristic doesn’t apply.
Your description of Occam/Ockham’s razor is wrong - “entities must not be multiplied beyond necessity” is one common statement. This would give equal chances to both storms and sea monsters (barring, e.g. the separate observation of storms and the lack of observation of sea monsters), though it gives a greater chance to sea monsters than green scaly sea monsters.
Modern science uses a few variations on Occam’s razor that add the requirement that you don’t pull any information out of thin air, mostly captured by the Einstein quote “Make everything as simple as possible, but not simpler.”
And here at LW we often use a quantitative measurement of simplicity called Kolmogorov complexity, which is how complicated a computer has to be before it can output your hypothesis. Not in natural language, but in terms of actual properties.
The reason it makes sense to act as if natural language is how we should describe things is because when natural language reflects things we’ve already seen, it’s simpler (in terms of properties) to make hypotheses about the whole universe that reuse parts, rather than hypotheses that have lots of new parts all the time—each of your mini-hypotheses is really part of the bigger hypothesis “what is the universe like?”
But since the correspondence between natural language and “stuff we’ve already seen” isn’t perfect, this breaks down in places. For example, in natural language, the hypothesis “god did it” is almost unsurpassed in simplicity. The fossil record, rainbows, why light things fall as fast as heavy things in a vacuum. The reason Occam’s razor does not suggest that “god did it” is the best explanation for everything is because god is a very complicated concept despite being a short word. So when you use something like Kolmogorov complexity that measures the size of concepts rather than the number of letters, you get evolution, diffraction, and gravity.
It’s more because “it” is a very complicated concept.
Kolmogorov complexity is not used at LessWrong; it is not used anywhere because it is uncomputable. Approximations of Kolmogorov complexity (replacing the Turing machine in the definition with something weaker) do not have the same nigh-magical properties that Kolmogorov complexity would have, if it were available.
Kolmogorov complexity is computable for some hypotheses, just not all (for each formal axiomatic system, there is an upper bound to the complexity of hypotheses that can have their complexity determined by the system). Anyways, while we can never use Kolmogorov complexity to analyze all hypotheses, I believe that Manfred merely meant that we use it as an object of study, rather than to implement full Solomonoff induction.
I am aware that my definition of Occam’s razor is not the “official” definition. However, it is the definition which I see used most often in discussions and arguments, which is why I chose it. The fact that this definition of Occam’s razor is common supports my claim that humans consider it a good heuristic.
Forgive me for my ignorance, as I have not studied Kolmogorov complexity in detail. As you suggest, it seems that human understanding of a “simple description” is not in line with Kolmogorov complexity.
I think the intention of my post may have been unclear. I am not trying to argue that natural language is a good way of measuring the complexity of statements. (I’m also not trying to argue that it’s bad.) My intention was merely to explore how humans understand the complexity of ideas, and to investigate how such judgements of complexity influence the way typical humans build models of the world.
The fact that human understanding of complexity is so far from Kolmogorov complexity indicates to me that if an AI were to model its environment using Kolmogorov complexity as a criterion for selecting models, the model it developed would be different from the models developed by typical humans. My concern is that this disparity in understanding of the world would make it difficult for most humans to communicate with the AI.
Rather than this, I’m suggesting that natural language is not in line with complexity of the “minimum description length” sort. Human understanding in general is pretty good at it, actually—it’s good enough to intuit, with a little work, that gravity really is a simpler explanation than “intelligent falling, ” and that the world is simpler than solipsism that just happens to replicate the world. Although humans may consider verbal complexity “a good heuristic,” humans can still reason well about complexity even when the heuristic doesn’t apply.