They both fit the same basic evidence: burning a candle or similar object in a small enclosure made it go out. Similar remarks applied to small living animals and combinations of candles and animals. Moreover, many forms of combustion visibly gave off something into the air. Indeed the theory “combustion occurs when something from the substance goes into the air” is simpler than “combustion occurs when something from the air combines with the substance and sometimes but not always something else is added back into the air.” It was only with careful measurements of the mass of objects before and after reactions (weighing gasses is really tough!), combined with the observation that some metals gained weight when being burned that really created a problem. A good Bayesian in 1680 who heard of both ideas arguably should favor phlogiston.
It’s not a post about how things usually go. It’s a post about the minimum requirements to know something with near certainty for an intelligent agent.
It is possible that I’m reading too much into this but it does seem that Eliezer is using Einstein’s success as an actual example of his argument about how brains should work. But there’s a problem: if brains are less than perfect Bayesians (and it seems that minds that are possible in this part of the Tegmark ensemble fit in that collection) they won’t bring one hypothesis to the front, they will often have a fair number of hypotheses to explain
based on incomplete data. In some cases, like Einstein, the sheer mathematical simplicity (in his case hitting on the simplest hypothesis that hit a large set of nice conditions (not to say that it is at all easy. Far from it.)) will make one hypothesis look like under some framework it takes less data than the others. But often the actual process will be that they need more data.
A mind, when investigating things will likely not just keep getting more and more clever insights. Things take effort. Let’s say you have a really smart strongly-Bayesian mind with the control of the resources of a planet, but with minimal prior knowledge. So it can likely figure some things out pretty quickly like the orbits of the planets, and some other stuff. But somewhere between that and trying to detect fundamental particles of the universe it will probably need to collect more data. The mind isn’t going to have any way to detect that neutrinos have mass (even if it suspects that) until it sees evidence that they oscillate. Etc. I suspect that no mind from the simple data that humans have from our naive senses will deduce the existence of quarks.
In physics, if they’ve truly narrowed it down like that, the conclusion is that they ought not need more evidence, not that the social forces of science will deterministically overturn every confusion dividing a professional field.
So this seems like a more valid point: There are problems of human cognitive biases that go in the other direction (that is making theories to overfit our data and our general tendency to be overconfident in our beliefs), But an actual good Bayesian should not need to specially test a hypothesis once the pre-existing evidence has singled it out as extremely likely. This feeling that we need to do this is an artifact of having to deal with the problems of human cognition and social biases.
If that’s what Eliezer meant I don’t think he said it very well.
Indeed the theory “combustion occurs when something from the substance goes into the air” is simpler...
Seems like a simpler theory. Is a shorter sentence.
The mind isn’t going to have any way to detect that neutrinos have mass (even if it suspects that) until it sees evidence that they oscillate. Etc.
Sure, knowledge increases far more than arithmetically with additions of either smarts or data.
But an actual good Bayesian should not need to specially test a hypothesis once the pre-existing evidence has singled it out as extremely likely.
Only if he or she was sub-optimal when gathering data is this true. If, when doddering about, you narrowed down to a hypothesis, smarten up and you could probably determine if it’s true. If you were doing your best, someone smarter in the relevant way probably can. A Bayesian of any quality must improve beyond the level of smarts that led him or her to merely single out a hypothesis to judge it well.
they won’t bring one hypothesis to the front, they will often have a fair number of hypotheses to explain based on incomplete data.
I think this feels like having no idea at all, with no conscious hypothesis.
in his case hitting on the simplest hypothesis that hit a large set of nice conditions
Doing this feels like reasoning, or concentrating, or even zoning out, I suspect. It is varyingly subconscious pruning a tree of hypotheses, not conscious searching through them one by one and getting lucky by stumbling on a good one.
But often the actual process will be that they need more data.
Usually not once one consciously notices a finite set of hypotheses. Data can substitute for better thinking, there’s no law against that, but it’s not a “need” in many senses of the term.
So this seems like a more valid point: There are problems of human cognitive biases that go in the other direction
The point is that human minds just aren’t efficient processors, biases aside. “The” point (there are multiple ones) isn’t about overcoming bias, it’s that a well designed AI brain would need computing resources less than those of our brains to be smarter than any human.
Seems like a simpler theory. Is a shorter sentence
Yes, simplicity of English language is not at all a good metric of actual simplicity for a decent prior. However, in this particular case, both account for the same qualitative observations, and I strongly suspect that if one did try to make these into some formal system one would find that the second hypothesis is actually more complicated since it has a conjunction.
I need to think more about the rest of your remarks more before responding. I think I agree with most of them.
(And right now I’m really tempted to pretend to be an internet crank and start going around the internet preaching that phlogiston is correct).
I don’t think I agree. To be equivalent the summary of the phlogiston hypothesis would also have to include that air has a definite, limited capacity for burned phlogiston and no other known substance does, nor does vacuum.
They both fit the same basic evidence: burning a candle or similar object in a small enclosure made it go out. Similar remarks applied to small living animals and combinations of candles and animals. Moreover, many forms of combustion visibly gave off something into the air. Indeed the theory “combustion occurs when something from the substance goes into the air” is simpler than “combustion occurs when something from the air combines with the substance and sometimes but not always something else is added back into the air.” It was only with careful measurements of the mass of objects before and after reactions (weighing gasses is really tough!), combined with the observation that some metals gained weight when being burned that really created a problem. A good Bayesian in 1680 who heard of both ideas arguably should favor phlogiston.
It is possible that I’m reading too much into this but it does seem that Eliezer is using Einstein’s success as an actual example of his argument about how brains should work. But there’s a problem: if brains are less than perfect Bayesians (and it seems that minds that are possible in this part of the Tegmark ensemble fit in that collection) they won’t bring one hypothesis to the front, they will often have a fair number of hypotheses to explain based on incomplete data. In some cases, like Einstein, the sheer mathematical simplicity (in his case hitting on the simplest hypothesis that hit a large set of nice conditions (not to say that it is at all easy. Far from it.)) will make one hypothesis look like under some framework it takes less data than the others. But often the actual process will be that they need more data.
A mind, when investigating things will likely not just keep getting more and more clever insights. Things take effort. Let’s say you have a really smart strongly-Bayesian mind with the control of the resources of a planet, but with minimal prior knowledge. So it can likely figure some things out pretty quickly like the orbits of the planets, and some other stuff. But somewhere between that and trying to detect fundamental particles of the universe it will probably need to collect more data. The mind isn’t going to have any way to detect that neutrinos have mass (even if it suspects that) until it sees evidence that they oscillate. Etc. I suspect that no mind from the simple data that humans have from our naive senses will deduce the existence of quarks.
So this seems like a more valid point: There are problems of human cognitive biases that go in the other direction (that is making theories to overfit our data and our general tendency to be overconfident in our beliefs), But an actual good Bayesian should not need to specially test a hypothesis once the pre-existing evidence has singled it out as extremely likely. This feeling that we need to do this is an artifact of having to deal with the problems of human cognition and social biases.
If that’s what Eliezer meant I don’t think he said it very well.
Seems like a simpler theory. Is a shorter sentence.
Sure, knowledge increases far more than arithmetically with additions of either smarts or data.
Only if he or she was sub-optimal when gathering data is this true. If, when doddering about, you narrowed down to a hypothesis, smarten up and you could probably determine if it’s true. If you were doing your best, someone smarter in the relevant way probably can. A Bayesian of any quality must improve beyond the level of smarts that led him or her to merely single out a hypothesis to judge it well.
I think this feels like having no idea at all, with no conscious hypothesis.
Doing this feels like reasoning, or concentrating, or even zoning out, I suspect. It is varyingly subconscious pruning a tree of hypotheses, not conscious searching through them one by one and getting lucky by stumbling on a good one.
Usually not once one consciously notices a finite set of hypotheses. Data can substitute for better thinking, there’s no law against that, but it’s not a “need” in many senses of the term.
The point is that human minds just aren’t efficient processors, biases aside. “The” point (there are multiple ones) isn’t about overcoming bias, it’s that a well designed AI brain would need computing resources less than those of our brains to be smarter than any human.
And they wouldn’t be limited to that.
Yes, simplicity of English language is not at all a good metric of actual simplicity for a decent prior. However, in this particular case, both account for the same qualitative observations, and I strongly suspect that if one did try to make these into some formal system one would find that the second hypothesis is actually more complicated since it has a conjunction.
I need to think more about the rest of your remarks more before responding. I think I agree with most of them.
(And right now I’m really tempted to pretend to be an internet crank and start going around the internet preaching that phlogiston is correct).
I don’t think I agree. To be equivalent the summary of the phlogiston hypothesis would also have to include that air has a definite, limited capacity for burned phlogiston and no other known substance does, nor does vacuum.