Li and Vitanyi’s bad scholarship
In An Introduction to Kolmogorov Complexity and Its Applications 4th edition Li and Vitanyi claim that Solomonoff Induction solves the problem of induction. In Section 5.1.4 they write:
The philosopher D. Hume (1711–1776) argued that true induction is impossible because we can reach conclusions only by using known data and methods. Therefore, the conclusion is logically already contained in the start configuration. Consequently, the only form of induction possible is deduction. Philosophers have tried to find a way out of this deterministic conundrum by appealing to probabilistic reasoning such as using Bayes’s rule. One problem with this is where the prior probability one uses has to come from. Unsatisfactory solutions have been proposed by philosophers such as R. Carnap (1891–1970) and K.R. Popper.
What Hume actually wrote about induction was (Section IV, Part II, pp. 16-17):
Should it be said that, from a number of uniform experiments, we infer a connexion between the sensible qualities and the secret powers; this, I must confess, seems the same difficulty, couched in different terms. The question still recurs, on what process of argument this inference is founded? Where is the medium, the interposing ideas, which join propositions so very wide of each other? It is confessed that the colour, consistence, and other sensible qualities of bread appear not, of themselves, to have any connexion with the secret powers of nourishment and support. For otherwise we could infer these secret powers from the first appearance of these sensible qualities, without the aid of experience; contrary to the sentiment of all philosophers, and contrary to plain matter of fact. Here, then, is our natural state of ignorance with regard to the powers and influence of all objects. How is this remedied by experience? It only shows us a number of uniform effects, resulting from certain objects, and teaches us that those particular objects, at that particular time, were endowed with such powers and forces. When a new object, endowed with similar sensible qualities, is produced, we expect similar powers and forces, and look for a like effect. From a body of like colour and consistence with bread we expect like nourishment and support. But this surely is a step or progress of the mind, which wants to be explained. When a man says, I have found, in all past instances, such sensible qualities conjoined with such secret powers: And when he says, Similar sensible qualities will always be conjoined with similar secret powers, he is not guilty of a tautology, nor are these propositions in any respect the same. You say that the one proposition is an inference from the other. But you must confess that the inference is not intuitive; neither is it demonstrative: Of what nature is it, then? To say it is experimental, is begging the question. For all inferences from experience suppose, as their foundation, that the future will resemble the past, and that similar powers will be conjoined with similar sensible qualities. If there be any suspicion that the course of nature may change, and that the past may be no rule for the future, all experience becomes useless, and can give rise to no inference or conclusion. It is impossible, therefore, that any arguments from experience can prove this resemblance of the past to the future; since all these arguments are founded on the supposition of that resemblance. Let the course of things be allowed hitherto ever so regular; that alone, without some new argument or inference, proves not that, for the future, it will continue so. In vain do you pretend to have learned the nature of bodies from your past experience. Their secret nature, and consequently all their effects and influence, may change, without any change in their sensible qualities. This happens sometimes, and with regard to some objects: Why may it not happen always, and with regard to all objects? What logic, what process or argument secures you against this supposition? My practice, you say, refutes my doubts. But you mistake the purport of my question. As an agent, I am quite satisfied in the point; but as a philosopher, who has some share of curiosity, I will not say scepticism, I want to learn the foundation of this inference. No reading, no enquiry has yet been able to remove my difficulty, or give me satisfaction in a matter of such importance. Can I do better than propose the difficulty to the public, even though, perhaps, I have small hopes of obtaining a solution? We shall at least, by this means, be sensible of our ignorance, if we do not augment our knowledge.
What Hume wrote isn’t that we can only use known data and methods. Rather, he said that no argument can prove that the future will resemble the past. So drawing conclusions about what will happen in the future from past data is illogical. He didn’t say that the only possible form of induction is deduction.
In addition, the future always resembles the past in some respects and not others, so saying the future resembles the past is irrelevant to creating and assessing ideas.
Popper wasn’t trying to solve the problem of how to make Bayesian induction work. He claimed that induction was impossible, not that he had a way of making it work by finding the right prior (Realism and the Aim of Science, Chapter I, Section 3, I):
It seems that almost everybody believes in induction; believes, that is, that we learn by the repetition of observations. Even Hume, in spite of his great discovery that a natural law can neither be established nor made ‘probable’ by induction, continued to believe firmly that animals and men do learn through repetition: through repeated observations as well as through the formation of habits, or the strengthening of habits, by repetition. And he upheld the theory that induction, though rationally indefensible and resulting in nothing better than unreasoned belief, was nevertheless reliable in the main—more reliable and useful at any rate than reason and the processes of reasoning; and that ‘experience’ was thus the unreasoned result of a (more or less passive) accumulation of observations.
As against all this, I happen to believe that in fact we never draw inductive inferences, or make use of what are now called ‘inductive procedures’. Rather, we always discover regularities by the essentially different method of trial and error, of conjecture and refutation, or of learning from our mistakes; a method which makes the discovery of regularities much more interesting than Hume thought.
Li and Vitanyi want us to think they can solve the problem of induction, but they can’t even summarise the arguments against their position accurately.
Well, yes, it’s not a perfect summary. I have no idea why they’d say Popper was working on Bayesianism—unless maybe “the problem” in that clause was the problem of induction, and something got lost in an edit.
But sometimes nitpicks aren’t that important. Like, for example, it’s spelled Vitanyi. But this isn’t really a crushing refutation of your post (though it is a very convenient illustration). You shouldn’t sweat this too much, because their textbook really is worth reading about algorithmic information theory.
You seem to be making an argument here against statements made in a particular book, and provide a lot of quotes, but not quotes of the specific statements you are arguing against. You claim they say Solomonoff induction solves the problem of induction, which it clearly doesn’t in full generality because that would also mean it solves the problem of the criterion, epistemic circularity, and other formulations of what we might call the hard problem of epistemology (how do you go from knowing nothing to knowing something) in a justified way, yet on most accounts Solomonoff induction is usually argued to formalize and address induction up to the limit of systematization, which seems more relevant to the rest of what you get at in your post.
So, uh, I guess just what are you trying to argue here, other than that you think these coauthors made a mistake because you don’t think they engaged with the literature on induction enough in the text of their book?
Doesn’t Solomonoff induction at least make a step towards resolving epistemic circularity, since Solomonoff prior dominates (I don’t remember in what way exactly) every probability distribution with the same or smaller support?
That’s not an entirely bad thing. Addressing a specific text is better than addressing an imaginary statement of your opponents position, a straw man. But it can still amount to weakmanning, which I think is your complaint.
I have corrected the spelling of Vitanyi.
But not all inductivists believe in a version of inductivism that supposedly generates theories or scientific knowledge.
Its also possible to accept a version of induction that deals purely with the probabilities of future observations based on past observations.
(Here, both claims are popular… but not equivalent It is possible to reject the idea that Solomonoff inductors are actually generating theories, whilst accepting a probabilistic basis for induction in the second sense. As I keep pointing out, inductors of that kind can be shown to exist by construction).
You could have credited inductivists with having detailed ideas about the distinction.
That version of inductivism isn’t in Li and Vitanyi who haven’t even stated the problem described by critics of inductivism. Where is it?
“Bacon’s method is an example of the application of inductive reasoning. However, Bacon’s method of induction is much more complex than the essential inductive process of making generalizations from observations. Bacon’s method begins with description of the requirements for making the careful, systematic observations necessary to produce quality facts. He then proceeds to use induction, the ability to generalize from a set of facts to one or more axioms. ” WP.
But what is the point? Not many people are Baconians nowadays.
You said earlier:
What is the version of inductivism that generates no theories or scientific knowledge and what does it accomplish?
There are many kinds of knowledge and learning that are useful but fall short of scientific knowledge. It is useful to any organism to learn from experience, and many can, even simple ones. There are many useful things learning algorithms can do. My cellphone has predictive text, which is based on learning: yours probably does too.
So how does a poor summary of Hume = a refutation Solomonoff’s induction?
Could I say something like, Alanf wants us to think he refuted a book but he can’t even spell the author’s name right...
Ok, but what does this have to do with the capital of Italy?