What you are talking about in terms of Solmonoff induction is usually called algorithmic information theory and the shortest-program-to-produce-a-bit-string is usually called Kolmogorov-Chaitin information. I am sure you know this. Which begs the question, why didn’t you mention this? I agree, it is the neatest way to think about Occam’s razor. I am not sure why some are raising PAC theory and VC-dimension. I don’t quite see how they illuminate Occam. Minimalist inductive learning is hardly the simplest “explanation” in the Occam sense, and is actually closer to Shannon entropy in spirit, in being more of a raw measure. Gregory Chaitin’s ‘Meta Math: The Search for Omega’, which I did a review summary of is a pretty neat look at this stuff.
What you are talking about in terms of Solmonoff induction is usually called algorithmic information theory and the shortest-program-to-produce-a-bit-string is usually called Kolmogorov-Chaitin information. I am sure you know this. Which begs the question, why didn’t you mention this? I agree, it is the neatest way to think about Occam’s razor. I am not sure why some are raising PAC theory and VC-dimension. I don’t quite see how they illuminate Occam. Minimalist inductive learning is hardly the simplest “explanation” in the Occam sense, and is actually closer to Shannon entropy in spirit, in being more of a raw measure. Gregory Chaitin’s ‘Meta Math: The Search for Omega’, which I did a review summary of is a pretty neat look at this stuff.