Occam’s razor is about using the past to predict the future

Occam’s razor is both intuitive and counter-intuitive. It seems obvious that a simpler explanation is probably better; but it’s not clear why simplicity is raised to such a level of philosophical importance.

My take is that, if you neglect simplicity, you will fail to use the past to predict the future (or to predict new data).

Example: angle of the sun

Traditionally, when you do want to predict the future, you look at past evidence you have, hypothesise, and test it on the past data.

For example, these are the latitudes of various cities and the angle of the sun above the horizon at midday on the 20th of March:

Ankara (Lat 40°N): 50°
Brisbane (Lat 27°S): 63°
Charlesbourg (Lat 47°N): 43°
...
Xian (Lat 34°N): 56°
Yongin (Lat 37°N): 53°
Zapopan (Lat 21°N): 69°

We want to predict the angle of the sun at Anchorage (61°N). Let me write a hypothesis:

H: “The solar angle of Ankara is 50°. The solar angle at Brisbane is 63°… The solar angle of Zapopan is 69°. Oh, and by the way, the solar angle for Anchorage is 90° - perfectly vertical.”

Let’s test H on the past data! We start with Ankara… that fits. Brisbane also confirmed… All the way to Zapopan. Ok, H is perfectly confirmed! Thus we know the solar angle for Anchorage is 90°.

List hypotheses

This is obviously complete nonsense. If I can write a “list hypothesis” that lists the past data and includes an arbitrary future prediction, then it’s a useless theory, no matter how much it fits the past data. Note that these “list hypotheses” will often fit the data even better than the true theory (which is that the angle of the sun at the equinox and the latitude add up to 90°) because they don’t have issues with measurement error and other imperfections: they just list the actual past measurements. So while the true theory loses probability with every measurement, list hypotheses do not. If there’s about a 50% chance of getting the true measurement exactly right, then a true theory will have odds of worse than against it after 100 measurements. The list hypothesis will not suffer at all.

Implicit information leakage

Of course, nobody would take the above H seriously. But it’s very easy to leak explicit information from the past observations into your current theory, without realising you’re doing so. A few decisions on a few conditional clauses in the model, a choice of one variable over another, a different threshold choice… All these can make an H that really overfits the past data, without it being obvious to the theory designer or an observer.

So that’s what Occam’s razor does: it makes you pay for the extraneous information you leak into your theory; and it makes you pay more for leaking more. Information that artificially increases the plausibility of your theory without actually improving your prediction. And lack of space for longer theories forces you to fit the future-predicting part of your theory into the past-confirming part.

And note that the fact that list theories avoided paying a probability penalty for measurement errors penalises it in terms of Occam’s razor. It avoided paying the measurement error-penalty by exactly listing the final measurements – by adding a lot of detailed information. Including the complexity of that information compensates almost^[1] perfectly for its probability bonus.

This is key to the Kolmogorov Complexity/Minimum Description Length formalism for Occam’s razor: you can easily increase the fit with past data, but only by paying a price in the prior probability.

Simplicity vs simplifiability

Which is simpler, Newtonian Gravity or General Relativity? Well, it depends on how you formally count simplicity^[2]. Certainly, looking at textbook sizes, you can make the case that, in our language, General Relativity is much, much, much more complicated that Newtonian Gravity. Maybe even enough that it would take a truly astronomical (ha!) amount of confirmation for General Relativity to beat Newtonian Gravity.

Occam’s razor is mis-slicing here. The absolute measure of complexity is not what’s relevant. It’s that neither theory is simplifiable: they cannot be easily made simpler from what they are (at least not without falling apart completely). If they aren’t simplifiable, then there is very little leakage of extraneous information.

Easy way to see this: take any theory, add extraneous information. It now becomes simplifiable by removing the extraneous information.

Note also that a clause like “but in the future, things are different” can be very short but makes the theory predictively useless. These sort of statements are the absolute priority for simplifying away. A longer theory without such clauses is in a much better state that a shorter one with it; non-simplifiable can be more important than simple.

However, while simplicity can be measured by things like algorithmic complexity, simplifiability is much harder to formally define. And, even if the theory appears non-simplifiable, maybe there is subtle information leakage in the design choices. Thus the razor in its formal form is still useful.

When the razor is less used

Of course, once you’ve made the hypothesis, you can test it, on future data or held-out past data (i.e. past data that was gathered but hidden from you while you designed your theory). So if you iterate rapidly, you don’t have to stick to the full discipline of the razor to produce the absolutely simplest or non-simplifiable theory, because you can swiftly detect errors (thus ML can progress rapidly with large models and hyper-parameter tuning^[3]).

But it still helps to get as simple theory you can, to avoid seeing nothing but error; a simpler theory is more likely to avoid information leakages and successfully predict the held-out final validation set (thus ML uses regularisation to simplify the weights of its large models—and ensure a form of non-simplifiability).

^
Only almost perfectly because the choice of language for the theories can change this cost by a bounded amount.
^
Your choice of language/prefix-free coding.
^
Note that ML has spent a lot of time designing the structures, properties, and training procedures of their models so that they are less likely to generate errors—a kind of pre-testing across multiple possible designs. So the large ML models are already somewhat pre-optimised for predictive success.