Author: Steven T. Piantadosi, ( University of Rochester)
Abstract:
We construct an elementary equation with a single real valued parameter that is capable of fitting any “scatter plot” on any number of points to within a fixed precision. Specifically, given given a fixed > 0, we may construct fθ so that for any collection of ordered pairs {(xj , yj )} n j=0 with n, xj ∈ N and yj ∈ (0, 1), there exists a θ ∈ [0, 1] giving |fθ(xj ) − yj | < for all j simultaneously. To achieve this, we apply prior results about the logistic map, an iterated map in dynamical systems theory that can be solved exactly. The existence of an equation fθ with this property highlights that “parameter counting” fails as a measure of model complexity when the class of models under consideration is only slightly broad.
After highlighting the two examples in the paper, Tabarrok provocatively writes:
Aside from the wonderment at the result, the paper also tells us that Occam’s Razor is wrong. Overfitting is possible with just one parameter and so models with fewer parameters are not necessarily preferable even if they fit the data as well or better than models with more parameters.
Occam’s Razor in its narrow form—the insight that simplicity is renders a claim more probable—is a consequence of the interaction between Kolmogorov complexity and Bayes’ theorem. I don’t see how this result affects this idea per se. But perhaps it shows the flaws of conceptualizing complexity as “number of parameters.”
Thanks to Alex Tabarrok at Marginal Revolution: https://marginalrevolution.com/marginalrevolution/2018/05/one-parameter-equation-can-exactly-fit-scatter-plot.html
Title: “One parameter is always enough”
Author: Steven T. Piantadosi, ( University of Rochester)
Abstract:
After highlighting the two examples in the paper, Tabarrok provocatively writes:
Occam’s Razor in its narrow form—the insight that simplicity is renders a claim more probable—is a consequence of the interaction between Kolmogorov complexity and Bayes’ theorem. I don’t see how this result affects this idea per se. But perhaps it shows the flaws of conceptualizing complexity as “number of parameters.”