This is something I’ve thought about recently—a full answer would take too long to write, but I’ll leave a couple comments.
First, what this implies about learning algorithms can be summarized as “it explains the manifold hypothesis.” The Telephone Theorem creates an information bottleneck that limits how much information can be captured at a distance. This means that a 64x64 RGB image, despite being nominally 12288-dimensional, in reality captures far less information and lies on a much lower-dimensional latent space. Chaos has irreversibly dispersed all the information about the microscopic details of your object. “Free lunch” follows quite easily from this, since the set of functions you care about is not really the set of functions on all RGB images, but the set of functions on a much smaller latent space.
Second, the vanilla Telephone Theorem isn’t quite sufficient—the only information that persists in the infinite-time limit is conserved quantities (e.g. energy), which isn’t very interesting. You need to restrict to some finite time (which is sufficiently longer than your microscopic dynamics) instead. In this case, persistent information now includes “conditionally conserved” quantities, such as the color of a solid object (caused by spontaneous symmetry-breaking reducing the permanently-valid Lorentz symmetry to the temporarily-valid space group symmetry). I believe the right direction to go here is ergodic theory and KAM theory, although the details are fuzzy to me.
This is something I’ve thought about recently—a full answer would take too long to write, but I’ll leave a couple comments.
First, what this implies about learning algorithms can be summarized as “it explains the manifold hypothesis.” The Telephone Theorem creates an information bottleneck that limits how much information can be captured at a distance. This means that a 64x64 RGB image, despite being nominally 12288-dimensional, in reality captures far less information and lies on a much lower-dimensional latent space. Chaos has irreversibly dispersed all the information about the microscopic details of your object. “Free lunch” follows quite easily from this, since the set of functions you care about is not really the set of functions on all RGB images, but the set of functions on a much smaller latent space.
Second, the vanilla Telephone Theorem isn’t quite sufficient—the only information that persists in the infinite-time limit is conserved quantities (e.g. energy), which isn’t very interesting. You need to restrict to some finite time (which is sufficiently longer than your microscopic dynamics) instead. In this case, persistent information now includes “conditionally conserved” quantities, such as the color of a solid object (caused by spontaneous symmetry-breaking reducing the permanently-valid Lorentz symmetry to the temporarily-valid space group symmetry). I believe the right direction to go here is ergodic theory and KAM theory, although the details are fuzzy to me.