Seems to me that the usage of Kolmogorov complexity in this context is a red herring. Complexity of what: the program alone, or the program and the data it gets? The former is irrelevant, because the entire idea is that an intelligence at human level or higher can observe the environment and learn from it. The latter, assuming that we can make an unlimited number of observations and experiments, is potentially unlimited.
Mathematically speaking, a universal program (an interpreter that can simulate an arbitrary program described in its data) has a constant Kolmogorov complexity, and yet can simulate a program with arbitrarily high Kolmogorov complexity. (The extra complexity is in the data describing the simulated program.)
If we taboo “Kolmogorov complexity”, it seems to me that the argument reduces to: “a machine cannot self-improve, because it can only build the machines it could simulate, in which case what’s the point of actually building them?”. Which, in some sense yes (assuming unlimited computing power and time), but the machine that is actually built can hypothetically run much faster than the simulated one.
Seems to me that the usage of Kolmogorov complexity in this context is a red herring. Complexity of what: the program alone, or the program and the data it gets? The former is irrelevant, because the entire idea is that an intelligence at human level or higher can observe the environment and learn from it. The latter, assuming that we can make an unlimited number of observations and experiments, is potentially unlimited.
Mathematically speaking, a universal program (an interpreter that can simulate an arbitrary program described in its data) has a constant Kolmogorov complexity, and yet can simulate a program with arbitrarily high Kolmogorov complexity. (The extra complexity is in the data describing the simulated program.)
If we taboo “Kolmogorov complexity”, it seems to me that the argument reduces to: “a machine cannot self-improve, because it can only build the machines it could simulate, in which case what’s the point of actually building them?”. Which, in some sense yes (assuming unlimited computing power and time), but the machine that is actually built can hypothetically run much faster than the simulated one.