I had a kinda different take (copied from twitter)
This tweet is an interesting argument but I think a central flaw is ignoring how long the program has to run to produce the prediction.
For example, âAlphaZero-after-1-game-of-self-playâ and âAlphaZero-after-10²â°-games-of-self-playâ have essentially the same Kolmogorov complexity. After all, they have the exact same source code, apart from like 4 characters that specify the number of games to play. But thereâs a real sense in which the latter is better at Go than the former. Specifically, itâs better in the sense of âI donât want to sit around while it does 10²Ⱐgames of self-play, I want to play Go right now.â
Another way to think about this argument: Suppose AI_1 builds AI_2, and then AI_2 does X. Well, itâs also true that âAI_1 did Xââspecifically, âAI_1 did Xâ by building AI_2.
In a certain sense, this is true! But thatâs a pretty weird way to think about things! AI_2 does in fact exist here!
K-complexity asks us to forget about AI_2, by focusing the discussion on what happens given infinite time, as opposed to how it happens and how long it takes.
I think the core true point in atroynâs argument is: there is a-priori-unpredictable complexity in the world that canât be deduced from an armchair, but rather has to be observed, and making a âmore intelligent successorâ does not substitute for that.
If you flip a coin and donât tell me, then I donât know whether itâs heads or tails. And I also canât make a âmore intelligent successorâ that knows whether itâs heads or tails.
This is entirely true! But I claim people talking about recursive self-improvement are not making that mistake.
For example:
Thereâs an âoverhangâ of possible logical inferences that an AI could make on the basis of its existing knowledge, but doesnât (e.g. if I tell an AI the axioms of math, it doesnât instantaneously prove every possible theorem),
Thereâs an âoverhangâ of possible input data that an AI could download and scrutinize, but doesnât (e.g. as of this writing, I believe no AI has watched all 100,000 years of YouTube)
Thereâs an âoverhangâ of possible plans that an AI could execute but doesnât (e.g. an early AGI is unlikely to be simultaneously doing every remote job on the planet, while also starting a zillion new ambitious projects in parallel).
So an AI could self-improve in a way that allows it to go farther and faster on those metrics.
An obvious example is tweaking the assembly code to make the same AI run faster.
I also want to put self-replication into this category: going from âone instance of an AIâ to âa million instances of the same AI running in parallel and collaboratingâ (e.g. by buying or stealing additional compute). If you think about it, I claim that should totally count as âself-improvementâ, because after all one AI system is creating a more powerful AI âsystemâ. The latter âsystemâ is composed of many instances of the original AI, but so what? It should still count, IMO.
I think the OP here is also valid (and complementary).
Each of these points look valid, but thereâs a much simpler refutation: ÂŤ Any good enough intelligence is smart enough to distribute part of its cognition to external devices. Âť.
Application: either my code includes wikipedia and whoever might change wikipedia just before I consult it, or itâs Kolmogorov complexity does not fully capture my capabilities. In a sense, this is showing the impact of putting too much confidence on a debatable picture of our capabilities and limitations as a single agent working from some cockpit.
I had a kinda different take (copied from twitter)
Then after sleeping on it I tweeted again:
I think the OP here is also valid (and complementary).
Each of these points look valid, but thereâs a much simpler refutation: ÂŤ Any good enough intelligence is smart enough to distribute part of its cognition to external devices. Âť.
Application: either my code includes wikipedia and whoever might change wikipedia just before I consult it, or itâs Kolmogorov complexity does not fully capture my capabilities. In a sense, this is showing the impact of putting too much confidence on a debatable picture of our capabilities and limitations as a single agent working from some cockpit.