The concept seems to me to involve higher levels of abstraction than natural selection. And note that even in natural selection, two concepts—“variation” and “differential survival” are indeed obvious, but the third—“heritability”—relies on lots of complex machinery, and we still have no good explanation how it came about in the first place.
To get the “intelligence explosion” idea, you need to assemble “goal-seeking behaviour” and “cross-domain generalization”, add “universality of computation”, “substrate independence” and “recursive self-improvement” into the mix, and these are complex ideas to start with.
The non-obvious part to me is “recursive self-improvement”. If I imagine myself looking at my own source code, and trying to figure out which part of it is responsible for how many dual N-back steps I can cope with, generalizing (admittedly from the limited perspective of having used a merely human brain) suggests that the search might prove fruitless - there may not be a single “line of code” where this limitation is encoded as a constant.
The concept seems to me to involve higher levels of abstraction than natural selection. And note that even in natural selection, two concepts—“variation” and “differential survival” are indeed obvious, but the third—“heritability”—relies on lots of complex machinery, and we still have no good explanation how it came about in the first place.
To get the “intelligence explosion” idea, you need to assemble “goal-seeking behaviour” and “cross-domain generalization”, add “universality of computation”, “substrate independence” and “recursive self-improvement” into the mix, and these are complex ideas to start with.
The non-obvious part to me is “recursive self-improvement”. If I imagine myself looking at my own source code, and trying to figure out which part of it is responsible for how many dual N-back steps I can cope with, generalizing (admittedly from the limited perspective of having used a merely human brain) suggests that the search might prove fruitless - there may not be a single “line of code” where this limitation is encoded as a constant.