It seems to me that this argument (and arguments which rely on unspecified methods and the like) boils down to breaking the world model to add things with unclear creation history and unclear decomposition into components, and resulting non-reductionist magic infested mental world model misbehaving. Just as it always did in the human history, yielding gods and the like.
You postulate that unspecific magic can create superhuman intelligence—it arises without mental model of necessary work, problems being solved, returns diminishing, and available optimizations being exhausted—is it a surprise that in this broken mental model (broken because we don’t know how the AI would be built), because the work is absent, the superhuman intelligence in question creates a greater still intelligence in days, merely continuing the trend of it’s unspecific creation? If it’s not at all surprising then it’s not informative that mental model goes in this direction.
It seems to me that this argument (and arguments which rely on unspecified methods and the like) boils down to breaking the world model to add things with unclear creation history and unclear decomposition into components, and resulting non-reductionist magic infested mental world model misbehaving. Just as it always did in the human history, yielding gods and the like.
You postulate that unspecific magic can create superhuman intelligence—it arises without mental model of necessary work, problems being solved, returns diminishing, and available optimizations being exhausted—is it a surprise that in this broken mental model (broken because we don’t know how the AI would be built), because the work is absent, the superhuman intelligence in question creates a greater still intelligence in days, merely continuing the trend of it’s unspecific creation? If it’s not at all surprising then it’s not informative that mental model goes in this direction.