About the notion of “mildly” superintelligent AI. How about the following typology of ASI:
AI that can find paths through reality which no human could have come up with, but could still understand (akin to verifying solutions to NP-hard problems)
AI that can find paths through reality which no human could come up with and which remain incomprehensible to humans even in retrospect, possibly because it would involve manipulating concepts in a way that doesn’t work with our neurological architecture.
Somewhere in between: paths through reality that seem comprehensible in principle, but are just too insanely complicated, because they consists of too many parts working together.
And I don’t mean in the sense of “this plan requires 1000 individual steps to succeed”, because such plan is almost guaranteed to fail, even if each step has a 99% chance of success. But more like “this plan has 1000000 individual steps, many of them are parallel ways to achieve the same thing (so only one of them needs to succeed), and actually quite a few steps have probability below 10%, it’s just that when the AI checks the entire graph and calculates the overall probability of success, it reports 99.99% chance”. A complicated network that cannot be easily factorized. Each step is comprehensible and relatively easy. The overall structure is incomprehensible.
A complicated network that cannot be easily factorized. Each step is comprehensible and relatively easy. The overall structure is incomprehensible.
Do you think that today’s neural networks are already in this category, insofar as one could in principle do the matrix multiplications for e.g. next-word prediction by hand without having any idea what it means?
Yes, that’s exactly what I meant. Are today’s networks “comprehensible”? If you ask whether humans are able to understand matrix multiplication, yes they are. But effectively, they are not.
I am not saying that the plans of superhuman AIs will be like this, but they could have a similar quality. Millions of pieces, individually easy to understand, the entire system too complicated to reason about, somehow achieving the intended outcome.
This is very interesting because the neural networks are not the product of AI as traditionally conceptualized. Incomprehensibly complex networks produced by repeatedly applying a comprehensible algorithm over a large surface area for an absurdly long time.
Reminds me of evolution producing genomes which allow a single cell to grow into a human. In this frame, our knowledge of cellular biology and individual genes is something like mechanistic interpretability research probing aspects of the underlying logic.
I’m not sure what to make of this with regard to ASI typology.
If comprehensible things become too large, in a way that cannot be factorized, they become incomprehensible. But at the boundary, increasing the complexity by +1 can mean that a more intelligent (and experienced) human could understand it, and a less intelligent one would not. So there is no exact line, it just requires more intellect the further you go.
Maybe an average nerd could visualize a 3x3 matrix multiplication, a specialized scientist could visualize 5x5 (I am just saying random numbers here), and… a superintelligence could visualize 100x100 or maybe even 1000000x1000000.
And similarly, a stupid person could make a plan “first this, then this”, a smart person could make a plan with a few alternatives ”...if it rains, we will go to this café; and if it’s closed, we will go to this gallery instead...”, and a superintelligence could make a plan with a vast network of alternatives.
And yes, just like with biology, a human can understand one simple protein maybe (again, I am just guessing here, what I mean is “there is a level of complexity that a human understands”), and a superintelligence could similarly understand the entire organism.
In each case, there is no clear line between comprehensibility and incomprehensibility, it just becomes intractable when it is too large.
Yet if we extend the “+1 complexity” argument, we eventually reach a boundary where no human, however smart, could understand it. In principle nature could produce a human with the specific mutation necessary to apprehend it, which pushes the human cognitive horizon by some amount without actually eliminating it.
To the extent that AI can be scaled unlike the human brain, it might be able to form conceptual primitives which are so far outside the human cognitive horizon that biology is unlikely to produce a human intelligent enough to apprehend them on any reasonable timescale.
About the notion of “mildly” superintelligent AI. How about the following typology of ASI:
AI that can find paths through reality which no human could have come up with, but could still understand (akin to verifying solutions to NP-hard problems)
AI that can find paths through reality which no human could come up with and which remain incomprehensible to humans even in retrospect, possibly because it would involve manipulating concepts in a way that doesn’t work with our neurological architecture.
Somewhere in between: paths through reality that seem comprehensible in principle, but are just too insanely complicated, because they consists of too many parts working together.
And I don’t mean in the sense of “this plan requires 1000 individual steps to succeed”, because such plan is almost guaranteed to fail, even if each step has a 99% chance of success. But more like “this plan has 1000000 individual steps, many of them are parallel ways to achieve the same thing (so only one of them needs to succeed), and actually quite a few steps have probability below 10%, it’s just that when the AI checks the entire graph and calculates the overall probability of success, it reports 99.99% chance”. A complicated network that cannot be easily factorized. Each step is comprehensible and relatively easy. The overall structure is incomprehensible.
Do you think that today’s neural networks are already in this category, insofar as one could in principle do the matrix multiplications for e.g. next-word prediction by hand without having any idea what it means?
Yes, that’s exactly what I meant. Are today’s networks “comprehensible”? If you ask whether humans are able to understand matrix multiplication, yes they are. But effectively, they are not.
I am not saying that the plans of superhuman AIs will be like this, but they could have a similar quality. Millions of pieces, individually easy to understand, the entire system too complicated to reason about, somehow achieving the intended outcome.
This is very interesting because the neural networks are not the product of AI as traditionally conceptualized. Incomprehensibly complex networks produced by repeatedly applying a comprehensible algorithm over a large surface area for an absurdly long time.
Reminds me of evolution producing genomes which allow a single cell to grow into a human. In this frame, our knowledge of cellular biology and individual genes is something like mechanistic interpretability research probing aspects of the underlying logic.
I’m not sure what to make of this with regard to ASI typology.
If comprehensible things become too large, in a way that cannot be factorized, they become incomprehensible. But at the boundary, increasing the complexity by +1 can mean that a more intelligent (and experienced) human could understand it, and a less intelligent one would not. So there is no exact line, it just requires more intellect the further you go.
Maybe an average nerd could visualize a 3x3 matrix multiplication, a specialized scientist could visualize 5x5 (I am just saying random numbers here), and… a superintelligence could visualize 100x100 or maybe even 1000000x1000000.
And similarly, a stupid person could make a plan “first this, then this”, a smart person could make a plan with a few alternatives ”...if it rains, we will go to this café; and if it’s closed, we will go to this gallery instead...”, and a superintelligence could make a plan with a vast network of alternatives.
And yes, just like with biology, a human can understand one simple protein maybe (again, I am just guessing here, what I mean is “there is a level of complexity that a human understands”), and a superintelligence could similarly understand the entire organism.
In each case, there is no clear line between comprehensibility and incomprehensibility, it just becomes intractable when it is too large.
Yet if we extend the “+1 complexity” argument, we eventually reach a boundary where no human, however smart, could understand it. In principle nature could produce a human with the specific mutation necessary to apprehend it, which pushes the human cognitive horizon by some amount without actually eliminating it.
To the extent that AI can be scaled unlike the human brain, it might be able to form conceptual primitives which are so far outside the human cognitive horizon that biology is unlikely to produce a human intelligent enough to apprehend them on any reasonable timescale.