All notations and formalisms used in this post are outdated and strictly superseded by the notations and formalisms used in “Towards a Formalisation of Returns on Cognitive Reinvestment (Part 1)”. I will come back one day and unify the notation, but I’m not sure how to do that without making this post too unwieldy.
Introduction
Consider that intelligence may be hard (let’s call this the “hard intelligence hypothesis” or HIH). I suspect that it is, I think there’s a lot of empirical evidence for it.
However, for the purpose of this post. I don’t want to explore the evidence, intuitions or arguments underpinning a belief in HIH. I would simply like to speculate on how HIH might bear on recursive self-improvement, and particularly on foom arising from recursive self-improvement.
I think knowing better the “hardness” of intelligence would better inform our position on takeoff dynamics.
A Pre Formal Definition of HIH
My current working definition of HIH is something like:
Returns on intelligence scale sublinearly with resources expended to increase intelligence.
Intelligence is itself just another resource that can be expended to increase intelligence. That particular case is what I will be considering here.
The “hardness” of intelligence with respect to a given resource can be thought of as how sublinear the curve is (perhaps what complexity class it belongs to?).
What is Recursive Self Improvement?
The basic story of recursive self improvement is that above a given level of intelligence (e.g. “par human”), the AI will be able to inspect and modify its own architecture/algorithms, to directly become more intelligent or design such a successor.
It doesn’t particularly matter whether the system directly self modifies or creates a successor. Self-modification can be understood as a particular case of creating a successor. So for the rest of this post, I’ll look only at the case of successors.
Interlude
Unless otherwise specified, all variables that appear in LaTeX should be understood to be positive real numbers.
A Semi-Formal Description of Succession
Consider an agent Ai, let the successor agent that Ai creates be denoted Ai+1.
Let A0 be the first agent intelligent enough to design successor agents.
Posit also that we have a linear, additive measure of “intelligence”. We’ll denote this measure by INT. I’ll denote an agent Ax with vINT as such: Ax[v]
To explain what I mean by “linear additive measure”, suppose we have 2 tuples of agents: (Ai[z+y]],Ak[z])&(Aj[k+y],Am[k]). Then intra-pair difference in capability is the same.
For convenience’s sake, let 100INT be the intelligence of A0.
So the story of recursive self-improvement is that A0 instantiates a successor A1.
A1 is even more intelligent (and thus better at creating successor agents) and so instantiates an even more capable successor A2.
The process continues until we approach saturation or the fundamental limits of intelligence (which may well be far beyond the human domain).
The Assumptions Underlying Fast Takeoff
It does not follow from succession that takeoff would be fast or hard. The idea that succession leads to fast takeoff/”foom” rests in an/a few (implicit?) assumptions that I am *very* sceptical of:
That e.g.A5 will have an “easier” time creating A6 than A2 had creating A3.
That it’s easier for Ak[V] to instantiate a successor (Ak+1[V+D]) than for Av[M] to instantiate a successor (Av+1[M+D]) (assuming Av+1 is far from the upper limit of intelligent agents).
“Ease” here can be understood to refer to resources expended in the endeavour:
Compute
Time
Data
Etc.
We can probably aggregate all of the above into the total financial cost of the endeavour. More capable agents may have more resources available, so perhaps the best metric is something like:
Total CostTotal Available/Extractable Resources
The earlier beliefs lead to the below beliefs:
The time between successive generations of agents will shorten. (Perhaps it takes 6 months to move from A3 to A4, but only a month to move from A7 to A8 [again assuming A_8 is far from the upper limits])
The gap in capability between successive agents will widen. (Perhaps the difference in capability between A3 and A4 is 50INT but between A7 and A8 it’s only 50INT)
If at least one of the above two statements is true, then succession leads to a fast/hard takeoff.
What if Intelligence Was Hard?
What if the above assumptions were wrong? Not just wrong, but the exact *opposite* of what is actually the case?
What if as you move to higher levels of intelligence, it becomes *much* harder to eke out additional performance?
I.e. if you graph INT improvement (difference between INT of successor and INT of parent) on the y-axis against the INT of the parent agent on the x-axis (holding “effort” constant), the result is a very sublinear curve (as opposed to superlinear if the assumptions were correct).
Consequences of HIH for the design of successor agents:
The time between successive generations of agents will lengthen progressively. (Perhaps it takes 1 month to move from A3 to A4, but 6 months to move from A7 to A8)
Gap in capability between successive agents will narrow. (Perhaps the difference in capability between A3 and A4 is 50INT but between A7 and A8 it’s only 20INT)
(Again assuming A8 is far from the upper limits)
Note that it follows from HIH that succession does not imply hard or fast takeoff scenarios.
Yes, there’ll eventually be a vastly superhuman successor agent (An for large enough n), but it’ll take a “long time” to get there.
HIH actually implies a slow takeoff. That is, it’ll take (much?) longer to transition from par human systems to superhuman systems than it did to reach par human systems from near-human systems.
Preliminary Conclusions
Thus the “hardness” of intelligence is a very important area of inquiry for determining takeoff dynamics from succession (and in general when hardness with respect to other resources is considered).
I think in worlds where intelligence is sufficiently hard, fast takeoff (via any route) is outright infeasible (all resources expended offer sublinear returns, and some curves are more sublinear than others).
Closing Thoughts
I would appreciate it if those making the claim that succession can induce foom could argue convincingly that intelligence is insufficiently hard.
That if you graphed INT improvement on the y-axis against INT on the x-axis, there’s a relevant non-trivial interval in which the graph is superlinear.
That’s what is required for succession induced foom to be feasible.
I do not recall the original arguments for succession induced foom grappling with the hardness of intelligence at all (however, I originally read them 3 − 5 years ago and may be misremembering them). They seemed to just take it for granted that it was “easy”.
HIH is a strong intuition of mine. I would need strong evidence (including compelling arguments) to shift away from it.
However, it’s not something I can demonstrate or prove definitively. Perhaps in subsequent posts, I’ll explore at length why I think intelligence is hard.
The Hard Intelligence Hypothesis and Its Bearing on Succession Induced Foom
Disclaimer
All notations and formalisms used in this post are outdated and strictly superseded by the notations and formalisms used in “Towards a Formalisation of Returns on Cognitive Reinvestment (Part 1)”. I will come back one day and unify the notation, but I’m not sure how to do that without making this post too unwieldy.
Introduction
Consider that intelligence may be hard (let’s call this the “hard intelligence hypothesis” or HIH). I suspect that it is, I think there’s a lot of empirical evidence for it.
However, for the purpose of this post. I don’t want to explore the evidence, intuitions or arguments underpinning a belief in HIH. I would simply like to speculate on how HIH might bear on recursive self-improvement, and particularly on foom arising from recursive self-improvement.
I think knowing better the “hardness” of intelligence would better inform our position on takeoff dynamics.
A Pre Formal Definition of HIH
My current working definition of HIH is something like:
Intelligence is itself just another resource that can be expended to increase intelligence. That particular case is what I will be considering here.
The “hardness” of intelligence with respect to a given resource can be thought of as how sublinear the curve is (perhaps what complexity class it belongs to?).
What is Recursive Self Improvement?
The basic story of recursive self improvement is that above a given level of intelligence (e.g. “par human”), the AI will be able to inspect and modify its own architecture/algorithms, to directly become more intelligent or design such a successor.
It doesn’t particularly matter whether the system directly self modifies or creates a successor. Self-modification can be understood as a particular case of creating a successor. So for the rest of this post, I’ll look only at the case of successors.
Interlude
Unless otherwise specified, all variables that appear in LaTeX should be understood to be positive real numbers.
A Semi-Formal Description of Succession
Consider an agent Ai, let the successor agent that Ai creates be denoted Ai+1.
Let A0 be the first agent intelligent enough to design successor agents.
Posit also that we have a linear, additive measure of “intelligence”. We’ll denote this measure by INT. I’ll denote an agent Ax with vINT as such: Ax[v]
To explain what I mean by “linear additive measure”, suppose we have 2 tuples of agents: (Ai[z+y]],Ak[z])&(Aj[k+y],Am[k]). Then intra-pair difference in capability is the same.
For convenience’s sake, let 100INT be the intelligence of A0.
So the story of recursive self-improvement is that A0 instantiates a successor A1.
A1 is even more intelligent (and thus better at creating successor agents) and so instantiates an even more capable successor A2.
The process continues until we approach saturation or the fundamental limits of intelligence (which may well be far beyond the human domain).
The Assumptions Underlying Fast Takeoff
It does not follow from succession that takeoff would be fast or hard. The idea that succession leads to fast takeoff/”foom” rests in an/a few (implicit?) assumptions that I am *very* sceptical of:
That e.g.A5 will have an “easier” time creating A6 than A2 had creating A3.
That it’s easier for Ak[V] to instantiate a successor (Ak+1[V+D]) than for Av[M] to instantiate a successor (Av+1[M+D]) (assuming Av+1 is far from the upper limit of intelligent agents).
“Ease” here can be understood to refer to resources expended in the endeavour:
Compute
Time
Data
Etc.
We can probably aggregate all of the above into the total financial cost of the endeavour. More capable agents may have more resources available, so perhaps the best metric is something like:
Total CostTotal Available/Extractable ResourcesThe earlier beliefs lead to the below beliefs:
The time between successive generations of agents will shorten.
(Perhaps it takes 6 months to move from A3 to A4, but only a month to move from A7 to A8 [again assuming A_8 is far from the upper limits])
The gap in capability between successive agents will widen.
(Perhaps the difference in capability between A3 and A4 is 50INT but between A7 and A8 it’s only 50INT)
If at least one of the above two statements is true, then succession leads to a fast/hard takeoff.
What if Intelligence Was Hard?
What if the above assumptions were wrong? Not just wrong, but the exact *opposite* of what is actually the case?
What if as you move to higher levels of intelligence, it becomes *much* harder to eke out additional performance?
I.e. if you graph INT improvement (difference between INT of successor and INT of parent) on the y-axis against the INT of the parent agent on the x-axis (holding “effort” constant), the result is a very sublinear curve (as opposed to superlinear if the assumptions were correct).
Consequences of HIH for the design of successor agents:
The time between successive generations of agents will lengthen progressively.
(Perhaps it takes 1 month to move from A3 to A4, but 6 months to move from A7 to A8)
Gap in capability between successive agents will narrow.
(Perhaps the difference in capability between A3 and A4 is 50INT but between A7 and A8 it’s only 20INT)
(Again assuming A8 is far from the upper limits)
Note that it follows from HIH that succession does not imply hard or fast takeoff scenarios.
Yes, there’ll eventually be a vastly superhuman successor agent (An for large enough n), but it’ll take a “long time” to get there.
HIH actually implies a slow takeoff. That is, it’ll take (much?) longer to transition from par human systems to superhuman systems than it did to reach par human systems from near-human systems.
Preliminary Conclusions
Thus the “hardness” of intelligence is a very important area of inquiry for determining takeoff dynamics from succession (and in general when hardness with respect to other resources is considered).
I think in worlds where intelligence is sufficiently hard, fast takeoff (via any route) is outright infeasible (all resources expended offer sublinear returns, and some curves are more sublinear than others).
Closing Thoughts
I would appreciate it if those making the claim that succession can induce foom could argue convincingly that intelligence is insufficiently hard.
That if you graphed INT improvement on the y-axis against INT on the x-axis, there’s a relevant non-trivial interval in which the graph is superlinear.
That’s what is required for succession induced foom to be feasible.
I do not recall the original arguments for succession induced foom grappling with the hardness of intelligence at all (however, I originally read them 3 − 5 years ago and may be misremembering them). They seemed to just take it for granted that it was “easy”.
HIH is a strong intuition of mine. I would need strong evidence (including compelling arguments) to shift away from it.
However, it’s not something I can demonstrate or prove definitively. Perhaps in subsequent posts, I’ll explore at length why I think intelligence is hard.