I believe that the disagreement is mostly about what happens before we build powerful AGI. I think that weaker AI systems will already have radically transformed the world, while I believe fast takeoff proponents think there are factors that makes weak AI systems radically less useful. This is strategically relevant because I’m imagining AGI strategies playing out in a world where everything is already going crazy, while other people are imagining AGI strategies playing out in a world that looks kind of like 2018 except that someone is about to get a decisive strategic advantage.
Huh, I think this is most useful framing of the Slow/Fast takeoff paradigm I’ve found. Normally when I hear about the slow takeoff strategy I think it comes with a vague sense of “and therefore, it’s not worth worrying out because our usual institutions will be able to handle it.”
I’m not sure if I’m convinced about the continuous takeoff, but it at least seems plausible. It still seems to me that recursive self improvement should be qualitatively different—not necessarily resulting in a “discontinuitous jump” but changing the shape of the curve.
The curve below is a bit sharper than I meant it to come out, I had some trouble getting photoshop to cooperate.
More thoughts to come. I agree with Qiaochu that this is a good “actually think for N minutes” kinda post.
It still seems to me that recursive self improvement should be qualitatively different
Could you say more about this intuition?
When I think of an AGI system engaging in RSI, I imagine it reasoning in a very general, first principles sort of way that, “I want to improve myself, and I run on TPUs, so I should figure out how to make better TPUs.” But at the point that we have a system that can do that, we’re already going to be optimizing the fuck out of TPUs. And we’re going to be using all our not-quite-so-general systems to search for other things to optimize.
So where does the discontinuity (or curve shape change) come from?
The curve is hyperbolic (rather than exponential) because it already includes the loop (output --> better tech --> higher productivity --> more output), and RSI just seems like an important kind of better tech. So I’m not convinced RSI changes the shape of the curve (though it might be a more “local” form of improvement).
This highlights perhaps a different thing, which is that… it’d seem very strange to me if the benefits from RSI exactly mapped onto an existing curve.
(This makes me think of an old Scott Alexander post reviewing the Great Stagnation, about how weird it is that all these economic curves seem pretty straight forwardly curvy. So this may be a domain where reality is normal and my intuitions are weird.)
There’s another thing that bothers me too, which is the sheer damnable linearity of the economic laws. The growth of science as measured in papers/year; the growth of innovation as measured in patents; the growth of computation as measured by Moore’s Law; the growth of the economy as measured in GDP. All so straight you could use them to hang up a picture. These lines don’t care what we humans do, who we vote for, what laws we pass, what we invent, what geese that lay long-hanging fruit have or have not been killed. The Gods of the Straight Lines seem right up there with the Gods of the Copybook Headings in the category of things that tend to return no matter how hard you try to kick them away. They were saying back a few decades ago Moore’s Law would stop because of basic physical restrictions on possible transistor size, and we worked around those, which means the Gods of the Straight Lines are more powerful than physics.
This is one of my major go-to examples of this really weird linear phenomenon:
150 years of a completely straight line! There were two world wars in there, the development of artificial fertilizer, the broad industrialization of society, the invention of the car. And all throughout the line just carries one, with no significant perturbations.
I’m confused. Moore’s law, GDP growth etc. are linear in log-space. That graph isn’t. Why are these spaces treated as identical for the purposes of carrying the same important and confusing intuition? (E.g. I imagine one could find lots of weird growth functions that are linear in some space. Why are they all important?)
Hmm, in either case a function being linear in a very easy to describe space (i.e. log, log-log or linear) highlights that the relationship the function describes is extremely simple, and seems to be independent of most factors that vary drastically over time. I expect the world to be stochastic and messy, with things going up and down for random reasons and with things over time going up and down quite a bit for very local reasons, but these graphs do not seem to conform with that intuition in a way that I don’t easily know how to reconcile.
It is remarkable how certain industries are protected against shocks, this reminds me of countries who easily bare financial crises as a result of being primary food exporters (NZ). What is even more interesting is the difficulting of fully getting rid of humans in an econmic endevour (latter part of the chart).
Regarding your point on linearity, this chart does show a linear decrease as of 1800. However lets say farming activity halved as a portion of all jobs in the first 100 years, in the latter hundred years it decreased with a multiple of 25. So in the first half it was transformed by X*1/2 and in the second half X*1/25. Is this still linear ? I was wondering about this phenomena, is there something like absolute and relative linearity or exponentiality ?
It seems that the percentage change then is exponential
I think this is just a sigmoid function, but mirrored over the y-axis. If you extended it farther into the past, it would certainly flatten out just below 100%. So I think it’s just another example of how specific technologies are adopted in sigmoid curves, except in reverse, because people are dis-adopting manual farming.
(And I think the question of why tech grows in sigmoid curves is because that’s the solution to the differential equation that models the fundamental dynamics of “grows proportional to position, up to a carrying capacity”.)
I don’t think that would be strange. It seems like a priori RSI should result in hyperbolic growth for the same reason that technology should be hyperbolic, since the dynamic is basically identical. You could argue that the relationship between human cognitive effort and machine cognitive effort is different than the relationship between human labor and machine labor, but that seems like it’s lost in the noise. You could make an argument about discontinuities near 100% automation, but as far as I can currently tell those arguments fall apart on inspection.
Cool, but what is the Y axis exactly? How do you measure it? I feel that I’m expected to have an emotional response to how steep those lines are, and it bothers me. When Y is undefined and X has no dates on it, the whole disagreement seems a bit vapid.
Huh, I think this is most useful framing of the Slow/Fast takeoff paradigm I’ve found. Normally when I hear about the slow takeoff strategy I think it comes with a vague sense of “and therefore, it’s not worth worrying out because our usual institutions will be able to handle it.”
I’m not sure if I’m convinced about the continuous takeoff, but it at least seems plausible. It still seems to me that recursive self improvement should be qualitatively different—not necessarily resulting in a “discontinuitous jump” but changing the shape of the curve.
The curve below is a bit sharper than I meant it to come out, I had some trouble getting photoshop to cooperate.
More thoughts to come. I agree with Qiaochu that this is a good “actually think for N minutes” kinda post.
100% I visualised the above graph and wanted to draw it but didn’t feel like I had the technical skills. Thanks.
Could you say more about this intuition?
When I think of an AGI system engaging in RSI, I imagine it reasoning in a very general, first principles sort of way that, “I want to improve myself, and I run on TPUs, so I should figure out how to make better TPUs.” But at the point that we have a system that can do that, we’re already going to be optimizing the fuck out of TPUs. And we’re going to be using all our not-quite-so-general systems to search for other things to optimize.
So where does the discontinuity (or curve shape change) come from?
The curve is hyperbolic (rather than exponential) because it already includes the loop (output --> better tech --> higher productivity --> more output), and RSI just seems like an important kind of better tech. So I’m not convinced RSI changes the shape of the curve (though it might be a more “local” form of improvement).
This highlights perhaps a different thing, which is that… it’d seem very strange to me if the benefits from RSI exactly mapped onto an existing curve.
(This makes me think of an old Scott Alexander post reviewing the Great Stagnation, about how weird it is that all these economic curves seem pretty straight forwardly curvy. So this may be a domain where reality is normal and my intuitions are weird.)
This is one of my major go-to examples of this really weird linear phenomenon:
150 years of a completely straight line! There were two world wars in there, the development of artificial fertilizer, the broad industrialization of society, the invention of the car. And all throughout the line just carries one, with no significant perturbations.
I’m confused. Moore’s law, GDP growth etc. are linear in log-space. That graph isn’t. Why are these spaces treated as identical for the purposes of carrying the same important and confusing intuition? (E.g. I imagine one could find lots of weird growth functions that are linear in some space. Why are they all important?)
Hmm, in either case a function being linear in a very easy to describe space (i.e. log, log-log or linear) highlights that the relationship the function describes is extremely simple, and seems to be independent of most factors that vary drastically over time. I expect the world to be stochastic and messy, with things going up and down for random reasons and with things over time going up and down quite a bit for very local reasons, but these graphs do not seem to conform with that intuition in a way that I don’t easily know how to reconcile.
It is remarkable how certain industries are protected against shocks, this reminds me of countries who easily bare financial crises as a result of being primary food exporters (NZ). What is even more interesting is the difficulting of fully getting rid of humans in an econmic endevour (latter part of the chart).
Regarding your point on linearity, this chart does show a linear decrease as of 1800. However lets say farming activity halved as a portion of all jobs in the first 100 years, in the latter hundred years it decreased with a multiple of 25. So in the first half it was transformed by X*1/2 and in the second half X*1/25. Is this still linear ? I was wondering about this phenomena, is there something like absolute and relative linearity or exponentiality ?
It seems that the percentage change then is exponential
I think this is just a sigmoid function, but mirrored over the y-axis. If you extended it farther into the past, it would certainly flatten out just below 100%. So I think it’s just another example of how specific technologies are adopted in sigmoid curves, except in reverse, because people are dis-adopting manual farming.
(And I think the question of why tech grows in sigmoid curves is because that’s the solution to the differential equation that models the fundamental dynamics of “grows proportional to position, up to a carrying capacity”.)
I don’t think that would be strange. It seems like a priori RSI should result in hyperbolic growth for the same reason that technology should be hyperbolic, since the dynamic is basically identical. You could argue that the relationship between human cognitive effort and machine cognitive effort is different than the relationship between human labor and machine labor, but that seems like it’s lost in the noise. You could make an argument about discontinuities near 100% automation, but as far as I can currently tell those arguments fall apart on inspection.
Cool, but what is the Y axis exactly? How do you measure it? I feel that I’m expected to have an emotional response to how steep those lines are, and it bothers me. When Y is undefined and X has no dates on it, the whole disagreement seems a bit vapid.