In fields with lots of spending and impact, most technological progress is made gradually rather than abruptly, for most ways of measuring...
Your arguments about technology X apply to any other technological goal—better fusion reactors or solar panels, more generally cheaper energy, rockets, semiconductors, whatever. So it seems like they should be visible in the base rate for 2. Do you think that a significant fraction of technological progress is abrupt and unpredictable in the sense that you are saying TAI will probably be?
I think that you can roughly divide progress into “qualitatively new ideas” (QNI) and “incremental improvement of existing technology” (ofc in reality it’s a spectrum). The first kind is much less predictable than the second kind. Now, when a QNI comes along, it doesn’t necessarily look like a discontinuity, because there might be a lot of work to bridge the distance between idea and implementation. And, this work involves a lot of small details. Because of this, the first version is probably often only a slight improvement on SOTA. So, I’m guessing that QNIs produce something more like a discontinuity in the derivative than a discontinuity in the SOTA itself.
Under this model, most progress is “gradual” in the sense that at most points the graph is differentiable. But (i) it doesn’t work too well to extrapolate trends across QNI points and (ii) the counterfactual impact of QNIs is a large fraction of progress.
Certainly I don’t see fusion reactors, solar panels or (use in electronics of) semiconductors as counterexamples, since each of these was invented at some point, and didn’t gradually evolve from some completely different technology.
Another factor is that in software the distance between idea and implementation tends to be smaller, because processors and operating systems abstract much of the details for you. I think this is partly responsible for “startups” being more of a thing in software than in other fields (ofc another part of it is that software is just more new with more low-hanging fruits). And, this probably makes progress in software less smooth.
For unaligned TAI, the effective distance might be shorter still. Because, for most software there’s a fair amount of work going towards UI and/or integration with other software. But, with TAI this can be unnecessary. Moreover, the alignment problem itself is part of the would-be idea-to-profitable-implementation gap. On the other hand, optimizing performance can also be a large part of idea-to-implementation, and if the first AGI is e.g. a drastically slow superintelligence, this can be compatible with a slow takeoff.
Certainly I don’t see fusion reactors, solar panels or (use in electronics of) semiconductors as counterexamples, since each of these was invented at some point, and didn’t gradually evolve from some completely different technology.
Your definition of “discontinuity” seems broadly compatible with my view of the future then. Definitely there are different technologies that are not all outgrowths of one another.
My main point of divergence is:
Now, when a QNI comes along, it doesn’t necessarily look like a discontinuity, because there might be a lot of work to bridge the distance between idea and implementation. And, this work involves a lot of small details. Because of this, the fist version is probably often only a slight improvement on SOTA.
I think that most of the time when a QNI comes along it is worse than the previous thing and takes work to bring up to the level of the previous thing. In small areas no one pays attention until it overtakes SOTA, but in big areas people usually start paying attention (and investing a significant fraction of the prior SOTA’s size) well before the cross-over point. This seems true for solar or fusion, or digital computers, or deep learning for that matter, or self-driving cars or early automobiles.
If that’s right, then you are looking at two continuous curves and you can think about when they cross and you usually start to get a lot of data before the crossover point. And indeed this is obviously how I’m thinking about technologies like deep learning, which are currently useless for virtually all tasks but which I expect to relatively soon overtake alternatives (like humans and other software) in a huge range of very important domains.
And if some other AI technology replaces deep learning, I generally expect the same story. There is a scale at which new things can burst onto the scene, but over time that scale becomes smaller and smaller relative to the scale of the field. At this point the appearance of “bursting onto the scene” is primarily driven by big private projects that don’t talk publicly about what they are doing for a while (e.g. putting in 20 person-years of effort before a public announcement, so that they get data internally but an outsider just sees a discontinuity), but even that seems to be drying up fairly quickly.
I’m not sure what’s the difference between what you’re saying here and what I said about QNIs. Is it that you expect being able to see the emergent technology before the singular (crossover) point? Actually, the fact you describe DL as “currently useless” makes me think we should be talking about progress as a function of two variables: time and “maturity”, where maturity inhabits, roughly speaking, a scale from “theoretical idea” to “proof of concept” to “beats SOTA in lab conditions” to “commercial product”. In this sense, the “lab progress” curve is already past the DL singularity but the “commercial progress” curve maybe isn’t.
On this model, if post-DL AI technology X appears tomorrow, it will take it some time to span the distance from “theoretical idea” to “commercial product”, in which time we would notice it and update our predictions accordingly. But, two things to note here:
First, it’s not clear which level of maturity is the relevant reference point for AI risk. In particular, I don’t think you need commercial levels of maturity for AI technology to become risky, for the reasons I discussed in my previous comment (and, we can also add regulatory barriers to that list, although I am not convinced they are as important as Yudkowsky seems to believe).
Second, all this doesn’t sound to me like “AI systems will grow relatively continuously and predictably”, although maybe I just interpreted this statement differently from its intent. For instance, I agree that it’s unlikely technology X will emerge specifically in the next year, so progress over the next year should be fairly predictable. On the other hand, I don’t think it would be very surprising if technology X emerges in the next decade.
IIUC, part of what you’re saying can be rephrased as: TAI is unlikely to be created by a small team, since once a small team shows something promising, tonnes of resources will be thrown at them (and at other teams that might be able to copy the technology) and they won’t be a small team anymore. Which sounds plausible, I suppose, but doesn’t make TAI predictable that long in advance.
Now, when a QNI comes along, it doesn’t necessarily look like a discontinuity, because there might be a lot of work to bridge the distance between idea and implementation. And, this work involves a lot of small details. Because of this, the first version is probably often only a slight improvement on SOTA. So, I’m guessing that QNIs produce something more like a discontinuity in the derivative than a discontinuity in the SOTA itself.
Don’t have a great source for this at hand, but my impression is that seemingly-QNIs surprisingly often just power existing exponential trends, meaning no change in derivative (on a log graph).
(A random comment in support of this — I remember chip design expert Jim Keller saying on Lex Fridman’s podcast that Moore’s Law is just a bunch of separate s-curves, as they have to come up with new ideas to work through challenges to shrinking transistors, and the new techniques work for a range of scales and then have to be replaced with new new ideas.)
Not sure if this question is easily settled, but it might be a crux for various views — how often do QNIs actually change the slope of the curve?
I think that you can roughly divide progress into “qualitatively new ideas” (QNI) and “incremental improvement of existing technology” (ofc in reality it’s a spectrum). The first kind is much less predictable than the second kind. Now, when a QNI comes along, it doesn’t necessarily look like a discontinuity, because there might be a lot of work to bridge the distance between idea and implementation. And, this work involves a lot of small details. Because of this, the first version is probably often only a slight improvement on SOTA. So, I’m guessing that QNIs produce something more like a discontinuity in the derivative than a discontinuity in the SOTA itself.
Under this model, most progress is “gradual” in the sense that at most points the graph is differentiable. But (i) it doesn’t work too well to extrapolate trends across QNI points and (ii) the counterfactual impact of QNIs is a large fraction of progress.
Certainly I don’t see fusion reactors, solar panels or (use in electronics of) semiconductors as counterexamples, since each of these was invented at some point, and didn’t gradually evolve from some completely different technology.
Another factor is that in software the distance between idea and implementation tends to be smaller, because processors and operating systems abstract much of the details for you. I think this is partly responsible for “startups” being more of a thing in software than in other fields (ofc another part of it is that software is just more new with more low-hanging fruits). And, this probably makes progress in software less smooth.
For unaligned TAI, the effective distance might be shorter still. Because, for most software there’s a fair amount of work going towards UI and/or integration with other software. But, with TAI this can be unnecessary. Moreover, the alignment problem itself is part of the would-be idea-to-profitable-implementation gap. On the other hand, optimizing performance can also be a large part of idea-to-implementation, and if the first AGI is e.g. a drastically slow superintelligence, this can be compatible with a slow takeoff.
Your definition of “discontinuity” seems broadly compatible with my view of the future then. Definitely there are different technologies that are not all outgrowths of one another.
My main point of divergence is:
I think that most of the time when a QNI comes along it is worse than the previous thing and takes work to bring up to the level of the previous thing. In small areas no one pays attention until it overtakes SOTA, but in big areas people usually start paying attention (and investing a significant fraction of the prior SOTA’s size) well before the cross-over point. This seems true for solar or fusion, or digital computers, or deep learning for that matter, or self-driving cars or early automobiles.
If that’s right, then you are looking at two continuous curves and you can think about when they cross and you usually start to get a lot of data before the crossover point. And indeed this is obviously how I’m thinking about technologies like deep learning, which are currently useless for virtually all tasks but which I expect to relatively soon overtake alternatives (like humans and other software) in a huge range of very important domains.
And if some other AI technology replaces deep learning, I generally expect the same story. There is a scale at which new things can burst onto the scene, but over time that scale becomes smaller and smaller relative to the scale of the field. At this point the appearance of “bursting onto the scene” is primarily driven by big private projects that don’t talk publicly about what they are doing for a while (e.g. putting in 20 person-years of effort before a public announcement, so that they get data internally but an outsider just sees a discontinuity), but even that seems to be drying up fairly quickly.
I’m not sure what’s the difference between what you’re saying here and what I said about QNIs. Is it that you expect being able to see the emergent technology before the singular (crossover) point? Actually, the fact you describe DL as “currently useless” makes me think we should be talking about progress as a function of two variables: time and “maturity”, where maturity inhabits, roughly speaking, a scale from “theoretical idea” to “proof of concept” to “beats SOTA in lab conditions” to “commercial product”. In this sense, the “lab progress” curve is already past the DL singularity but the “commercial progress” curve maybe isn’t.
On this model, if post-DL AI technology X appears tomorrow, it will take it some time to span the distance from “theoretical idea” to “commercial product”, in which time we would notice it and update our predictions accordingly. But, two things to note here:
First, it’s not clear which level of maturity is the relevant reference point for AI risk. In particular, I don’t think you need commercial levels of maturity for AI technology to become risky, for the reasons I discussed in my previous comment (and, we can also add regulatory barriers to that list, although I am not convinced they are as important as Yudkowsky seems to believe).
Second, all this doesn’t sound to me like “AI systems will grow relatively continuously and predictably”, although maybe I just interpreted this statement differently from its intent. For instance, I agree that it’s unlikely technology X will emerge specifically in the next year, so progress over the next year should be fairly predictable. On the other hand, I don’t think it would be very surprising if technology X emerges in the next decade.
IIUC, part of what you’re saying can be rephrased as: TAI is unlikely to be created by a small team, since once a small team shows something promising, tonnes of resources will be thrown at them (and at other teams that might be able to copy the technology) and they won’t be a small team anymore. Which sounds plausible, I suppose, but doesn’t make TAI predictable that long in advance.
Don’t have a great source for this at hand, but my impression is that seemingly-QNIs surprisingly often just power existing exponential trends, meaning no change in derivative (on a log graph).
(A random comment in support of this — I remember chip design expert Jim Keller saying on Lex Fridman’s podcast that Moore’s Law is just a bunch of separate s-curves, as they have to come up with new ideas to work through challenges to shrinking transistors, and the new techniques work for a range of scales and then have to be replaced with new new ideas.)
Not sure if this question is easily settled, but it might be a crux for various views — how often do QNIs actually change the slope of the curve?