I think 24% for “there will be a big discontinuity at some point in the history of a field” is pretty reasonable, though I have some quibbles with your estimates (detailed below). I think there are a bunch of additional facts that make me go a lot lower than that on the specific question we have with AI:
We’re talking about a discontinuity at a specific moment along the curve—not just “there will be a discontinuity in AI progress at some point”, but specifically “there will be a discontinuity around the point where AI systems first reach approximately human-level intelligence”. Assigning 5% to a discontinuity at a specific region of the curve can easily be compatible with 24% of a discontinuity overall.
We also know that the field of AI has been around for 60 years and that there is a lot of effort being put into building powerful AI systems. I expect that the more effort is being put into something, the more the low-hanging fruit / “secrets” are already plucked, and the less likely discontinuities are.
We can’t currently point to anything that seems like it should cause a discontinuity in the future. It seems to me like for many of the “physics hack” style of discontinuity, the discontinuity would have been predictable in advance (I’m thinking especially of nukes and spaceflight here). Though possibly this is just hindsight bias on my part.
We were talking about a huge discontinuity—that the first time we destroy cities, we will also destroy the Earth. (And I think we’re talking about a similarly large discontinuity in the AI case, though I’m not actually sure.) These intuitively feel way larger than your big discontinuities. Though as a counterpoint, I also think AI will be a way bigger deal than most of the other technologies, so a similar discontinuity in some underlying trend could lead to a much bigger discontinuity in terms of impact. (Still, if we talk about “smaller” discontinuities like 1-year doubling of GDP before 4-year doubling of GDP, I put more probability on it, relative to something like “the world looks pretty similar to today’s world, and then everyone drops dead”.)
(All of these together would push me way lower than 5%, if ignoring model uncertainty / “maybe I’m wrong” + noting that the future is hard to predict.)
It seems to me that the biggest point of disagreement is on (3), and this is why in the conversation I keep coming back to
My impression is that Eliezer thinks that “general intelligence” is a qualitatively different sort of thing than that-which-neural-nets-are-doing, and maybe that’s what’s analogous to “entirely new physics”. I’m pretty unconvinced of this, but something in this genre feels quite crux-y for me.
I do think “look at historical examples” is a good thing to do, so I’ll go through each of your discontinuities in turn. Note that I know very little about most of these areas and haven’t even read the Wikipedia page for most of them, so lots of things I say could be completely wrong:
Aviation: I assume you’re talking about the zero-to-one discontinuity from “no flight” to “flight”? I do agree that we’ll see zero-to-one discontinuities on particular AI capabilities, e.g. language models learn to do arithmetic quite discontinuously. This seems pretty irrelevant to the case with AI. (Notably, the Wright flyer didn’t have much of an impact on things people cared about, and not that many people were working on flight to my knowledge.)
Nukes: Agree that this is a zero-to-one discontinuity from “physics hack” (but note that it did involve a huge amount of effort). Unlike the Wright flyer, it did have a huge impact on things people cared about.
Petroleum: Not sure what the discontinuity is—is it that “amount of petroleum used” increased discontinuously? If so, I very much expect such discontinuities to happen; they’ll happen any time a better technology replaces a worse technology. (Put another way, the reason to expect continuous progress is that there is optimization pressure on the metric and so the low-hanging fruit / “secrets” have already been taken; there wouldn’t have been much optimization pressure on “amount of petroleum used”.) I also expect that there has been a discontinuity in “use of neural nets for machine learning”, and similarly I expect AI coding assistants will become hugely more popular in the nearish future. The relevant question to me is whether we saw a discontinuity in something like “ability to heat your home” or “ability to travel long distances” or something like that.
Printing: Going off of AI Impacts’ investigation, I’d count this one. I think partly this was because there wasn’t much effort going into this. (It looks like when the printing press was invented we were producing ~50,000 manuscripts per year, using about 25,000 person-years of labor. Presumably much much less than that was going into optimizing the process, similarly to how R&D in machine translation is way way lower than the size of the translation market.)
Spaceflight: Agree that this is a zero-to-one discontinuity from “physics hack” (but note that it did involve a huge amount of effort). Although if you’re saying that more resources were spent on it, same comment as petroleum.
Rockets: I’d love to see numbers, but this does sound like a discontinuity that’s relevant to the case with AI. I’d also want to know how much people cared about it (plausibly quite a lot).
Aluminium: Looking at AI Impacts, I think I’m at “probably a discontinuity relevant to the case with AI, but not a certainty”.
Radar: I’d need more details about what happened here, but it seems like this is totally consistent with the “continuous view” (since “with more effort you got more progress” seems like a pretty central conclusion of the model ).
Radio: Looking at AI Impacts, I think this one looks more like “lots of crazy fast progress that is fueled by frequent innovations”, which seems pretty compatible with the “continuous view” on AI. (Though I’m sympathetic to the critique that the double exponential curve chosen by AI Impacts is an instance of finding a line by which things look smooth; I definitely wouldn’t have chosen that functional form in advance of seeing the data.)
Automobile: I’d assume that there was very little optimization on “speed of production of cars” at the time, given that cars had only just become commercially viable, so a discontinuity seems unsurprising.
Transistors: Wikipedia claims “the MOSFET was also initially slower and less reliable than the BJT”, and further discussion seems to suggest that its benefits were captured with further work and effort (e.g. it was a twentieth the size of a BJT by the 1990s, decades after invention). This sounds like it wasn’t a discontinuity to me. What metric did you think it was a discontinuity for?
PCR: I don’t know enough about the field—sounds like a zero-to-one discontinuity (or something very close, where ~no one was previously trying to do the things PCR does). See aviation.
Transistors: Wikipedia claims “the MOSFET was also initially slower and less reliable than the BJT”, and further discussion seems to suggest that its benefits were captured with further work and effort (e.g. it was a twentieth the size of a BJT by the 1990s, decades after invention). This sounds like it wasn’t a discontinuity to me.
I am also skeptical that the MOSFET produced a discontinuity. Plausibly, what we care about is the number of computations we can do per dollar. Nordhaus (2007) provides data showing that that the rate of progress on this metric was practically unchanged at the time the MOSFET was invented, in 1959.
I think 24% for “there will be a big discontinuity at some point in the history of a field” is pretty reasonable, though I have some quibbles with your estimates (detailed below). I think there are a bunch of additional facts that make me go a lot lower than that on the specific question we have with AI:
We’re talking about a discontinuity at a specific moment along the curve—not just “there will be a discontinuity in AI progress at some point”, but specifically “there will be a discontinuity around the point where AI systems first reach approximately human-level intelligence”. Assigning 5% to a discontinuity at a specific region of the curve can easily be compatible with 24% of a discontinuity overall.
We also know that the field of AI has been around for 60 years and that there is a lot of effort being put into building powerful AI systems. I expect that the more effort is being put into something, the more the low-hanging fruit / “secrets” are already plucked, and the less likely discontinuities are.
We can’t currently point to anything that seems like it should cause a discontinuity in the future. It seems to me like for many of the “physics hack” style of discontinuity, the discontinuity would have been predictable in advance (I’m thinking especially of nukes and spaceflight here). Though possibly this is just hindsight bias on my part.
We were talking about a huge discontinuity—that the first time we destroy cities, we will also destroy the Earth. (And I think we’re talking about a similarly large discontinuity in the AI case, though I’m not actually sure.) These intuitively feel way larger than your big discontinuities. Though as a counterpoint, I also think AI will be a way bigger deal than most of the other technologies, so a similar discontinuity in some underlying trend could lead to a much bigger discontinuity in terms of impact. (Still, if we talk about “smaller” discontinuities like 1-year doubling of GDP before 4-year doubling of GDP, I put more probability on it, relative to something like “the world looks pretty similar to today’s world, and then everyone drops dead”.)
(All of these together would push me way lower than 5%, if ignoring model uncertainty / “maybe I’m wrong” + noting that the future is hard to predict.)
It seems to me that the biggest point of disagreement is on (3), and this is why in the conversation I keep coming back to
I do think “look at historical examples” is a good thing to do, so I’ll go through each of your discontinuities in turn. Note that I know very little about most of these areas and haven’t even read the Wikipedia page for most of them, so lots of things I say could be completely wrong:
Aviation: I assume you’re talking about the zero-to-one discontinuity from “no flight” to “flight”? I do agree that we’ll see zero-to-one discontinuities on particular AI capabilities, e.g. language models learn to do arithmetic quite discontinuously. This seems pretty irrelevant to the case with AI. (Notably, the Wright flyer didn’t have much of an impact on things people cared about, and not that many people were working on flight to my knowledge.)
Nukes: Agree that this is a zero-to-one discontinuity from “physics hack” (but note that it did involve a huge amount of effort). Unlike the Wright flyer, it did have a huge impact on things people cared about.
Petroleum: Not sure what the discontinuity is—is it that “amount of petroleum used” increased discontinuously? If so, I very much expect such discontinuities to happen; they’ll happen any time a better technology replaces a worse technology. (Put another way, the reason to expect continuous progress is that there is optimization pressure on the metric and so the low-hanging fruit / “secrets” have already been taken; there wouldn’t have been much optimization pressure on “amount of petroleum used”.) I also expect that there has been a discontinuity in “use of neural nets for machine learning”, and similarly I expect AI coding assistants will become hugely more popular in the nearish future. The relevant question to me is whether we saw a discontinuity in something like “ability to heat your home” or “ability to travel long distances” or something like that.
Printing: Going off of AI Impacts’ investigation, I’d count this one. I think partly this was because there wasn’t much effort going into this. (It looks like when the printing press was invented we were producing ~50,000 manuscripts per year, using about 25,000 person-years of labor. Presumably much much less than that was going into optimizing the process, similarly to how R&D in machine translation is way way lower than the size of the translation market.)
Spaceflight: Agree that this is a zero-to-one discontinuity from “physics hack” (but note that it did involve a huge amount of effort). Although if you’re saying that more resources were spent on it, same comment as petroleum.
Rockets: I’d love to see numbers, but this does sound like a discontinuity that’s relevant to the case with AI. I’d also want to know how much people cared about it (plausibly quite a lot).
Aluminium: Looking at AI Impacts, I think I’m at “probably a discontinuity relevant to the case with AI, but not a certainty”.
Radar: I’d need more details about what happened here, but it seems like this is totally consistent with the “continuous view” (since “with more effort you got more progress” seems like a pretty central conclusion of the model ).
Radio: Looking at AI Impacts, I think this one looks more like “lots of crazy fast progress that is fueled by frequent innovations”, which seems pretty compatible with the “continuous view” on AI. (Though I’m sympathetic to the critique that the double exponential curve chosen by AI Impacts is an instance of finding a line by which things look smooth; I definitely wouldn’t have chosen that functional form in advance of seeing the data.)
Automobile: I’d assume that there was very little optimization on “speed of production of cars” at the time, given that cars had only just become commercially viable, so a discontinuity seems unsurprising.
Transistors: Wikipedia claims “the MOSFET was also initially slower and less reliable than the BJT”, and further discussion seems to suggest that its benefits were captured with further work and effort (e.g. it was a twentieth the size of a BJT by the 1990s, decades after invention). This sounds like it wasn’t a discontinuity to me. What metric did you think it was a discontinuity for?
PCR: I don’t know enough about the field—sounds like a zero-to-one discontinuity (or something very close, where ~no one was previously trying to do the things PCR does). See aviation.
I am also skeptical that the MOSFET produced a discontinuity. Plausibly, what we care about is the number of computations we can do per dollar. Nordhaus (2007) provides data showing that that the rate of progress on this metric was practically unchanged at the time the MOSFET was invented, in 1959.