We’re in the Eliezerverse with huge kinks in loss graphs on automated programming/Putnam problems.
Not from scaling up inputs but from a local discovery that is much bigger in impact than the sorts of jumps we observe from things like Transformers.
[Yudkowsky][22:21]
but, sure, “huge kinks in loss graphs on automated programming / Putnam problems” sounds like something that is, if not mandated on my model, much more likely than it is in the Paulverse. though I am a bit surprised because I would not have expected Paul to be okay betting on that.
to this,
[Rohin] To the extent this is accurate, it doesn’t seem like you really get to make a bet that resolves before the end times, since you agree on basically everything until the point at which Eliezer predicts that you get the zero-to-one transition on the underlying driver of impact.
Eliezer does (at least weakly) expect more trend breaks before The End (even on metrics that aren’t qualitative measures of impressiveness/intelligence, but just things like model loss functions), despite the fact that Rohin’s summary of his view is (I think) roughly accurate.
What explains this? I think it’s something roughly like, part of the reason Eliezer expects a sudden transition when we reach the core of generality in the first place is because he thinks that’s how things usually go in the history of tech/AI progress—there’s also specific reasons to think it will happen in the case of finding the core of generality, but there are also general reasons. See e.g. this from Eliezer:
well, the Eliezerverse has more weird novel profitable things, because it has more weirdness
I take ‘more weirdness’ to mean something like more discoveries that induce sudden improvements out there in general.
So I think that’s why his view does make (weaker) differential predictions about earlier events that we can test, not because the zero-to-one core of generality hypothesis predicts anything about narrow AI progress, but because some of the beliefs that led to that hypothesis do.
We can see there’s two (connected) lines of argument and that Eliezer and Paul/Carl/Richard have different things to say on each − 1 is more localized and about seeing what we can learn about AGI specifically, and 2 is about reference class reasoning and what tech progress in general tells us about AGI:
Specific to AGI: What can we infer from human evolution and interrogating our understanding of general intelligence (?) about whether AGI will arrive suddenly?
Reference Class: What can we infer about AGI progress from the general record of technological progress, especially how common big impacts are when there’s lots of effort and investment?
My sense is that Eliezer answers
Big update for Eliezer’s view: This tells us a lot, in particular we learn evolution got to the core of generality quickly, so AI progress will probably get there quickly as well. Plus, Humans are an existence proof for the core of generality, which suggests our default expectation should be sudden progress when we hit the core.
Smaller update for Eliezer’s view: This isn’t that important—there’s no necessary connection between AGI and e.g. bridges or nukes. But, you can at least see that there’s not a strong consistent track record of continuous improvement once you understand the historical record the right way (plus the underlying assumption in 2 that there will be a lot of effort and investment is probably wrong as well). Nonetheless, if you avoid retrospective trend-fitting and look at progress in the most natural (qualitative?) way, you’ll see that early discoveries that go from 0 to 1 are all over the place—Bitcoin, the Wright flyer, nuclear weapons are at least not crazy exceptions and quite possibly the default.
While Paul and Carl(?) answer,
Smaller update for Paul’s view: The disanalogies between AI progress and Evolution all point in the direction of AI progress being smoother than evolution (we’re intelligently trying to find the capabilities we want) - we get a weak update in favour of the smooth progress view from understanding this disanalogy between AI progress and evolution, but really we don’t learn much, except that there aren’t any good reasons to think there are only a few paths to a large set of powerful world affecting capabilities. Also, the core of generality idea is wrong, so the idea that Humans are an existence proof for it or that evolution tells us something about how to find it is wrong.
Big update for Paul’s view: reasoning from the reference class of ‘technologies where there are many opportunities for improvement and many people trying different things at once’ lets us see why expecting smooth progress should be the default. It’s because as long as there are lots of paths to improvement in the underlying capability landscape (which is the default because that’s how the world works by default), and there are lots of people trying to make improvements in different ways, the incremental changes add up to smooth outputs.
So Eliezer’s claim that Paul et al’s trend-fitting must include
doing something sophisticated but wordless, where they fit a sophisticated but wordless universal model of technological permittivity to bridge lengths, then have a wordless model of cognitive scaling in the back of their minds
is sort of correct, but the model isn’t really sophisticated or wordless.
The model is: as long as the underlying ‘capability landscape’ offers many paths to improvements, not just a few really narrow ones that swamp everything else, lots of people intelligently trying lots of different approaches will lead to lots of small discoveries that add up. Additionally, most examples of tech progress look like ‘multiple ways of doing something that add up’, this is confirmed by the historical record.
And then the model of cognitive scaling consists of (among other things) specific counterarguments to the claim that AGI progress is one of those cases with a few big paths to improvement (e.g. Evolution doesn’t give us evidence that AGI progress will be sudden).
[Shulman]
As I work through sectors and the rollout of past automation I see opportunities for large-scale rollout that is not heavily blocked by regulation...[Long list of examples]
[Yudkowsky]
so… when I imagine trying to deploy this style of thought myself to predict the recent past without benefit of hindsight, it returns a lot of errors. perhaps this is because I do not know how to use this style of thought...
...”There are many possible regulatory regimes in the world, some of which would permit rapid construction of mRNA-vaccine factories well in advance of FDA approval. Given the overall urgency of the pandemic some of those extra-USA vaccines would be sold to individuals or a few countries like Israel willing to pay high prices for them, which would provide evidence of efficacy and break the usual impulse towards regulatory uniformity among developed countries...”
On Carl’s view, it sure seems like you’d just say something like “Healthcare is very overregulated, there will be an unusually strong effort anyway in lots of countries because Covid is an emergency, so it’ll be faster by some hard to predict amount but still bottlenecked by regulatory pressures.” And indeed the fastest countries got there in ~10 months instead of the multiple years predicted by superforecasters, or the ~3 months it would have taken with immediate approval.
The obvious object-level difference between Eliezer ‘applying’ Carl’s view to retrodict covid vaccine rollout and Carl’s prediction about AI is that Carl is saying there’s an enormous number of potential applications of intermediately general AI tech, and many of them aren’t blocked by regulation, while Eliezer’s attempted operating of Carl’s view for covid vaccines is saying “There are many chances for countries with lots of regulatory barriers to do the smart thing”.
The vaccine example is a different argument than AI predictions, because what Carl is saying is that there are many completely open goals for improvement like automating factories and call centres etc. not that there are many opportunities to avoid the regulatory barriers that will block everything by default.
But it seems like Eliezer is making a more outside view appeal, i.e. approach stories where big innovations are used wisely with a lot of scepticism because of our past record, even if you can tell a story about why it will be quite different this time.
Compare this,
to this,
Eliezer does (at least weakly) expect more trend breaks before The End (even on metrics that aren’t qualitative measures of impressiveness/intelligence, but just things like model loss functions), despite the fact that Rohin’s summary of his view is (I think) roughly accurate.
What explains this? I think it’s something roughly like, part of the reason Eliezer expects a sudden transition when we reach the core of generality in the first place is because he thinks that’s how things usually go in the history of tech/AI progress—there’s also specific reasons to think it will happen in the case of finding the core of generality, but there are also general reasons. See e.g. this from Eliezer:
I take ‘more weirdness’ to mean something like more discoveries that induce sudden improvements out there in general.
So I think that’s why his view does make (weaker) differential predictions about earlier events that we can test, not because the zero-to-one core of generality hypothesis predicts anything about narrow AI progress, but because some of the beliefs that led to that hypothesis do.
We can see there’s two (connected) lines of argument and that Eliezer and Paul/Carl/Richard have different things to say on each − 1 is more localized and about seeing what we can learn about AGI specifically, and 2 is about reference class reasoning and what tech progress in general tells us about AGI:
Specific to AGI: What can we infer from human evolution and interrogating our understanding of general intelligence (?) about whether AGI will arrive suddenly?
Reference Class: What can we infer about AGI progress from the general record of technological progress, especially how common big impacts are when there’s lots of effort and investment?
My sense is that Eliezer answers
Big update for Eliezer’s view: This tells us a lot, in particular we learn evolution got to the core of generality quickly, so AI progress will probably get there quickly as well. Plus, Humans are an existence proof for the core of generality, which suggests our default expectation should be sudden progress when we hit the core.
Smaller update for Eliezer’s view: This isn’t that important—there’s no necessary connection between AGI and e.g. bridges or nukes. But, you can at least see that there’s not a strong consistent track record of continuous improvement once you understand the historical record the right way (plus the underlying assumption in 2 that there will be a lot of effort and investment is probably wrong as well). Nonetheless, if you avoid retrospective trend-fitting and look at progress in the most natural (qualitative?) way, you’ll see that early discoveries that go from 0 to 1 are all over the place—Bitcoin, the Wright flyer, nuclear weapons are at least not crazy exceptions and quite possibly the default.
While Paul and Carl(?) answer,
Smaller update for Paul’s view: The disanalogies between AI progress and Evolution all point in the direction of AI progress being smoother than evolution (we’re intelligently trying to find the capabilities we want) - we get a weak update in favour of the smooth progress view from understanding this disanalogy between AI progress and evolution, but really we don’t learn much, except that there aren’t any good reasons to think there are only a few paths to a large set of powerful world affecting capabilities. Also, the core of generality idea is wrong, so the idea that Humans are an existence proof for it or that evolution tells us something about how to find it is wrong.
Big update for Paul’s view: reasoning from the reference class of ‘technologies where there are many opportunities for improvement and many people trying different things at once’ lets us see why expecting smooth progress should be the default. It’s because as long as there are lots of paths to improvement in the underlying capability landscape (which is the default because that’s how the world works by default), and there are lots of people trying to make improvements in different ways, the incremental changes add up to smooth outputs.
So Eliezer’s claim that Paul et al’s trend-fitting must include
is sort of correct, but the model isn’t really sophisticated or wordless.
The model is: as long as the underlying ‘capability landscape’ offers many paths to improvements, not just a few really narrow ones that swamp everything else, lots of people intelligently trying lots of different approaches will lead to lots of small discoveries that add up. Additionally, most examples of tech progress look like ‘multiple ways of doing something that add up’, this is confirmed by the historical record.
And then the model of cognitive scaling consists of (among other things) specific counterarguments to the claim that AGI progress is one of those cases with a few big paths to improvement (e.g. Evolution doesn’t give us evidence that AGI progress will be sudden).
On Carl’s view, it sure seems like you’d just say something like “Healthcare is very overregulated, there will be an unusually strong effort anyway in lots of countries because Covid is an emergency, so it’ll be faster by some hard to predict amount but still bottlenecked by regulatory pressures.” And indeed the fastest countries got there in ~10 months instead of the multiple years predicted by superforecasters, or the ~3 months it would have taken with immediate approval.
The obvious object-level difference between Eliezer ‘applying’ Carl’s view to retrodict covid vaccine rollout and Carl’s prediction about AI is that Carl is saying there’s an enormous number of potential applications of intermediately general AI tech, and many of them aren’t blocked by regulation, while Eliezer’s attempted operating of Carl’s view for covid vaccines is saying “There are many chances for countries with lots of regulatory barriers to do the smart thing”.
The vaccine example is a different argument than AI predictions, because what Carl is saying is that there are many completely open goals for improvement like automating factories and call centres etc. not that there are many opportunities to avoid the regulatory barriers that will block everything by default.
But it seems like Eliezer is making a more outside view appeal, i.e. approach stories where big innovations are used wisely with a lot of scepticism because of our past record, even if you can tell a story about why it will be quite different this time.