I think we’re on the same page about what factors exist.
For the intelligence explosion I think all we need is experience with AI RnD?
Well, sort of, but precisely because intelligence isn’t single-dimensional, we have to ask ‘what is exploding here?‘. And like I said, I think you straightforwardly get speed. And plausibly you get sample efficiency (in the inference-from-given-data sense). And maybe you get ‘exploratory planning’, which is hypothetically one, more transferrable, factor of general exploration efficiency.
But you don’t get domain-specific novelty/interestingness taste, which is another critical factor, except for the domains you can easily get lots of data on. So either you need to hoover that up from existing data, or get it some other how. That might be interviews, it might be a bunch of robotics and other actuator/sensor equipment, it might be the Shulman-style humans-with-headsets-as-actuators thing and learning by experience. But then the question becomes whether there’s enough data to hoover it up real fast with your speed+efficiency learning, or whether you hit a bunch of scaling up bottlenecks (and in which domains).
BTW I also think novelty taste depreciates quickly as the frontier of a domain moves, so I’m less bullish on hoovering up taste from existing data/interviews for a permanent taste boost. But it might take you some way past frontier, which may or may not be sizeably impactful in a given domain.
Yes more ‘data’, but not necessarily more frontier-taste-relevant experience, which requires either direct or indirect contact with frontier experiments. That might be a crux.
I agree we should weight frontier-taste-relevant data more heavily, but I think my point still goes through.
With that weighting, human experts have much less frontier-taste-relevant data than they have total data, and it’s still true that AIs could acquire more of that data than an expert in <1 year.
As AI advances SOTA technology it will in general have more data than humans on the frontier, as AI can learn from all copies. E.g. if it takes 1000 experiments to advance the frontier one step, human experts might typically experience 10 experiments at each step, whereas AI will experience 1000.
With that weighting, human experts have much less frontier-taste-relevant data than they have total data… AI can learn from all copies.
Ah, yep yep. (Though humans do in fact learn (with losses) from other copies, and within a given firm/lab/community the transfer is probably quite high.)
Hmm, I think we have basically the same model, with maybe a bit different parameters (about which we’re both somewhat uncertain). But think that readers of the OP without some of that common model might be misled.
From a lot of convos and writing, I infer that many people conflate a lot of aspects of intelligence into one thing. With that view, ‘intelligence explosion’ is just a dynamic where a single variable, ‘the intelligence’ (or ‘the algorithmic quality’), gets real big real fast. And then of course, because intelligence is the thing that gets you new technology, you can get all the new technology.
About this, you said,
There is then a further question, if that IE goes very far, of whether AI will generalize to chemistry etc
I’d have thought “yes” bc you achieve somewhat superhuman sample efficiency and can quickly get as much chemistry exp as a human expert
revealing that you correctly distinguish different factors of intelligence like ‘sample efficiency’ and ‘chemistry knowledge’ (I think I already knew this but it’s good to have local confirmation), and that you don’t think a software-only IE yields all of them.
Regarding the second sentence, it could be a misleading use of terms to call that ‘generalisation’[1], but I agree that ‘sample efficiency’ is among the relevant aspects of intelligence (and is a candidate for one that could be mostly generalisably built up automated in silico), and a relevant complement is ‘chemistry (frontier research) experience’, and that a lot of each taken together may effectively get you chemistry research taste in addition (which can yield new chemistry knowledge if applied with suitable experimental apparatus).
I’m emphasising[2] (in my exploration post, in this thread, and the sister comment about frontier taste depreciation) that there’s practically a wide gulf between ‘hoover taste up from web data’ and ‘robotics or humans-with-headsets’, in two ways. The first tops out somewhere (probably at sub-frontier) due to the depreciation of frontier research taste. The second cluster doesn’t top out anywhere, but is slower and has more barriers to getting started. Is a year to exceed humanity’s peak taste bold in most domains? Not sure! If a lot of in silico is possible, maybe it’s doable. That might include cyber and software (and maybe narrow particular areas of chem/bio where simulation is especially good).
If you know for sure that those other bottlenecks proceed super fast, you don’t need to necessarily clarify what intelligence factors you’re talking about for practical purposes, but if you’re not sure, I think it’s worth being super clear about it where possible.
Incidentally, the other thing I’m emphasising (but I think you’d agree?) is that on this view, R&Ds are always substantially driven by experimental throughput, with ‘sample efficiency (of the combined workforce) at accruing research taste’ being the main other rate-determining factor (because the steady state of research taste depends on this, and exploration quality * experimental throughput is progress). Throwing more labour at it can make your serial experimentation a bit faster, and can parallelise experimentation (with some parallelism discount), with presumably very diminishing returns. Throwing smarter labour (as in, better sample efficiency, and maybe faster-thinking, with diminishing returns), can increase the rate, by getting more insight per experiment and choosing better experiments.
For the intelligence explosion I think all we need is experience with AI RnD?
There is then a further question, if that IE goes very far, of whether AI will generalize to chemistry etc
I’d have thought “yes” bc you achieve somewhat superhuman sample efficiency and can quickly get as much chemistry exp as a human expert
I think we’re on the same page about what factors exist.
Well, sort of, but precisely because intelligence isn’t single-dimensional, we have to ask ‘what is exploding here?‘. And like I said, I think you straightforwardly get speed. And plausibly you get sample efficiency (in the inference-from-given-data sense). And maybe you get ‘exploratory planning’, which is hypothetically one, more transferrable, factor of general exploration efficiency.
But you don’t get domain-specific novelty/interestingness taste, which is another critical factor, except for the domains you can easily get lots of data on. So either you need to hoover that up from existing data, or get it some other how. That might be interviews, it might be a bunch of robotics and other actuator/sensor equipment, it might be the Shulman-style humans-with-headsets-as-actuators thing and learning by experience. But then the question becomes whether there’s enough data to hoover it up real fast with your speed+efficiency learning, or whether you hit a bunch of scaling up bottlenecks (and in which domains).
BTW I also think novelty taste depreciates quickly as the frontier of a domain moves, so I’m less bullish on hoovering up taste from existing data/interviews for a permanent taste boost. But it might take you some way past frontier, which may or may not be sizeably impactful in a given domain.
I think the methods you describe for gaining data suffice to get more data in <1 year than a human expert sees in a lifetime
Which is why I don’t expect big delays from this
But I agree this stuff will be a bottleneck
Yes more ‘data’, but not necessarily more frontier-taste-relevant experience, which requires either direct or indirect contact with frontier experiments. That might be a crux.
I agree we should weight frontier-taste-relevant data more heavily, but I think my point still goes through.
With that weighting, human experts have much less frontier-taste-relevant data than they have total data, and it’s still true that AIs could acquire more of that data than an expert in <1 year.
As AI advances SOTA technology it will in general have more data than humans on the frontier, as AI can learn from all copies. E.g. if it takes 1000 experiments to advance the frontier one step, human experts might typically experience 10 experiments at each step, whereas AI will experience 1000.
Ah, yep yep. (Though humans do in fact learn (with losses) from other copies, and within a given firm/lab/community the transfer is probably quite high.)
Hmm, I think we have basically the same model, with maybe a bit different parameters (about which we’re both somewhat uncertain). But think that readers of the OP without some of that common model might be misled.
From a lot of convos and writing, I infer that many people conflate a lot of aspects of intelligence into one thing. With that view, ‘intelligence explosion’ is just a dynamic where a single variable, ‘the intelligence’ (or ‘the algorithmic quality’), gets real big real fast. And then of course, because intelligence is the thing that gets you new technology, you can get all the new technology.
About this, you said,
revealing that you correctly distinguish different factors of intelligence like ‘sample efficiency’ and ‘chemistry knowledge’ (I think I already knew this but it’s good to have local confirmation), and that you don’t think a software-only IE yields all of them.
Regarding the second sentence, it could be a misleading use of terms to call that ‘generalisation’[1], but I agree that ‘sample efficiency’ is among the relevant aspects of intelligence (and is a candidate for one that could be mostly generalisably built up automated in silico), and a relevant complement is ‘chemistry (frontier research) experience’, and that a lot of each taken together may effectively get you chemistry research taste in addition (which can yield new chemistry knowledge if applied with suitable experimental apparatus).
I’m emphasising[2] (in my exploration post, in this thread, and the sister comment about frontier taste depreciation) that there’s practically a wide gulf between ‘hoover taste up from web data’ and ‘robotics or humans-with-headsets’, in two ways. The first tops out somewhere (probably at sub-frontier) due to the depreciation of frontier research taste. The second cluster doesn’t top out anywhere, but is slower and has more barriers to getting started. Is a year to exceed humanity’s peak taste bold in most domains? Not sure! If a lot of in silico is possible, maybe it’s doable. That might include cyber and software (and maybe narrow particular areas of chem/bio where simulation is especially good).
If you know for sure that those other bottlenecks proceed super fast, you don’t need to necessarily clarify what intelligence factors you’re talking about for practical purposes, but if you’re not sure, I think it’s worth being super clear about it where possible.
I might instead prefer terminology like ‘quickly implies … (on assumptions...)’
Incidentally, the other thing I’m emphasising (but I think you’d agree?) is that on this view, R&Ds are always substantially driven by experimental throughput, with ‘sample efficiency (of the combined workforce) at accruing research taste’ being the main other rate-determining factor (because the steady state of research taste depends on this, and exploration quality * experimental throughput is progress). Throwing more labour at it can make your serial experimentation a bit faster, and can parallelise experimentation (with some parallelism discount), with presumably very diminishing returns. Throwing smarter labour (as in, better sample efficiency, and maybe faster-thinking, with diminishing returns), can increase the rate, by getting more insight per experiment and choosing better experiments.