When an intelligence builds another intelligence, in a single direct step, the output intelligence o is a function of the input intelligence i, and the resources used r. o(i,r). This function is clearly increasing in both i and r. Set r to be a reasonably large level of resources, eg 1030flops, 20 years to think about it. A low input intelligence, eg a dog, would be unable to make something smarter than itself. o(id,r)<id. A team of experts (by assumption that ASI is made), can make something smarter than themselves. o(ie,r)>ie. So there must be a fixed point. o(if,r)=f. The questions then become, how powerful is a pre fixed point AI. Clearly less good at AI research than a team of experts. As there is no reason to think that AI research is uniquely hard to AI, and there are some reasons to think it might be easier, or more prioritized, if it can’t beat our AI researchers, it can’t beat our other researchers. It is unlikely to make any major science or technology breakthroughs.
I recon that ∂o/∂i(if,r) is large (>10) because on an absolute scale, the difference between an IQ 90 and an IQ120 human is quite small, but I would expect any attempt at AI made by the latter to be much better. In a world where the limiting factor is researcher talent, not compute, the AI can get the compute it needs for r in hours (seconds? milliseconds??) As the lumpiness of innovation puts the first post fixed point AI a non-exponentially tiny distance ahead, (most innovations are at least 0.1% that state of the art better in a fast moving field) then a handful of cycles or recursive self improvement (<1 day) is enough to get the AI into the seriously overpowered range.
The question of economic doubling times would depend on how fast an economy can grow when tech breakthroughs are limited by human researchers. If we happen to have cracked self replication at about this point, it could be very fast.
Humans are already capable of self-improvement. This argument would suggest that the smartest human (or the one who was best at self-improvement, if you prefer) should have undergone fast takeoff and become seriously overpowered, but this doesn’t seem to have happened.
In a world where the limiting factor is researcher talent, not compute
Compute is definitely a limiting factor currently. Why would that change?
Humans are not currently capable of self improvement in the understanding your our own source code sense. The “self improvement” section in bookstores doesn’t change the hardware or the operating system, it basically adds more data.
Of course talent and compute both make a difference, in the sense that ∂o/∂i>0 and ∂o/∂r>0. I was talking about the subset of worlds where research talent was by far the most important. ∂o/∂r<<∂o/∂i.
In a world where researchers have little idea what they are doing, and are running a new AI every hour hoping to stumble across something that works, the result holds.
In a world where research involves months thinking about maths, then a day writing code, then an hour running it, this result holds.
In a world where everyone knows the right algorithm, but it takes a lot of compute, so AI research consists of building custom hardware and super-computing clusters, this result fails.
Currently, we are somewhere in the middle. I don’t know which of these options future research will look like, although if its the first one, friendly AI seems unlikely.
In most of the scenarios where the first smarter than human AI, is orders of magnitude faster than a human, I would expect a hard takeoff. As we went from having no algorithms that could say (tell a cat from a dog) straight to having algorithms superhumanly fast at doing so, there was no algorithm that worked, but took supercomputer hours, this seems like a plausible assumption.
Humans are not currently capable of self improvement in the understanding your o. I was talking about the subset of worlds where research talent ense. The “self improvement” section in bookstores doesn’t change the hardware or the operating system, it basically adds more data.
I’m not sure I understand this. Are you claiming δoδr is not positive for humans?
In most of the scenarios where the first smarter than human AI, is orders of magnitude faster than a human, I would expect a hard takeoff.
This sounds like “conditioned on a hard takeoff, I expect a hard takeoff”. It’s not exactly saying that, since speed could be different from intelligence, but you need to argue for the premise too: nearly all of the arguments in the linked post could be applied to your premise as well.
In a world where researchers have little idea what they are doing, and are running a new AI every hour hoping to stumble across something that works, the result holds.
In a world where research involves months thinking about maths, then a day writing code, then an hour running it, this result holds.
Agreed on both counts, and again I think the arguments in the linked posts suggest that the premises are not true.
As we went from having no algorithms that could say (tell a cat from a dog) straight to having algorithms superhumanly fast at doing so, there was no algorithm that worked, but took supercomputer hours, this seems like a plausible assumption.
This seems false to me. At what point would you say that we had AI systems that could tell a cat from a dog? I don’t know the history of object recognition, but I would guess that depending on how you operationalize it, I think the answer could be anywhere between the 60s and “we still can’t do it”. (Though it’s also possible that people didn’t care about object recognition until the 21st century, and only did other types of computer vision in the 60s-90s. It’s quite strange that object recognition is an “interesting” task, given how little information you get from it.)
My claim at the start had a typo in it. I am claiming that you can’t make a human seriously superhuman with a good education. Much like you can’t get a chimp up to human level with lots of education and “self improvement”. Serious genetic modification is another story, but at that point, your building an AI out of protien.
It does depend where you draw the line, but the for a wide range of performance levels, we went from no algorithm at that level, to a fast algorithm at that level. You couldn’t get much better results just by throwing more compute at it.
I am claiming that you can’t make a human seriously superhuman with a good education.
Is the claim that δo/δr for humans goes down over time so that o eventually hits an asymptote? If so, why won’t that apply to AI?
Serious genetic modification is another story, but at that point, your building an AI out of protien.
But it seems quite relevant that we haven’t successfully done that yet.
You couldn’t get much better results just by throwing more compute at it.
Okay, so my new story for this argument is:
For every task T, there are bottlenecks that limit its performance, which could be compute, data, algorithms, etc.
For the task of “AI research”, compute will not be the bottleneck.
So, once we get human-level performance on “AI research”, we can apply more compute to get exponential recursive self-improvement.
Is that your argument? If so, I think my question would be “why didn’t the bottleneck in point 2 vanish in point 3?” I think the only way this would be true would be if the bottleneck was algorithms, and there was a discontinuous jump in the capability of algorithms. I agree that in that world you would see a hard/fast/discontinuous takeoff, but I don’t see why we should expect that (again, the arguments in the linked posts argue against that premise).
When an intelligence builds another intelligence, in a single direct step, the output intelligence o is a function of the input intelligence i, and the resources used r. o(i,r). This function is clearly increasing in both i and r. Set r to be a reasonably large level of resources, eg 1030flops, 20 years to think about it. A low input intelligence, eg a dog, would be unable to make something smarter than itself. o(id,r)<id. A team of experts (by assumption that ASI is made), can make something smarter than themselves. o(ie,r)>ie. So there must be a fixed point. o(if,r)=f. The questions then become, how powerful is a pre fixed point AI. Clearly less good at AI research than a team of experts. As there is no reason to think that AI research is uniquely hard to AI, and there are some reasons to think it might be easier, or more prioritized, if it can’t beat our AI researchers, it can’t beat our other researchers. It is unlikely to make any major science or technology breakthroughs.
I recon that ∂o/∂i(if,r) is large (>10) because on an absolute scale, the difference between an IQ 90 and an IQ120 human is quite small, but I would expect any attempt at AI made by the latter to be much better. In a world where the limiting factor is researcher talent, not compute, the AI can get the compute it needs for r in hours (seconds? milliseconds??) As the lumpiness of innovation puts the first post fixed point AI a non-exponentially tiny distance ahead, (most innovations are at least 0.1% that state of the art better in a fast moving field) then a handful of cycles or recursive self improvement (<1 day) is enough to get the AI into the seriously overpowered range.
The question of economic doubling times would depend on how fast an economy can grow when tech breakthroughs are limited by human researchers. If we happen to have cracked self replication at about this point, it could be very fast.
Humans are already capable of self-improvement. This argument would suggest that the smartest human (or the one who was best at self-improvement, if you prefer) should have undergone fast takeoff and become seriously overpowered, but this doesn’t seem to have happened.
Compute is definitely a limiting factor currently. Why would that change?
Humans are not currently capable of self improvement in the understanding your our own source code sense. The “self improvement” section in bookstores doesn’t change the hardware or the operating system, it basically adds more data.
Of course talent and compute both make a difference, in the sense that ∂o/∂i>0 and ∂o/∂r>0. I was talking about the subset of worlds where research talent was by far the most important. ∂o/∂r<<∂o/∂i.
In a world where researchers have little idea what they are doing, and are running a new AI every hour hoping to stumble across something that works, the result holds.
In a world where research involves months thinking about maths, then a day writing code, then an hour running it, this result holds.
In a world where everyone knows the right algorithm, but it takes a lot of compute, so AI research consists of building custom hardware and super-computing clusters, this result fails.
Currently, we are somewhere in the middle. I don’t know which of these options future research will look like, although if its the first one, friendly AI seems unlikely.
In most of the scenarios where the first smarter than human AI, is orders of magnitude faster than a human, I would expect a hard takeoff. As we went from having no algorithms that could say (tell a cat from a dog) straight to having algorithms superhumanly fast at doing so, there was no algorithm that worked, but took supercomputer hours, this seems like a plausible assumption.
I’m not sure I understand this. Are you claiming δoδr is not positive for humans?
This sounds like “conditioned on a hard takeoff, I expect a hard takeoff”. It’s not exactly saying that, since speed could be different from intelligence, but you need to argue for the premise too: nearly all of the arguments in the linked post could be applied to your premise as well.
Agreed on both counts, and again I think the arguments in the linked posts suggest that the premises are not true.
This seems false to me. At what point would you say that we had AI systems that could tell a cat from a dog? I don’t know the history of object recognition, but I would guess that depending on how you operationalize it, I think the answer could be anywhere between the 60s and “we still can’t do it”. (Though it’s also possible that people didn’t care about object recognition until the 21st century, and only did other types of computer vision in the 60s-90s. It’s quite strange that object recognition is an “interesting” task, given how little information you get from it.)
My claim at the start had a typo in it. I am claiming that you can’t make a human seriously superhuman with a good education. Much like you can’t get a chimp up to human level with lots of education and “self improvement”. Serious genetic modification is another story, but at that point, your building an AI out of protien.
It does depend where you draw the line, but the for a wide range of performance levels, we went from no algorithm at that level, to a fast algorithm at that level. You couldn’t get much better results just by throwing more compute at it.
Is the claim that δo/δr for humans goes down over time so that o eventually hits an asymptote? If so, why won’t that apply to AI?
But it seems quite relevant that we haven’t successfully done that yet.
Okay, so my new story for this argument is:
For every task T, there are bottlenecks that limit its performance, which could be compute, data, algorithms, etc.
For the task of “AI research”, compute will not be the bottleneck.
So, once we get human-level performance on “AI research”, we can apply more compute to get exponential recursive self-improvement.
Is that your argument? If so, I think my question would be “why didn’t the bottleneck in point 2 vanish in point 3?” I think the only way this would be true would be if the bottleneck was algorithms, and there was a discontinuous jump in the capability of algorithms. I agree that in that world you would see a hard/fast/discontinuous takeoff, but I don’t see why we should expect that (again, the arguments in the linked posts argue against that premise).