Maybe the problem is that we don’t have a good metaphor for what the path for “rapidly shooting past human-level capability” is like in a general sense, rather than on a specific domain.
One domain-specific metaphor you mention is AlphaZero, but games like chess are an unusual domain of learning for the AI, because it doesn’t need any external input beyond the rules of the game and objective, and RL can proceed just by the program playing against itself. It’s not clear to me how we can generalize the AlphaZero learning curve to problems that are not self-contained games like that, where the limiting factor may not be computing power or memory, but just the availability (and rate of acquisition) of good data to do RL on.
In §1.8.1 I also mentioned going from zero to beyond-world-expert-level understanding of cryptocurrency over the course of 24 hours spent reading up on the topic and all its prerequisites, and playing with the code, etc.
And in §3.2 here I talked about a hypothetical AI that, by itself, is running the equivalent of the entire global human R&D enterprise, i.e. the AI is running hundreds of thousands of laboratory experiments simultaneously, 24 hours a day, publishing millions of papers a year, and synthesizing brilliant new insights in every field at once.
Can we agree that those examples would be “rapidly shooting past human-level capability”, and would not violate any laws of probability by magically acquiring knowledge that it has no way to know?
With regards to the super-scientist AI (the global human R&D equivalent), wouldn’t we see it coming based on the amount of resources it would need to hire? Are you claiming that it could reach the required AGI capacity in its “brain in a box in a basement” state and only after scale up in terms of resource use? The part I’m most skeptical about remains this idea that the resource use to get to human-level performance is minimal if you just find the right algorithm, because at least in my view it neglects the evaluation step in learning that can be resource intensive from the start and maybe can’t be done “covertly”.
---
That said, I want to stress that I agree with the conclusion:
So we need to be working frantically on technical alignment, sandbox test protocols, and more generally having a plan, right now, long before the future scary paradigm seems obviously on the path to AGI.
(And no, inventing that next AI paradigm is not part of the solution, but rather part of the problem, despite the safety-vibed rhetoric of the researchers who are doing exactly that as we speak—see §1.6.1.)
But then, if AI researchers believe a likely scenario is:
the development of strong superintelligence from a small group working on a new AI paradigm, with essentially no warning and little resources,
Does that imply that the people who work on technical alignment, or at least their allies, need to also put effort to “win the race” for AGI? It seems the idea that “any small group could create this with no warning” could motivate acceleration in that race even from people who are well-meaning in terms of alignment.
Maybe the problem is that we don’t have a good metaphor for what the path for “rapidly shooting past human-level capability” is like in a general sense, rather than on a specific domain.
One domain-specific metaphor you mention is AlphaZero, but games like chess are an unusual domain of learning for the AI, because it doesn’t need any external input beyond the rules of the game and objective, and RL can proceed just by the program playing against itself. It’s not clear to me how we can generalize the AlphaZero learning curve to problems that are not self-contained games like that, where the limiting factor may not be computing power or memory, but just the availability (and rate of acquisition) of good data to do RL on.
In §1.8.1 I also mentioned going from zero to beyond-world-expert-level understanding of cryptocurrency over the course of 24 hours spent reading up on the topic and all its prerequisites, and playing with the code, etc.
And in §3.2 here I talked about a hypothetical AI that, by itself, is running the equivalent of the entire global human R&D enterprise, i.e. the AI is running hundreds of thousands of laboratory experiments simultaneously, 24 hours a day, publishing millions of papers a year, and synthesizing brilliant new insights in every field at once.
Can we agree that those examples would be “rapidly shooting past human-level capability”, and would not violate any laws of probability by magically acquiring knowledge that it has no way to know?
With regards to the super-scientist AI (the global human R&D equivalent), wouldn’t we see it coming based on the amount of resources it would need to hire? Are you claiming that it could reach the required AGI capacity in its “brain in a box in a basement” state and only after scale up in terms of resource use? The part I’m most skeptical about remains this idea that the resource use to get to human-level performance is minimal if you just find the right algorithm, because at least in my view it neglects the evaluation step in learning that can be resource intensive from the start and maybe can’t be done “covertly”.
---
That said, I want to stress that I agree with the conclusion:
But then, if AI researchers believe a likely scenario is:
Does that imply that the people who work on technical alignment, or at least their allies, need to also put effort to “win the race” for AGI? It seems the idea that “any small group could create this with no warning” could motivate acceleration in that race even from people who are well-meaning in terms of alignment.