Agent-3, having excellent knowledge of both the human brain and modern AI algorithms, as well as many thousands of copies doing research, ends up making substantial algorithmic strides, narrowing the gap to an agent that’s only around 4,000x less compute-efficient than the human brain
I recognize that this is not the main point of this document, but am I interpreting correctly that you anticipate that rapid recursive improvement in AI research / AI capabilities is cracked before sample efficiency is cracked (e.g. via active learning)?
If so, that does seem like a continuation of current trends, but the implications seem pretty wild. e.g.
Most meme-worthy: We’ll get the discount sci-fi future where humanoid robots become commonplace, not because the human form is optimal, but because it lets AI systems piggyback off human imitation for physical tasks even when that form is wildly suboptimal for the job
Human labor will likely become more valuable relative to raw materials, not less (as long as most humans are more sample efficient than the best AI). In a world where all repetitive, structured tasks can be automated, humans will be prized specifically for handling novel one-off tasks that remain abundant in the physical world
Repair technicians and debuggers of physical and software systems become worth their weight in gold. The ability to say “This situation reminds me of something I encountered two years ago in Minneapolis” becomes humanity’s core value proposition
Large portions of the built environment begin resembling Amazon warehouses—robot restricted areas and corridors specifically designed to minimize surprising scenarios, with humans stationed around the perimeter for exception handling
We accelerate toward living in a panopticon, not primarily for surveillance, but because ubiquitous observation provides the massive datasets needed for AI training pipelines
Still, I feel like I have to be misinterpreting what you mean by “4,000x less sample efficient” here, because passages like the following don’t make sense under that interpretation
> The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.
Those implications are only correct if we remain at subhuman data-efficiency for an extended period. In AI 2027 the AIs reach superhuman data-efficiency by roughly the end of 2027 (it’s part of the package of being superintelligent) so there isn’t enough time for the implications you describe to happen. Basically in our story, the intelligence explosion gets started in early 2027 with very data-inefficient AIs, but then it reaches superintelligence by the end of the year, solving data-efficiency along the way.
In that case, “2027-level AGI agents are not yet data efficient but are capable of designing successors that solve the data efficiency bottleneck despite that limitation” seems pretty cruxy.
I probably want to bet against that. I will spend some time this weekend contemplating how that could be operationalized, and particularly trying to think of something where we could get evidence before 2027.
That excerpt says “compute-efficient” but the rest of your comment switches to “sample efficient”, which is not synonymous, right? Am I missing some context?
Nope, I just misread. Over on ACX I saw that Scott had left a comment
Our scenario’s changes are partly due to change in intelligence, but also partly to change in agency/time horizon/planning, and partly serial speed. Data efficiency comes later, downstream of the intelligence explosion.
I hadn’t remembered reading that in the post Still “things get crazy before models get data-efficient” does sound like the sort of thing which could plausibly fit with the world model in the post (but would be understated if so). Then I re-skimmed the post, and in the October 2027 section I saw
The gap between human and AI learning efficiency is rapidly decreasing.
Agent-3, having excellent knowledge of both the human brain and modern AI algorithms, as well as many thousands of copies doing research, ends up making substantial algorithmic strides, narrowing the gap to an agent that’s only around 4,000x less compute-efficient than the human brain
and when I read that my brain silently did a s/compute-efficient/data-efficient.
Though now I am curious about the authors’ views on how data efficiency will advance over the next 5 years, because that seems very world-model-relevant.
We are indeed imagining that they begin 2027 only about as data-efficient as they are today, but then improve significantly over the course of 2027 reaching superhuman data-efficiency by the end. We originally were going to write “data-efficiency” in that footnote but had trouble deciding on a good definition of it, so we went with compute-efficiency instead.
I recognize that this is not the main point of this document, but am I interpreting correctly that you anticipate that rapid recursive improvement in AI research / AI capabilities is cracked before sample efficiency is cracked (e.g. via active learning)?
If so, that does seem like a continuation of current trends, but the implications seem pretty wild. e.g.
Most meme-worthy: We’ll get the discount sci-fi future where humanoid robots become commonplace, not because the human form is optimal, but because it lets AI systems piggyback off human imitation for physical tasks even when that form is wildly suboptimal for the job
Human labor will likely become more valuable relative to raw materials, not less (as long as most humans are more sample efficient than the best AI). In a world where all repetitive, structured tasks can be automated, humans will be prized specifically for handling novel one-off tasks that remain abundant in the physical world
Repair technicians and debuggers of physical and software systems become worth their weight in gold. The ability to say “This situation reminds me of something I encountered two years ago in Minneapolis” becomes humanity’s core value proposition
Large portions of the built environment begin resembling Amazon warehouses—robot restricted areas and corridors specifically designed to minimize surprising scenarios, with humans stationed around the perimeter for exception handling
We accelerate toward living in a panopticon, not primarily for surveillance, but because ubiquitous observation provides the massive datasets needed for AI training pipelines
Still, I feel like I have to be misinterpreting what you mean by “4,000x less sample efficient” here, because passages like the following don’t make sense under that interpretation
> The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.
Those implications are only correct if we remain at subhuman data-efficiency for an extended period. In AI 2027 the AIs reach superhuman data-efficiency by roughly the end of 2027 (it’s part of the package of being superintelligent) so there isn’t enough time for the implications you describe to happen. Basically in our story, the intelligence explosion gets started in early 2027 with very data-inefficient AIs, but then it reaches superintelligence by the end of the year, solving data-efficiency along the way.
In that case, “2027-level AGI agents are not yet data efficient but are capable of designing successors that solve the data efficiency bottleneck despite that limitation” seems pretty cruxy.
I probably want to bet against that. I will spend some time this weekend contemplating how that could be operationalized, and particularly trying to think of something where we could get evidence before 2027.
That excerpt says “compute-efficient” but the rest of your comment switches to “sample efficient”, which is not synonymous, right? Am I missing some context?
Nope, I just misread. Over on ACX I saw that Scott had left a comment
I hadn’t remembered reading that in the post Still “things get crazy before models get data-efficient” does sound like the sort of thing which could plausibly fit with the world model in the post (but would be understated if so). Then I re-skimmed the post, and in the October 2027 section I saw
and when I read that my brain silently did a
s/compute-efficient/data-efficient.Though now I am curious about the authors’ views on how data efficiency will advance over the next 5 years, because that seems very world-model-relevant.
We are indeed imagining that they begin 2027 only about as data-efficient as they are today, but then improve significantly over the course of 2027 reaching superhuman data-efficiency by the end. We originally were going to write “data-efficiency” in that footnote but had trouble deciding on a good definition of it, so we went with compute-efficiency instead.