Re 1a: Intuitively what I mean by “lots of data” is something comparable in size to what ChatGPT was trained on (e.g. the Common Crawl, in the roughly 1 petabyte range); or rather, not just comparable in disk-space-usage, but in the number of distinct events to which reinforcement learning is applied. So when ChatGPT is being trained, each token (of which there are a ~quadrillion) is a chance to test the model’s predictions and adjust the model accordingly. (Incidentally, the fact that humans are able to learn language with far less data input than this suggests that there’s something fundamentally different in the way LLMs vs. humans work.)
Therefore, for a similarly-architected AI that generates action plans (rather than text tokens), we’d expect it to require a training set with a ~quadrillion different historical cases. Now I’m pretty sure this already exceeds the amount of “stuff happening” that has ever been documented in all of history.
I would change my opinion on this if it turns out that AI advancement is making it possible to achieve the same predictive accuracy / generative quality with ever less training data, in a way that doesn’t seem to be levelling off soon. (Has work been done on this?)
Re 2a: Accordingly, the reference class for the “experiments” that need to be performed here is not like “growing cells in a petri dish overnight”, but rather more like “run a company according to this business plan for 3 months and see how much money you make.” And at the end you’ll get one data point—just 999,999,999,999,999 to go...
Re 2b:
What do you see as the upper bound for what can, in principle, be done with a plan that an army of IQ-180 humans (aka no better qualitative thinking than what the smartest humans can do, so that this is a strict lower bound on ASI capabilities) came up with over subjective millennia with access to all recorded information that currently exists in the world?
I’ll grant that such an army could do some pretty impressive stuff. But this is already presupposing the existence of the superintelligence whose feasibility we are trying to explain.
Re 3c/d:
I haven’t looked into this or tried it myself. (I mostly use LLMs for informational purpose, not for planning actions). Do you have any examples handy of AI being successful at real-world goals?
(I may add some thoughts on your other points later, but I didn’t want to hold up my reply on that account.)
Stepping back, I should reiterate that I’m talking about “the current AI paradigm”, i.e. “deep neural networks + big data + gradient descent”, and not the capabilities of any hypothetical superintelligent AI that may exist in the future. Maybe this is conceding too much, inasmuch as addressing just one specific kind of architecture doesn’t do much to alleviate fear of doom by other means. But IABIED leans heavily on this paradigm in making its case for concern:
the claim that AIs are “grown, not crafted”
the claim that AIs will develop desires (or desire-like behavior) via gradient descent
the claim that the risk is imminent because superintelligence is the inevitable endpoint of the work that AI researchers are currently doing, and because no new fundamental breakthroughs stand in the way of that outcome.
But this is already presupposing the existence of the superintelligence whose feasibility we are trying to explain.
Strictly speaking I only presupposed an AI could reach close to the limits of human intelligence in terms of thinking ability, but with the inherent speed and parallelizability and memory advantages of a digital mind.
Do you have any examples handy of AI being successful at real-world goals?
In small ways (aka sized appropriately for current AI capabilities) this kind of thing shows up all the time in chains of thought in response to all kinds of prompts, to the point that no, I don’t have specific examples, because I wouldn’t know how to pick one. The one that first comes to mind, I guess, was using AI to help me develop a personalized nutrition/supplement/weight loss/training regimen.
Stepping back, I should reiterate that I’m talking about “the current AI paradigm”
That’s fair, and a reasonable thing to discuss. After all, the fundamental claim of the book’s title is about a conditional probability: IF it turns out that the anything like our current methods scale to superintelligent agents, we’d all be screwed.
Re 1a: Intuitively what I mean by “lots of data” is something comparable in size to what ChatGPT was trained on (e.g. the Common Crawl, in the roughly 1 petabyte range); or rather, not just comparable in disk-space-usage, but in the number of distinct events to which reinforcement learning is applied. So when ChatGPT is being trained, each token (of which there are a ~quadrillion) is a chance to test the model’s predictions and adjust the model accordingly. (Incidentally, the fact that humans are able to learn language with far less data input than this suggests that there’s something fundamentally different in the way LLMs vs. humans work.)
Therefore, for a similarly-architected AI that generates action plans (rather than text tokens), we’d expect it to require a training set with a ~quadrillion different historical cases. Now I’m pretty sure this already exceeds the amount of “stuff happening” that has ever been documented in all of history.
I would change my opinion on this if it turns out that AI advancement is making it possible to achieve the same predictive accuracy / generative quality with ever less training data, in a way that doesn’t seem to be levelling off soon. (Has work been done on this?)
Re 2a: Accordingly, the reference class for the “experiments” that need to be performed here is not like “growing cells in a petri dish overnight”, but rather more like “run a company according to this business plan for 3 months and see how much money you make.” And at the end you’ll get one data point—just 999,999,999,999,999 to go...
Re 2b:
I’ll grant that such an army could do some pretty impressive stuff. But this is already presupposing the existence of the superintelligence whose feasibility we are trying to explain.
Re 3c/d:
I haven’t looked into this or tried it myself. (I mostly use LLMs for informational purpose, not for planning actions). Do you have any examples handy of AI being successful at real-world goals?
(I may add some thoughts on your other points later, but I didn’t want to hold up my reply on that account.)
Stepping back, I should reiterate that I’m talking about “the current AI paradigm”, i.e. “deep neural networks + big data + gradient descent”, and not the capabilities of any hypothetical superintelligent AI that may exist in the future. Maybe this is conceding too much, inasmuch as addressing just one specific kind of architecture doesn’t do much to alleviate fear of doom by other means. But IABIED leans heavily on this paradigm in making its case for concern:
the claim that AIs are “grown, not crafted”
the claim that AIs will develop desires (or desire-like behavior) via gradient descent
the claim that the risk is imminent because superintelligence is the inevitable endpoint of the work that AI researchers are currently doing, and because no new fundamental breakthroughs stand in the way of that outcome.
Strictly speaking I only presupposed an AI could reach close to the limits of human intelligence in terms of thinking ability, but with the inherent speed and parallelizability and memory advantages of a digital mind.
In small ways (aka sized appropriately for current AI capabilities) this kind of thing shows up all the time in chains of thought in response to all kinds of prompts, to the point that no, I don’t have specific examples, because I wouldn’t know how to pick one. The one that first comes to mind, I guess, was using AI to help me develop a personalized nutrition/supplement/weight loss/training regimen.
That’s fair, and a reasonable thing to discuss. After all, the fundamental claim of the book’s title is about a conditional probability: IF it turns out that the anything like our current methods scale to superintelligent agents, we’d all be screwed.