are you thinking about sub-human-level of AGIs? the standard definition of AGI involves it being it better than most humans in most of the tasks humans can do
the first human hackers were not trained on “take over my data center” either, but humans can behave out of distribution and so will the AGI that is better than humans at behaving out of distribution
the argument about AIs that generalize to many tasks but are not “actually dangerous yet” is about speeding up creation of the actually dangerous AGIs, and it’s the speeding up that is dangerous, not that AI Safety researchers believe that those “weak AGIs” created from large LLMs would actually be capable of killing everyone immediatelly on their own
if you believe “weak AGIs” won’t speed creation of “dangerous AGIs”, can you spell out why, please?
The above approach is similar to Gato and now Palm-E. I would define it as :
Subhuman AGI. General purpose machine that does not have the breadth and depth of the average human. Gato and Palm E are examples. At a minimum it must have vision, ability to read instructions, output text, and robotics control. (Audio or smell/taste I don’t think are necessary for a task performing AGI, though audio is easy and often supported)
AGI. Has the breadth/depth of the average human
ASI : soundly bets humans in MOST tasks. Or “low superintelligence”. It still has gaps and is throttled by (architecture, data, compute, or robotics access)
Post Singularity ASI: “high superintelligence”. Throttled by the laws of physics.
Note that for 3 and 4 I see no need to impose irrelevant goalposts. The machine needs the cognitive breadth and depth of a human or +++ at real world tool use, innovation and communication. It needs not be able to “actually” feel emotion or have a self modifying architecture for the 3 case. As a consequence there will remain tasks humans are better at, they just won’t be ones with measurable objectives.
I believe we can safely and fairly easily reach “low superintelligence” using variations on current approaches. (“easily” meaning straightforward engineering over several years and 100+ billion USD)
Thanks for sharing your point of view. I tried to give myself a few days, but I’m aftraid I still don’t understand where you see the magic barrier for the transition from 3 to 4 to happen outside of the realm of human control.
3 says the reason right there. Compute, data, or robotics/money.
What are you not able to understand with a few days of thought?
There is extremely strong evidence that compute is the limit right now. This is trivially correct : the current llms architectures are very similar to prior working attempts for the simple reason that one “try” to train to scale costs millions of dollars in compute. (And getting more money saturates, there is a finite number of training accelerators manufactured per quarter and it takes time to ramp to higher volumes)
To find something better, a hard superintelligence only capped by physics obviously requires many tries at exploring the possibility space. (Even intelligent search algorithms need many function evaluations)
yes, it takes millions to advance, but companies are pouring BILLIONS into this and number 3 can earn its own money and create its own companies/DAOs/some new networks of cooperation if it wanted without humans realizing … have you seen any GDP per year charts whatsoever, why would you think we are anywhere close to saturation of money? have you seen any emergent capabilities from LLMs in the last year, why do you think we are anywhere close to saturation of capabilities per million of dollars? Alpaca-like improvemnts are somehow one-off miracle and things are not getting cheaper and better and more efficient in the future somehow?
it could totally happen, but what I don’t see is why are you so sure it will happen by default, are you extrapolating some trend from non-public data or just overly optimistic that 1+1 from previous trends is less than 2 in the future, totally unlike the compount effects in AI advancement in the last year?
Because we are saturated right now and I gave evidence and you can read the gpt-4 paper for more evidence. See:
“getting more money saturates, there is a finite number of training accelerators manufactured per quarter and it takes time to ramp to higher volume”
“Billions” cannot buy more accelerators than exist, and the robot/compute/capabilities limits also limit the ROI that can be provided, which makes the billions not infinite as eventually investors get impatient.
What this means is that it may take 20 years or more of steady exponential growth (but only 10-50 percent annually) to reach ASI and self replicating factories and so on.
On a cosmic timescale or even a human lifespan this is extremely fast. I am noting this is more likely than “overnight” scenarios where someone tweaks a config file, an AI reaches high superintelligence and fills the earth with grey goo in days. There was not enough data in existence for the AI to reach high superintelligence, a “high” superintelligence would require thousands or millions of times as much training compute as GPT-4 (because it’s a power law), even once it’s trained it doesn’t have sufficient robotics to bootstrap to nanoforges without years or decades of steady ramping to be ready to do that.
(a high superintelligence is a machine that is not just a reasonable amount better than humans at all tasks but is essentially a deity outputting perfect moves on every task that take into account all of the machines plans and cross task and cross session knowledge.
So it might communicate with a lobbyist and 1e6 people at once and use information from all conversations in all conversations, essentially manipulating the world like a game of pool. Something genuinely uncontainable.)
are you thinking about sub-human-level of AGIs? the standard definition of AGI involves it being it better than most humans in most of the tasks humans can do
the first human hackers were not trained on “take over my data center” either, but humans can behave out of distribution and so will the AGI that is better than humans at behaving out of distribution
the argument about AIs that generalize to many tasks but are not “actually dangerous yet” is about speeding up creation of the actually dangerous AGIs, and it’s the speeding up that is dangerous, not that AI Safety researchers believe that those “weak AGIs” created from large LLMs would actually be capable of killing everyone immediatelly on their own
if you believe “weak AGIs” won’t speed creation of “dangerous AGIs”, can you spell out why, please?
The above approach is similar to Gato and now Palm-E. I would define it as :
Subhuman AGI. General purpose machine that does not have the breadth and depth of the average human. Gato and Palm E are examples. At a minimum it must have vision, ability to read instructions, output text, and robotics control. (Audio or smell/taste I don’t think are necessary for a task performing AGI, though audio is easy and often supported)
AGI. Has the breadth/depth of the average human
ASI : soundly bets humans in MOST tasks. Or “low superintelligence”. It still has gaps and is throttled by (architecture, data, compute, or robotics access)
Post Singularity ASI: “high superintelligence”. Throttled by the laws of physics.
Note that for 3 and 4 I see no need to impose irrelevant goalposts. The machine needs the cognitive breadth and depth of a human or +++ at real world tool use, innovation and communication. It needs not be able to “actually” feel emotion or have a self modifying architecture for the 3 case. As a consequence there will remain tasks humans are better at, they just won’t be ones with measurable objectives.
I believe we can safely and fairly easily reach “low superintelligence” using variations on current approaches. (“easily” meaning straightforward engineering over several years and 100+ billion USD)
Thanks for sharing your point of view. I tried to give myself a few days, but I’m aftraid I still don’t understand where you see the magic barrier for the transition from 3 to 4 to happen outside of the realm of human control.
3 says the reason right there. Compute, data, or robotics/money.
What are you not able to understand with a few days of thought?
There is extremely strong evidence that compute is the limit right now. This is trivially correct : the current llms architectures are very similar to prior working attempts for the simple reason that one “try” to train to scale costs millions of dollars in compute. (And getting more money saturates, there is a finite number of training accelerators manufactured per quarter and it takes time to ramp to higher volumes)
To find something better, a hard superintelligence only capped by physics obviously requires many tries at exploring the possibility space. (Even intelligent search algorithms need many function evaluations)
yes, it takes millions to advance, but companies are pouring BILLIONS into this and number 3 can earn its own money and create its own companies/DAOs/some new networks of cooperation if it wanted without humans realizing … have you seen any GDP per year charts whatsoever, why would you think we are anywhere close to saturation of money? have you seen any emergent capabilities from LLMs in the last year, why do you think we are anywhere close to saturation of capabilities per million of dollars? Alpaca-like improvemnts are somehow one-off miracle and things are not getting cheaper and better and more efficient in the future somehow?
it could totally happen, but what I don’t see is why are you so sure it will happen by default, are you extrapolating some trend from non-public data or just overly optimistic that 1+1 from previous trends is less than 2 in the future, totally unlike the compount effects in AI advancement in the last year?
Because we are saturated right now and I gave evidence and you can read the gpt-4 paper for more evidence. See:
“getting more money saturates, there is a finite number of training accelerators manufactured per quarter and it takes time to ramp to higher volume”
“Billions” cannot buy more accelerators than exist, and the robot/compute/capabilities limits also limit the ROI that can be provided, which makes the billions not infinite as eventually investors get impatient.
What this means is that it may take 20 years or more of steady exponential growth (but only 10-50 percent annually) to reach ASI and self replicating factories and so on.
On a cosmic timescale or even a human lifespan this is extremely fast. I am noting this is more likely than “overnight” scenarios where someone tweaks a config file, an AI reaches high superintelligence and fills the earth with grey goo in days. There was not enough data in existence for the AI to reach high superintelligence, a “high” superintelligence would require thousands or millions of times as much training compute as GPT-4 (because it’s a power law), even once it’s trained it doesn’t have sufficient robotics to bootstrap to nanoforges without years or decades of steady ramping to be ready to do that.
(a high superintelligence is a machine that is not just a reasonable amount better than humans at all tasks but is essentially a deity outputting perfect moves on every task that take into account all of the machines plans and cross task and cross session knowledge.
So it might communicate with a lobbyist and 1e6 people at once and use information from all conversations in all conversations, essentially manipulating the world like a game of pool. Something genuinely uncontainable.)