Consider the AI doomer position. They believe that AI models are fundamentally constrained by intelligence, and recursive self-improvement will enable AI models to infinitely improve themselves until they attain godlike levels of intelligence (and thus capability).
This is a misframing. AI does not need to infinitely improve themselves to godlike levels to be capable of out maneuvering human beings and cause human extinction. It’s one scenario, but people like Eliezer don’t think that it’s the only scenario.
So far, it seems like the majority of recent (~late 2024) AI gains came from inference-scaling—the amount of compute used every time a model answers a question—as opposed to training, regardless of whether that training is pre-training or post-training.
This thesis is basically: “The fact that we know have agents that can code well has little to do with the training data that comes out of AI deployment.” I’m not sure why it seems that way to you.
AlphaGo did improve by playing against itself. If you have a coding agent that gets feedback about the results of it’s coding work, that does give you training data on which you can train there’s a recursive element to it. The more agents get used in environments where you get feedback about the quality of the output, the more they can be recursively trained on those domains.
Re misframing: fair enough. Maybe I should have said “a popular AI doomer position”.
On the other thing: I’m not quite sure what you mean? My thesis in the quoted text was basically what I said: since most AI improvements have come from inference scaling, aka scaling up compute requirements, we can expect that future progress will also come from scaling up compute requirements. Obviously this only holds true until another paradigm shift happens.
Do you think agents will be trained on themselves in a similar fashion to AlphaGo, and do you think that training will reduce compute requirements / provide a performance increase driven by training instead of inference?
Re misframing: fair enough. Maybe I should have said “a popular AI doomer position”.
I don’t think that the belief that godlike intelligence is necessary for human extinction via AI is a popular AI doomer position among people who are intellectually sophisticated. It’s more like those people hold complex position and it’s easy for people who are skeptics to frame this as “a popular position”.
There’s an argument that some of the risk comes from “godlike intelligence” but that’s not necessary to believe in high risk. If you take an agent as smart as the smartest human and able to act faster while the agent can copy themselves after learning skills and able to potentially better coordinate among billions of copies of the agent that might be enough to overpower humanity.
You can’t conclude from the fact that inference scaling happened that most AI improvements are due to scaling.
Do you think agents will be trained on themselves in a similar fashion to AlphaGo
I’m saying that this is already happening. It’s not as straightforward as with AlphaGo as it’s easier to judge whether a move helps with winning a game in the constrained environment of go, but when it comes to coding you have quality measurements such as whether or not the agent managed to write code that successfully made the unit tests pass and
There’s a lot of training on ‘synthetic data’ and data from user interactions and if you have a better agents that leads to higher data quality for both.
When it comes to inference it’s also worth noting that they found a lot of tricks to make inference cheaper. It’s not just more/better hardware:
The cost of querying an AI model that scores the equivalent of GPT-3.5 (64.8) on MMLU, a popular benchmark for assessing language model performance, dropped from $20.00 per million tokens in November 2022 to just $0.07 per million tokens by October 2024 (Gemini-1.5-Flash-8B)—a more than 280-fold reduction in approximately 18 months.
I don’t think that the belief that godlike intelligence is necessary for human extinction via AI is a popular AI doomer position among people who are intellectually sophisticated. It’s more like those people hold complex position and it’s easy for people who are skeptics to frame this as “a popular position”.
Hang on, I don’t think I said that godlike intelligence was necessary for human extinction, and actually, didn’t make any claim about human extinction at all. This post was just about the possibility of an intelligence explosion, and I think “AI will reach godlike levels of intelligence” is an accurate description of the AI 2027 position.
You can’t conclude from the fact that inference scaling happened that most AI improvements are due to scaling.
Did you read the cited link that you quoted? Toby Ord’s argument was pretty convincing to me. What do you disagree with?
When it comes to inference it’s also worth noting that they found a lot of tricks to make inference cheaper. It’s not just more/better hardware
This is a misframing. AI does not need to infinitely improve themselves to godlike levels to be capable of out maneuvering human beings and cause human extinction. It’s one scenario, but people like Eliezer don’t think that it’s the only scenario.
This thesis is basically: “The fact that we know have agents that can code well has little to do with the training data that comes out of AI deployment.” I’m not sure why it seems that way to you.
AlphaGo did improve by playing against itself. If you have a coding agent that gets feedback about the results of it’s coding work, that does give you training data on which you can train there’s a recursive element to it. The more agents get used in environments where you get feedback about the quality of the output, the more they can be recursively trained on those domains.
Re misframing: fair enough. Maybe I should have said “a popular AI doomer position”.
On the other thing: I’m not quite sure what you mean? My thesis in the quoted text was basically what I said: since most AI improvements have come from inference scaling, aka scaling up compute requirements, we can expect that future progress will also come from scaling up compute requirements. Obviously this only holds true until another paradigm shift happens.
Do you think agents will be trained on themselves in a similar fashion to AlphaGo, and do you think that training will reduce compute requirements / provide a performance increase driven by training instead of inference?
I don’t think that the belief that godlike intelligence is necessary for human extinction via AI is a popular AI doomer position among people who are intellectually sophisticated. It’s more like those people hold complex position and it’s easy for people who are skeptics to frame this as “a popular position”.
There’s an argument that some of the risk comes from “godlike intelligence” but that’s not necessary to believe in high risk. If you take an agent as smart as the smartest human and able to act faster while the agent can copy themselves after learning skills and able to potentially better coordinate among billions of copies of the agent that might be enough to overpower humanity.
You can’t conclude from the fact that inference scaling happened that most AI improvements are due to scaling.
I’m saying that this is already happening. It’s not as straightforward as with AlphaGo as it’s easier to judge whether a move helps with winning a game in the constrained environment of go, but when it comes to coding you have quality measurements such as whether or not the agent managed to write code that successfully made the unit tests pass and
There’s a lot of training on ‘synthetic data’ and data from user interactions and if you have a better agents that leads to higher data quality for both.
When it comes to inference it’s also worth noting that they found a lot of tricks to make inference cheaper. It’s not just more/better hardware:
Hang on, I don’t think I said that godlike intelligence was necessary for human extinction, and actually, didn’t make any claim about human extinction at all. This post was just about the possibility of an intelligence explosion, and I think “AI will reach godlike levels of intelligence” is an accurate description of the AI 2027 position.
Did you read the cited link that you quoted? Toby Ord’s argument was pretty convincing to me. What do you disagree with?
Right, ending in about late 2024, which is why I specified (~late 2024) in “most recent gains”. It doesn’t seem like that trend has continued.