Re misframing: fair enough. Maybe I should have said “a popular AI doomer position”.
I don’t think that the belief that godlike intelligence is necessary for human extinction via AI is a popular AI doomer position among people who are intellectually sophisticated. It’s more like those people hold complex position and it’s easy for people who are skeptics to frame this as “a popular position”.
There’s an argument that some of the risk comes from “godlike intelligence” but that’s not necessary to believe in high risk. If you take an agent as smart as the smartest human and able to act faster while the agent can copy themselves after learning skills and able to potentially better coordinate among billions of copies of the agent that might be enough to overpower humanity.
You can’t conclude from the fact that inference scaling happened that most AI improvements are due to scaling.
Do you think agents will be trained on themselves in a similar fashion to AlphaGo
I’m saying that this is already happening. It’s not as straightforward as with AlphaGo as it’s easier to judge whether a move helps with winning a game in the constrained environment of go, but when it comes to coding you have quality measurements such as whether or not the agent managed to write code that successfully made the unit tests pass and
There’s a lot of training on ‘synthetic data’ and data from user interactions and if you have a better agents that leads to higher data quality for both.
When it comes to inference it’s also worth noting that they found a lot of tricks to make inference cheaper. It’s not just more/better hardware:
The cost of querying an AI model that scores the equivalent of GPT-3.5 (64.8) on MMLU, a popular benchmark for assessing language model performance, dropped from $20.00 per million tokens in November 2022 to just $0.07 per million tokens by October 2024 (Gemini-1.5-Flash-8B)—a more than 280-fold reduction in approximately 18 months.
I don’t think that the belief that godlike intelligence is necessary for human extinction via AI is a popular AI doomer position among people who are intellectually sophisticated. It’s more like those people hold complex position and it’s easy for people who are skeptics to frame this as “a popular position”.
Hang on, I don’t think I said that godlike intelligence was necessary for human extinction, and actually, didn’t make any claim about human extinction at all. This post was just about the possibility of an intelligence explosion, and I think “AI will reach godlike levels of intelligence” is an accurate description of the AI 2027 position.
You can’t conclude from the fact that inference scaling happened that most AI improvements are due to scaling.
Did you read the cited link that you quoted? Toby Ord’s argument was pretty convincing to me. What do you disagree with?
When it comes to inference it’s also worth noting that they found a lot of tricks to make inference cheaper. It’s not just more/better hardware
I don’t think that the belief that godlike intelligence is necessary for human extinction via AI is a popular AI doomer position among people who are intellectually sophisticated. It’s more like those people hold complex position and it’s easy for people who are skeptics to frame this as “a popular position”.
There’s an argument that some of the risk comes from “godlike intelligence” but that’s not necessary to believe in high risk. If you take an agent as smart as the smartest human and able to act faster while the agent can copy themselves after learning skills and able to potentially better coordinate among billions of copies of the agent that might be enough to overpower humanity.
You can’t conclude from the fact that inference scaling happened that most AI improvements are due to scaling.
I’m saying that this is already happening. It’s not as straightforward as with AlphaGo as it’s easier to judge whether a move helps with winning a game in the constrained environment of go, but when it comes to coding you have quality measurements such as whether or not the agent managed to write code that successfully made the unit tests pass and
There’s a lot of training on ‘synthetic data’ and data from user interactions and if you have a better agents that leads to higher data quality for both.
When it comes to inference it’s also worth noting that they found a lot of tricks to make inference cheaper. It’s not just more/better hardware:
Hang on, I don’t think I said that godlike intelligence was necessary for human extinction, and actually, didn’t make any claim about human extinction at all. This post was just about the possibility of an intelligence explosion, and I think “AI will reach godlike levels of intelligence” is an accurate description of the AI 2027 position.
Did you read the cited link that you quoted? Toby Ord’s argument was pretty convincing to me. What do you disagree with?
Right, ending in about late 2024, which is why I specified (~late 2024) in “most recent gains”. It doesn’t seem like that trend has continued.