Unfortunately, those are two importantly different questions. While it is certainly possible that AGI will arrive before 2030, I am not sure that it would be predictable in advance without a “textbook of the (far) future” on deep learning. Of course, it is much more likely to be predictable a couple of days in advance than a year in advance, but even the former may conceivably be quite challenging depending on situational awareness of near-human-level models in training.
What would you expect the world to look like if AGI < 2030?
In many respects, similar to how it does look (e.g. AI passes the Turing test pretty easily / convincingly, is competent at some coding tasks, etc. etc.).
It’s more “interesting” to discuss the departures:
Claude would probably be better at Pokemon
Maybe self-driving vehicles are more common / robust
I think the METR plot would look different. I don’t think task length should be the bottleneck. I think that indicates that something isn’t scalable. But I am not sure.
I would weakly expect to find AI a little more useful, but more strongly, I would expect to be finding it increasingly useful over the last couple of years, and I don’t.
“Hallucinations” (I think @Abram Demski has a better term I can’t remember) would be easier to get rid of.
Each point is capable of shifting my opinion to a greater or lesser degree. Mainly seeing novel insights would change my mind. If METR’s results hold up that will also change my mind.
Personally I think 2030 is possible but aggressive, and my timeline estimate it more around 2035. Two years ago I would have said 2040 or a bit later, and capabilities gains relevant to my own field and several others I know reasonably well have shortened that, along with the increase in funding for further development.
The Claude/Pokemon thing is interesting, and overall Pokemon-playing trend across Anthropic’s models is clearly positive. I can’t say I had any opinion at all about how far along an LLM would get at Pokemon before that result got publicized, so I’m curious if you did. What rate of progress on that benchmark would you expect in a short-timelines world? If there’s an LLM agent that can beat Pokemon in six months, or a year, or two years?
Self-driving vehicles are already more of a manufacturing and regulatory problem than a technical one. For example, as long as the NHTSA only lets manufacturers deploy 2500 self-driving vehicles a year each in the US, broad adoption cannot happen, regardless of technical capabilities or willingness to invest and build.
I also don’t think task length is a perfect metric. But it’s a useful one, a lower bound on what’s needed to be able to complete all human-complete intellectual tasks. Like everything else to date, there is likely something else to look at as we saturate the benchmark.
I agree novel insights (or more of them, I can’t say there haven’t been any) will be strong evidence. I don’t understand the reason for thinking this should already be observable. Very, very few humans ever produce anything like truly novel insights at the forefront of human knowledge. “They have not yet reached the top <0.1% of human ability in any active research field” is an incredibly high bar I wouldn’t expect to pass until we’re already extremely close to AGI, and it should be telling that that late bar is on the short list of signs you are looking for. I would also add two other things: First, how many research labs do you think there are that have actually tried to use AI to make novel discoveries, given how little calendar time there has been to actually figure out how to adopt and use the models we do have? If Gemini 2.5 could do this today, I don’t think we’d necessarily have any idea. And second, do you believe it was a mistake that two of the 2024 Nobel prizes went to AI researchers, for work that contributes to the advancement of chemistry and physics?
AI usefulness is strongly field dependent today. In my own field, it went from a useful supplementary tool to “This does 50-80% of what new hires did and 30-50% of what I used to do, and were scrambling to refactor workflows to take advantage of it.”
Hallucinations are annoying, but good prompting strategy, model selection, and task definition can easily get the percentages down to the low single digits. In many cases the rates can easily be lower than those of a smart human given a similar amount of context. I can often literally just tell an LLM “Rewrite this prompt in such a way as to reduce the risk of hallucinations or errors, answer that prompt, then go back and check for and fix any mistakes” and that’ll cut it down a good 50-90% depending on the topic and the question complexity. I can ask the model to cite sources for factual claims, dump the sources back into the next prompt, and ask if there are any factual claims not supported by the sources. It’s a little circular, but also a bit Socratic and not really any worse than when I’ve tried to teach difficult mental skills to some bright human adults
Of course, it is much more likely to be predictable a couple of days in advance than a year in advance, but even the former may conceivably be quite challenging depending on situational awareness of near-human-level models in training.
Do I understand correctly that you think that we are likely to only recognize AGI after it has been built? If so, how would we recognize AGI as you define it?
Do you also think that AGI will result in a fast take-off?
That’s alright. Would you be able to articulate what you associate with AGI in general? For example, do you associate AGI with certain intellectual or physical capabilities, or do you associate it more with something like moral agency, personhood or consciousness?
Unfortunately, those are two importantly different questions. While it is certainly possible that AGI will arrive before 2030, I am not sure that it would be predictable in advance without a “textbook of the (far) future” on deep learning. Of course, it is much more likely to be predictable a couple of days in advance than a year in advance, but even the former may conceivably be quite challenging depending on situational awareness of near-human-level models in training.
In many respects, similar to how it does look (e.g. AI passes the Turing test pretty easily / convincingly, is competent at some coding tasks, etc. etc.).
It’s more “interesting” to discuss the departures:
Claude would probably be better at Pokemon
Maybe self-driving vehicles are more common / robust
I think the METR plot would look different. I don’t think task length should be the bottleneck. I think that indicates that something isn’t scalable. But I am not sure.
There would perhaps be a few novel insights or discoveries from LLMs in a few domains where they have an advantage over us. I am not convinced this is true and in fact absence of evidence is evidence of absence: https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights
I would weakly expect to find AI a little more useful, but more strongly, I would expect to be finding it increasingly useful over the last couple of years, and I don’t.
“Hallucinations” (I think @Abram Demski has a better term I can’t remember) would be easier to get rid of.
Each point is capable of shifting my opinion to a greater or lesser degree. Mainly seeing novel insights would change my mind. If METR’s results hold up that will also change my mind.
Personally I think 2030 is possible but aggressive, and my timeline estimate it more around 2035. Two years ago I would have said 2040 or a bit later, and capabilities gains relevant to my own field and several others I know reasonably well have shortened that, along with the increase in funding for further development.
The Claude/Pokemon thing is interesting, and overall Pokemon-playing trend across Anthropic’s models is clearly positive. I can’t say I had any opinion at all about how far along an LLM would get at Pokemon before that result got publicized, so I’m curious if you did. What rate of progress on that benchmark would you expect in a short-timelines world? If there’s an LLM agent that can beat Pokemon in six months, or a year, or two years?
Self-driving vehicles are already more of a manufacturing and regulatory problem than a technical one. For example, as long as the NHTSA only lets manufacturers deploy 2500 self-driving vehicles a year each in the US, broad adoption cannot happen, regardless of technical capabilities or willingness to invest and build.
I also don’t think task length is a perfect metric. But it’s a useful one, a lower bound on what’s needed to be able to complete all human-complete intellectual tasks. Like everything else to date, there is likely something else to look at as we saturate the benchmark.
I agree novel insights (or more of them, I can’t say there haven’t been any) will be strong evidence. I don’t understand the reason for thinking this should already be observable. Very, very few humans ever produce anything like truly novel insights at the forefront of human knowledge. “They have not yet reached the top <0.1% of human ability in any active research field” is an incredibly high bar I wouldn’t expect to pass until we’re already extremely close to AGI, and it should be telling that that late bar is on the short list of signs you are looking for. I would also add two other things: First, how many research labs do you think there are that have actually tried to use AI to make novel discoveries, given how little calendar time there has been to actually figure out how to adopt and use the models we do have? If Gemini 2.5 could do this today, I don’t think we’d necessarily have any idea. And second, do you believe it was a mistake that two of the 2024 Nobel prizes went to AI researchers, for work that contributes to the advancement of chemistry and physics?
AI usefulness is strongly field dependent today. In my own field, it went from a useful supplementary tool to “This does 50-80% of what new hires did and 30-50% of what I used to do, and were scrambling to refactor workflows to take advantage of it.”
Hallucinations are annoying, but good prompting strategy, model selection, and task definition can easily get the percentages down to the low single digits. In many cases the rates can easily be lower than those of a smart human given a similar amount of context. I can often literally just tell an LLM “Rewrite this prompt in such a way as to reduce the risk of hallucinations or errors, answer that prompt, then go back and check for and fix any mistakes” and that’ll cut it down a good 50-90% depending on the topic and the question complexity. I can ask the model to cite sources for factual claims, dump the sources back into the next prompt, and ask if there are any factual claims not supported by the sources. It’s a little circular, but also a bit Socratic and not really any worse than when I’ve tried to teach difficult mental skills to some bright human adults
Thank you for the clarification!
Do I understand correctly that you think that we are likely to only recognize AGI after it has been built? If so, how would we recognize AGI as you define it?
Do you also think that AGI will result in a fast take-off?
I don’t think I have anything better than a guess about any of those questions.
That’s alright. Would you be able to articulate what you associate with AGI in general? For example, do you associate AGI with certain intellectual or physical capabilities, or do you associate it more with something like moral agency, personhood or consciousness?