Software engineer and repeat startup founder; best known for Writely (aka Google Docs). Now president of the Golden Gate Institute for AI (https://www.goldengateinstitute.org/), where I’m working to foster constructive expert conversations about open questions in AI and AI policy, and posting at https://secondthoughts.ai/ and https://x.com/snewmanpv.
snewman
Karma: 692
AIs trained on the training set of ARC-AGI (and given a bunch of compute) can beat humans on ARC-AGI-1.
Say more? At https://arcprize.org/leaderboard, I see “Stem Grad” at 98% on ARC-AGI-1, and the highest listed AI score is 75.7% for “o3-preview (Low)”. I vaguely recall seeing a higher reported figure somewhere for some AI model, but not 98%.
ARC-AGI-2 isn’t easy for humans. It’s hard for humans and AIs probably do similarly to random humans (e.g. mturkers) given a limited period.
This post states that the “average human” scores 60% on ARC-AGI-2, though I was unable to verify the details (it claims to be a linkpost for an article which does not seem to contain that figure). Personally I tried 10-12 problems when the test was first launched, and IIRC I missed either 1 or 2.
The leaderboard shows “Grok 4 (Thinking)” on top at 16%… and, unfortunately, does not present data for “Stem Grad” or “Avg. Mturker” (in any case I’m not sure what I think of the latter as a baseline here).
Agreed that perception challenges may be badly coloring all of these results.
There isn’t some dictomy between “sample efficient learning” and not.
Agreed, but (as covered in another comment – thanks for all the comments!), I do have the intuition that the AI field is not currently progressing toward rapid improvement on sample efficient learning, and may currently be heading toward a fairly low local maximum.
Yeah, I was probably too glib here. I was extrapolating from the results of the competition Epoch organized at MIT, where “o4-mini-medium outperformed the average human team, but worse than the combined score across all teams, where we look at the fraction of problems solved by at least one team”. This was AI vs. teams of people (rather than any one individual person), and it was only o4-mini, but none of those people were Terence Tao, and it only outperformed the average team.
I would be fascinated to see how well he’d actually perform in the scenario you describe, but presumably we’re not going to find out.
if you compared AIs to a group of humans who are pretty good at this type of math, the humans would probably also destroy the AI.
I wonder? Given that, to my understanding, each FrontierMath problem is deep in a different subfield of mathematics. But I have no understanding of the craft of advanced / research mathematics, so I have no intuition here.
Anyway, I think we may be agreeing on the main point here: my suggestion that LLMs solve FrontierMath problems “the wrong way”, and your point about depth arguably being more important than breadth, seem to be pointing in the same direction.
I do believe that AIs will eventually surpass humans at fluid intelligence, though I’m highly uncertain as to the timeline.
My point here is really just the oft-repeated observation that when we see an AI do X, intuitively we tend to assess the AI the way we would assess a human being who could do X, and that intuition can lead to very poor estimates of whether the AI can also do Y. (For instance, bar exam → practicing law.) For instance, the relative ratios of fluid vs. crystal intelligence may capture much of the reason that AIs are approaching superhuman status at competition coding problems but are still far from superhuman at many real-world coding tasks. It doesn’t mean AIs will never get to real-world tasks. It just suggests (to me) that they might be farther from that milestone than their performance on crystal-intelligence-friendly tasks would imply.
Agreed that I have not supported my claims here – this was a vibes piece.
I agree that LLMs are improving at ~everything, but my intuition is that some of those improvements – for instance, regarding continuous learning – may currently be of the “climbing a ladder to get closer to the moon” variety. Sounds like we just have very different intuitions here.
I’m having trouble parsing what you’ve said here in a way that makes sense to me. Let me try to lay out my understanding of the facts very explicitly, and you can chime in with disagreements / corrections / clarifications:
The human brain has, very roughly, 100B neurons (nodes) and 100T synapses (connections). Each synapse represents at least one “parameter”, because connections can have different strengths. I believe there are arguments that it would in fact take multiple parameters to characterize a synapse (connection strength + recovery time + sensitivity to various neurotransmitters + ???), and I’m sympathetic to this idea on the grounds that everything in the body turns out to be more complicated than you think, but I don’t know much about it.
Regarding GPT-4, I believe the estimate was that it has 1.8 trillion parameters, which if shared weights are used may not precisely correspond to connections or FLOPs. For purposes of information storage (“learning”) capacity, parameter count seems like the correct metric to focus on? (In the post, I equated parameters with connections, which is incorrect in the face of shared weights, but does not detract from the main point, unless you disagree with my claim that parameter count is the relevant metric here.)
To your specific points:
Probably the effective number of parameters in the human brain is actually lower than 100 trillion because many of these “parameters” are basically randomly initialized or mostly untrained. (Or are trained very slowly/weakly.) The brain can’t use a global learning algorithm, so it might effectively use parameters much less efficiently.
What is your basis for this intuition? LLM parameters are randomly initialized. Synapses might start with better-than-random starting values, I have no idea, but presumably not worse than random. LLMs and brains both then undergo a training process; what makes you think that the brain is likely to do the worse job of training its available weights, or that many synapses are “mostly untrained”?
Also note that the brain has substantial sources of additional parameters that we haven’t accounted for yet: deciding which synapses to prune (out of the much larger early-childhood count), which connections to form in the first place (the connection structure of an LLM can be described in a relative handful of bits, while the connection structure of the brain has an enormous number of free parameters; I don’t know how “valuable” those parameters are, but natural systems are clever!), where to add additional connections later in life.
It’s a bit confusing to describe GPT-4 as having 1.8 trillion connections as 1.8 trillion is the number of floating point operations (roughly) not the number of neurons.
I never mentioned neurons. 1.8 trillion is, I believe, the best estimate for GPT-4′s parameter count. Certainly we know that the largest open-weight models have parameter counts of this order of magnitude (somewhat smaller but not an OOM smaller). As noted, I forgot about shared weights when equating parameters to connections, but again I don’t think that matters here. FLOPs to my understanding would correspond to connections (and not parameter counts, if shared weights are used), but I don’t think FLOPs are relevant here either.
In general, the analogy between the human brain and LLMs is messy because a single neuron probably has far fewer learned parameters than a LLM neuron, but plausibly somewhat more than a single floating point number.
GPT-5 estimates that GPT-4 had just O(100M) neurons. Take that figure with a grain of salt, but I mention it to point out that in both modern LLMs and the human brain, there are far more connections / synapses than nodes / neurons, and the vast majority of parameters will be associated with connections, not nodes. (Which is why I didn’t mention neurons in the post, and I don’t think it’s useful to talk about learned parameters in reference to neurons.)
Nice analysis. I can’t add anything substantive, but this writeup crystallized for me just how much we’re all focusing on METR’s horizon lengths work. On the one hand, it’s the best data set we have at the moment for quantitative extrapolation, so of course we should focus on it. On the other hand, it’s only one data set, and could easily turn out to not imply what we think it implies.
My only points are (a) we shouldn’t weight the horizon length trends too heavily, and (b) boy do we need additional metrics that are both extrapolatable, and plausibly linked to actual outcomes of interest.
Thanks. This is helpful, but my intuition is substantially coming from the idea that there might be other factors involved (activities / processes involved in improving models that aren’t “thinking about algorithms”, “writing code”, or “analyzing data”). In other words, I have a fair amount of model uncertainty, especially when thinking about very large speedups.
quantity of useful environments that AI companies have
Meaning, the number of distinct types of environments they’ve built (e.g. one to train on coding tasks, one on math tasks, etc.)? Or the number of instances of those environments they can run (e.g. how much coding data they can generate)?
GPT-4.5 is going to be quickly deprecated
It’s still a data point saying that OpenAI chose to do a large training run, though, right? Even if they’re currently not planning to make sustained use of the resulting model in deployment. (Also, my shaky understanding is that expectations are for a GPT-5 to be released in the coming months and that it may be a distilled + post-trained derivative of GPT-4.5, meaning GPT-5 would be downstream of a large-compute-budget training process?)
Oops, I forgot to account for the gap from 50% success rate to 80% success (and actually I’d argue that the target success rate should be higher than 80%).
Also potential factors for “task messiness” and the 5-18x context penalty, though as you’ve pointed out elsewhere, the latter should arguably be discounted.
Agreed that we should expect the performance difference between high- and low-context human engineers to diminish as task sizes increase. Also agreed that the right way to account for that might be to simply discount the 5-18x multiplier when projecting forwards, but I’m not entirely sure. I did think about this before writing the post, and I kept coming back to the view that when we measure Claude 3.7 as having a 50% success rate at 50-minute tasks, or o3 at 1.5-hour tasks, we should substantially discount those timings. On reflection, I suppose the counterargument is that this makes the measured doubling times look more impressive, because (plausibly) if we look at a pair of tasks that take low-context people 10 and 20 minutes respectively, the time ratio for realistically high-context people might be more than 2x. But I could imagine this playing out in other ways as well (e.g. maybe we aren’t yet looking at task sizes where people have time to absorb a significant amount of context, and so as the models climb from 1 to 4 to 16 to 64 minute tasks, the humans they’re being compared against aren’t yet benefiting from context-learning effects).
One always wishes for more data – in this case, more measurements of human task completion times with high and low context, on more problem types and a wider range of time horizons...
Surely if AIs were completing 1 month long self contained software engineering tasks (e.g. what a smart intern might do in the first month) that would be a big update toward the plausiblity of AGI within a few years!
Agreed. But that means time from today to AGI is the sum of:
Time for task horizons to increase from 1.5 hours (the preliminary o3 result) to 1 month
Plausibly “a few years” to progress from 1-month-coder to AGI.
If we take the midpoint of Thomas Kwa’s “3-4 months” guess for subsequent doubling time, we get 23.8 months for (1). If we take “a few years” to be 2 years, we’re in 2029, which is farther out than “the most aggressive forecasts” (e.g. various statements by Dario Amodei, or the left side of the probability distribution in AI 2027).
And given the starting assumptions, those are fairly aggressive numbers. Thomas’ guess that “capability on more realistic tasks will follow the long-term 7-month doubling time” would push this out another two years, and one could propose longer timelines from one-month-coder to AGI.
Of course this is not proof of anything – for instance, task horizon doubling times could continue to accelerate, as envisioned in AI 2027 (IIRC), and one could also propose shorter timelines from one-month-coder to AGI. But I think the original statement is fair, even if we use 3-4 months as the doubling time, this is an update away from “the most aggressive forecasts”?
(When I wrote this, I was primarily thinking about Dario projecting imminent geniuses-in-a-data-center, and similar claims that AGI is coming within the next couple of years or even is already here.)
To be clear, I agree that the bad interpretations were not coming from METR.
Thanks, I’ve edited the post to note this.
Sure – I was presenting these as “human-only, software-only” estimates:
Here are the median estimates of the “human-only, software-only” time needed to reach each milestone:
So it doesn’t seem like there’s a problem here?
I added up the median “Predictions for gap size” in the “How fast can the task difficulty gaps be crossed?” table, summing each set of predictions separately (“Eli”, “Nikola”, “FutureSearch”) to get three numbers ranging from 30-75.
Does this table cover the time between now and superhuman coder? I thought it started at RE-Bench, because:
I took all of this to be in context of the phrase, about one page back, “For each gap after RE-Bench saturation”
The earlier explanation that Method 2 is “a more complex model starting from a forecast saturation of an AI R&D benchmark (RE-Bench), and then how long it will take to go from that system to one that can handle real-world tasks at the best AGI company” [emphasis added]
The first entry in the table (“Time horizon: Achieving tasks that take humans lots of time”) sounds more difficult than saturating RE-Bench.
Earlier, there’s a separate discussion forecasting time to RE-bench saturation.
But sounds like I was misinterpreting?
Correct. Am I wrong in thinking that it’s usual to use the word “timelines” to refer to the entire arc of AI progress, including both the periods covered in the “Timelines Forecast” and “Takeoff Forecast”? But, since this is all in the context of AI 2027 I should have clarified.
What’s your basis for “well-defined tasks” vs. “realistic tasks” to have very different doubling times going forward? Is the idea that the recent acceleration seems to be specifically due to RL, and RL will be applicable to well-defined tasks but not realistic tasks?
This seems like an extremely important question, so if you have any further thoughts / intuitions / data to share, I’d be very interested.
Thanks everyone for all the feedback and answers to my unending questions! The branching comments are starting to become too much to handle, so I’m going to take a breather and then write a followup post – hopefully by the end of the week but we’ll see – in which I’ll share some consolidated thoughts on the new (to me) ideas that surfaced here and also respond to some specific points.
Good point.