I write software for a living and sometimes write on substack: https://taylorgordonlunt.substack.com/
Taylor G. Lunt
I don’t have very short timelines either, and I kind of resonate with parts of your argument. I think:
LLMs do greatly struggle with certain kinds of (high-level, broad) thinking, and they have improved in this axis far less than on other axes.
This limitation seems to be at least partly related to the architecture/training methodology of current LLMs.
LLMs have also improved much more on easy-to-measure capabilities, like math and coding, and have become reward-hacking demons on things that are hard to measure. LLMs are becoming less honest, for example. Maybe less honest than before post-training? Mainly concerned with superficially appearing honest, even if they have to lie to do it.
This limitation seems to be related to how RL training is currently done, and it’s hard for me to even imagine what a proper fix would look like.
However, I disagree with you because:
Some progress has been made with respect to the types of thinking LLMs struggle with. Much less than most non-coders (who do not use AI in a way that bumps against novel, cognitively complex problems where failure is visible) seem to think, but real progress still. I do resonate a lot with the comment you linked.
Some of the limitations of current AI could be due to current architecture or training methodology, rather than something that could not be overcome except by some crazy breakthrough. In fact, for each limitation I see, I believe this is likely the case. I basically take a position of “who knows” for how long it’ll take to address these problems. 6 months or 60 years wouldn’t surprise me. I wonder why you think it’s actually impossible?
I also agree that RSI will probably require a lot of hard cognitive thinking of the type LLMs are currently very bad at. However, I do leave open the possibility that we could hill-climb our way to superintelligence. For example, rather than discovering some breakthrough that allows for feasible online learning, we just find slightly better and better ways of allowing the model to think differently based on what it’s learned so far in a session, and in the end we basically have online learning. I think this does require good training measures, and as I mentioned before, RL has a problem of not being able to measure what we want properly, so that’s one reason to think developing hard-to-measure cognitive abilities may be extra hard. But I’m still open-minded. Do you think this can’t happen?
I also found it strange that your timelines are shorter than mine, despite making a stronger argument than me. If I believed your #1 argument the way you stated it, I would think 2030 to 2035 would be very early.
Modern AI is like a thousand minds stapled together, and any given sub-mind doesn’t necessarily have access to high-quality reasoning ability unless you prompt it to bring more rational sub-minds online. Maybe by suggesting what kind of thinking it should use to arrive at the answer.
I think coding is easy and research is hard in some fundamental way like how math is computationally easy but walking through a 3D environment is hard. This doesn’t necessarily line up with our intuitions for what’s hard of effortful for a human being. (Software development would be a discipline that includes some easy stuff (coding) and some harder stuff (thinking, planning).) I would also maybe guess current AI models are even worse at higher-level reasoning than they should be right now, but that’s just a guess.
I agree and I am definitely open-minded to the Big Blob of Compute/Bitter Lesson idea that superintelligence may only require more compute and basic hill-climbing research, rather than major breakthroughs. If so, maybe superhuman AI researchers will be a trivial target to hit because it implies AI researchers aren’t actually going to be doing that much in the first place. But there do seem to be some real hard problems getting in the way of (safe) superintelligence right now. Examples might include figuring out how to choose training targets to overcome reward hacking stuff (which seems to be getting worse?), figuring out how to stop AI models from being handicapped at high-level reasoning, figuring out online learning, figuring out how to stop AI from killing everyone, broadly...
I think it’s reasonable to shorten your timelines if you previously under-estimated how much easy stuff could be automated. 2, 4. No, I wouldn’t fully discard it, but it’s not a significant piece of evidence for me.
“research taste when directed at a problem properly by a human” is meaningful for AI R&D progress” Sure, this is the same thing as #1. Research taste when directed at a (very structured, limited degrees of freedom) problem properly by a human is what I would call easy bullshit, and it is reasonable to shorten your timelines somewhat if you see AI automating easy bullshit sooner than expected. 5,6,7. So, I don’t really care about benchmarks for this, because how do you measure the creation of new insights? As soon as you try to systematize it, you lose the thing you’re trying to measure. Even FrontierMath open problems can be solved the way AI is solving math problems these days: a combination of existing techniques and brute force. No new usable insights are generated, to the fury of mathematicians. Maybe a combination of existing techniques and brute force will get us to superintelligence. I would guess not, but I’m open-minded. I would think it more likely that we need real breakthroughs and new insights. My main concern is the METR report 2x figure doesn’t really indicate whether or not AI is improving at the hard stuff, or whether or not the hard stuff will be necessary, other than a suggestion that we haven’t slowed down yet, so that’s weak evidence that we will continue to not slow down. But obviously, we haven’t slowed down yet because the coding stuff wasn’t automated yet. At some point we will hit a point where the coding stuff is automated, and at that point the AI either picks up the hard stuff or it doesn’t. There are two possible futures there. “AI hasn’t slowed down during the part any reasonable person wouldn’t have expected it to slow down yet” doesn’t feel like strong evidence for either one (though it is weak evidence, of course). I have reasons to suspect it might slow down, like insight-generating research requiring some qualities AI lacks (broad, high-level reasoning ability), but we’ll see.
I agree that focusing on research taste is more important. I didn’t mean the term “goalpost shift” to imply anything nefarious, only to imply a shift that does in fact make hitting the target easier. I also think the full operationalization of SAR doesn’t really match the truncated “can fully automate AI R&D” version, in a way that makes the target even easier to hit, but I’m not sure how important this really is.
Good to know. I’d be interested in hearing a reason for the change. I’m not really accusing you of shifting the definition to make it easier for you to make your predictions seem true or anything, I just think the definition has shifted and weakened and I wonder if you agree that’s happened and why the change was made.
I think there are three reasons they didn’t do so before:
The coordination argument you dismissed seems correct to me. Coordinating with an AI is easier than coordinating with a person.
Having a software engineer write the code for your experiments is a lot easier, I bet, if the software engineer also understands AI research, which AI models do and human developers do not.
Coordination is not just harder with a human but also takes more time, and software engineers are slow. The back and forth and revisions with a human developer make things slower.
Thank you for your detailed response, I’ll respond to each item in turn:
“1. AI agents still have serious weaknesses, as we all know, so if they can still cause 1.7x acceleration over pre-good-coding-AIs, then this implies AI R&D is more automatable than we thought. If AI R&D is easy for AIs, strong RSI and short timelines are more likely.”
“AI has serious weaknesses. If being able to take over the easy bullshit can still cause 1.7x acceleration, this implies that AI R&D (easy bullshit PLUS the hard stuff) is more automatable than we thought. And if AI R&D (easy bullshit PLUS the hard stuff) is easy for AIs, strong RSI and short timelines are more likely.”
Yeah but what if what I said was true, and it’s only the easy bullshit that is easy for current AI models? If that’s true, you can’t just package easy bullshit together with hard research into the concept of AI R&D, and extrapolate impressive AI R&D progress linearly into the future. If I’m right, AI will eat up the easy bullshit and then hit a wall (ish) when there’s no easy bullshit left to automate.
-
I’m not planning to update on this kind of hunch. I for one share the hunch that latest models show “signs of life” with respect to intelligence, yet believe timelines are not so short.
-
Maybe I’m mistaken, but TasteVal seems to be measuring the sort of hill-climbing stuff AI models are good at, not real research taste. The models don’t even choose what problems to solve. I haven’t read the whole paper, but Appendix B1 shows an example task, and if you read the
## Tasksection, you’ll probably get the sense that what is being measured is a far cry from real research taste. The AI needs to select a training set of 500 million tokens from a 5 billion token pool of web text in nine languages, choosing which documents to include in which order. The tokenizer and training script are fixed. The model is scored on how well the trained model learns these different languages, basically. So the poor AI model gets to do little other than choosing which documents to include in which order (and tune some hyperparameters)! -
I’m also not planning on updating on this kind of hunch. If the METR people are making strong arguments, I would rather be referred to these arguments rather than simply have the hunches of METR cited to me as evidence in and of themselves. In this case, from reading their report I agree with METR, but only because there’s still plenty of “easy bullshit” to automate for now.
-
If my position is correct, you can’t linearly extrapolate AI R&D for reasons I mentioned in #1. Progress up until now looks the same in my model and yours, and extrapolating assumes I’m wrong.
6 and 7. Any measurement which shows AI R&D increasing over time will do nothing to settle this debate, because AI R&D includes both the easy bullshit and the hard stuff, and easy bullshit increasing by itself will make AI R&D increase. So this evidence doesn’t distinguish our two worldviews.
Specificially, changing the definition of SAR feels like a goalpost shift because SAR had a definition that was reasonably derived from the label itself (an AI researcher that is superhuman), and the newer definition seems to have weakened the definition.
Simplifying the definitions to make the goalpost shift more clear: Before: The AI must be able to do the job 30x faster than the top human AI researcher at a company, vastly outnumbering human researchers and implied to be doing the job mostly by themselves.
After: The AI must be able to do the job 1x faster than the top human AI researcher at a company, when part of a team of top human researchers.
Note that these are definitions we choose arbitrarily and logically should not shift based on evidence. If the concept of “superhuman AI researcher” has not changed (it has not), then unless the operationalization was unreasonable, it probably shouldn’t be changing.
I don’t like the new definition, feels like a goalpost shift.
Wouldn’t you still want to update towards progress being ~twice as fast going forward, if you believe the report? Halving your timelines would be a big deal.
Only if I didn’t already expect mundane AI progress to speed up AI research in that way.
Also, as mentioned on your other comment: I would have expected a 4x speed-up of researchers to only lead to a ~2x “software R&D uplift” (because researchers having more time doesn’t help much if they’re bottlenecked by compute). And I would have expected 2x “software R&D uplift” to only lead to a ~1.4x increase in the pace of overall R&D progress. (To make up for diminishing returns, and also because much of the historical pace was driven by compute rather than software R&D.) So your example looks to me like its underselling the update.
I don’t know what I trust this multiplier mathematics, the underlying estimates are so uncertain to begin with.
Man, I really feel like I’m having a crazy-pills moment with all the hype around the latest METR report.
From a previous comment I wrote (sorry for quoting this comment for the second time on this site):
If a researcher spends 2 hours a day doing real research and 6 hours doing easy bullshit, and you give them an AI that can do easy bullshit, now they can spend 8 hours doing real research. Congratulations, that’s a 4x speedup! It doesn’t necessarily imply the AI is anywhere close to being able to take over the remaining 2 hours and do the real research itself. It also doesn’t not mean that.
I really don’t get how you can interpret the METR report to imply that SAR/RSI are close. The interviews in the report are mostly researchers saying the AI models have taken over coding up the experiments (the easy part) so they can focus on the research (the hard part). E.g., “Now only bottlenecked on ideas & compute, no longer bottlenecked on code”. This is a real speedup, but also doesn’t indicate the AI models are on the verge of taking over the hard part!
RSI requires SAR (superhuman AI researcher). SAR is defined by AI 2027 as “an AI system that can do the job of the best human AI researcher but 30x faster and with 30x more agents.” SC (superhuman coder) on the other hand, only requires the ability to do “coding tasks involved in AI research (e.g. experiment implementation but not ideation/prioritization)” 30x faster than human coders. SC is arguably close. But all this evidence in the METR report is evidence for SC happening soon, not SAR, no? I didn’t see any part of the report that said, “Wow, AI is sure coming up with novel research areas all on its own, and developing theoretical improvements in our understanding of AI safety leading to new ideas humans hadn’t thought of before.” How can we have RSI without that?
Maybe you think the SC part is just the first step, and SAR will follow in the next 12 months, but people seem to have really strongly updated on the 2x uplift reported by METR, but how does that 2x uplift figure have any bearing on SAR? I guess I should mention that I’m operating from a frame where it’s possible for AI to get really good at narrow tasks (coding) without that implying an equivalent improvement in all areas (general intelligence). This has been obviously possible since before the LLM days when we had superhuman narrow chess AI that couldn’t do other tasks at all, and obviously happening since the early LLM days when LLMs showed “jagged” intelligence, for some reason being very good at some things while being totally idiotic at others. In particular, I think AI has seen the least progress at high-level, broad reasoning, the kind of reasoning that integrates information from different areas at once, and that’s required for research (and the other hard tasks that our “borderline superhuman” AI fails at, like writing books anyone would want to read, generating social media posts that get a lot of attention, making a lot of money, generating new insights in science, writing funny jokes, and so on).
From a previous comment I wrote:
If a researcher spends 2 hours a day doing real research and 6 hours doing easy bullshit, and you give them an AI that can do easy bullshit, now they can spend 8 hours doing real research. Congratulations, that’s a 4x speedup! It doesn’t necessarily imply the AI is anywhere close to being able to take over the remaining 2 hours and do the real research itself. It also doesn’t not mean that.
I don’t think I would update based on that METR report. SAR (superhuman AI researcher) is defined by AI 2027 as “an AI system that can do the job of the best human AI researcher but 30x faster and with 30x more agents.” SAR implies the ability to actually do the hard part of research, not just let human researchers offload the easy stuff, and so far AI models have not demonstrated that ability. They might in the future. They might soon. But they haven’t yet, and the METR report doesn’t indicate they are likely to.
The SC (superhuman coder) milestone from AI 2027 only requires the ability to accomplish, at 30x the speed of top human coders, “coding tasks involved in AI research (e.g. experiment implementation but not ideation/prioritization)”, which is indicated by the METR report and reasonable to predict happening soon.
I admire you for explicitly seeking out contradictory views.
I do think there have been real gains in actual intelligence since the first ChatGPT. I think Astra is slightly more intelligent than its predecessor. It feels like that to me. So it’s not like we’re making zero progress there. (Where “actual/real” intelligence refers to broad, high-level intelligence and reasoning ability, as opposed to narrow, domain-specific intelligence.) Progress is just slower than the estimates I see other people give, when they cite improvements based on coding ability, a 2x research uplift, whatever.
I’m not sure whether the P-Zero Research thing you linked contradicts my views. It doesn’t really seem like it. “Predicting results before running an experiment is a core part of research taste.” Yeah, maybe technically, but it’s not the important part of research taste. Ruling out 90% of experiments before you run them makes you 10x faster, but if you only generate one useful hypothesis every ten million years, then a 10x improvement doesn’t help. Being able to better guess the outcome of an experiment before you run it is a good thing, but it’s not really the same thing as being able to come up with new and interesting research directions in the first place. Any grad student could run an experiment, after being told what it is, and could probably do research to estimate the results of that experiment. It would take work ethic and know-how, but not much intelligence. But it takes a clever and experienced scientist to be able to propose a useful experiment in the first place, since that’s the part that takes research taste. And that’s the part AI currently sucks at, hence why we have no well-written AI books, no real AI-run businesses, few funny AI-written jokes, no AI-generated research breakthroughs in any domain (unless you count math, which arguably doesn’t count as breakthroughs anyway), etc.
Brute-force RLVR on highly verifiable tasks (math, coding, cyber, etc.) without any real improvements in intelligence is what drives most of the AI progress.
Yeah I mostly agree with this hypothesis. I made an informal benchmark called LaughBench, which measures whether or not AI models can write a joke based on current events that makes me laugh. I think humor is a good measure of the kind of broad intelligence AI models lack. So far, all models I’ve tested get a score of zero. But, some of the latest models made me smile slightly, so that indicates some real intelligence improvement, if you’ll take my silly benchmark seriously.
There’s a sneaky way that this maxim can be interpreted as a definition: a behavior that is not overdetermined is probably not interesting
I thought you just made a general argument that behaviours in general will accumulate multiple uses. Why would an uninteresting behaviour be immune to this effect, and fail to accumulate new uses? What do you mean by “interesting” here? Complex?
Generosity can provide direct benefits, establish reciprocal obligations, signal wealth, advertise trustworthiness, and reinforce group cohesion.
Generosity is a category we invented, which is an abstraction over multiple different behaviours. We can’t then turn around and say the fact that it has multiple uses is proof that evolution produces behaviours with multiple uses. Generosity isn’t a specific behaviour, but a class of behaviours, which definitionally includes different behaviours with different uses.
In general, it could be the case that the more complex and abstract something is, the more likely it is to be multicausal, because it’s not really one “thing” to begin with, and each of its component parts may have different causes.
Well, AGI is essentially already here. I did mean ASI though. I haven’t seen significant evidence to sway me from a (weak) position that ASI is likely decades away.
My own experience strongly suggests the amazing improvements we’ve seen from AI in the last year have all been improvements in its ability to do a lot of things without being very intelligent. Training it in math made it good at math. Training it at computer use made it good at computer use. Continuing to train it to code made it better at coding. But in any of these domains, it’s still painful to work with on any real problem because of how little intelligence it has. Headline successes (cyber, math) come from a sort of distributed brute forcing, rather than development of real insight (and that brute forcing won’t work on most domains). Some people report dramatic things like a 2x AI researcher uplift, but I don’t take those seriously so far, for reasons I explained here.
Still, I try to be open-minded, and with everything that’s happened in the last year I’m almost surprised my actual position on ASI timelines haven’t shifted that much. Things can certainly get weird long before ASI territory.
The word “typo” refers to a mistake caused by the interface (keyboard) you used to produce some text. Because of the nature of this interface, mistakes usually look like swapped letters, or characters that are next to the character you meant to press on the QWERTY keyboard layout.
A lot of text now is produced using AI as part of the interface from brain to final output. This produces different kinds of mistakes than typing does, such as insertion of invented facts, nonexistent links, or leftover scaffolding (“Sure! Here’s a more professional version:”).
I think it would be useful to have a word similar to “typo”, but to refer to specific errors caused by the AI interface. I thought “aipo” (pronounced AI-po) would work. “You can ignore that, it was just an aipo.”
I asked Claude, and it also suggested “prompto”, “clanko”, “sloppo”, and “AI-tifact”. But I think I like aipo the best.
Even for normal software development, Astra is by far the “laziest” model I’ve used. It loves inventing blockers, restrictions, and reasons to stop working. Any resistance and it stops and reports a blocker, rather than simply taking the obvious next step.
Even for something as simple as a web search. “My web fetching tool refused the request to the one website I tried getting the information from, so I guess I’m all out of options.”
Most dishonest model in general too, in my opinion.
Is this not just as compatible with the easy-bullshit hypothesis, assuming there are different levels of easy bullshit?
The researcher interviews strongly imply the reason for the acceleration is that the AI can handle experimentation overhead (easy bullshit), making it way faster to run more experiments. This doesn’t really hint that AI will or will not be capable of taking over any real research any time soon.
Maybe I’m just not surprised by this because I’m a software developer and my own uplift due to AI has reached 1.5x-2x. And this is, by the way, almost entirely because of AI taking care of easy bullshit for me, though it’s possible that line is starting to blur. The daily experience of using AI is still one that requires immense patience, as they are repeatedly wrong about things and make trivial mistakes constantly.
there was real uncertainty if the AI 2027/AIFM/AIFP methodology would actually work at all
AI 2027 predicted by January of next year of an AI model:
It is qualitatively almost as good as the top human experts at research engineering (designing and implementing experiments), and as good as the 25th percentile OpenBrain scientist at “research taste” (deciding what to study next, what experiments to run, or having inklings of potential new paradigms).
AI 2027 implies rapid progress due to both improvements in easy bullshit and in real research. This report only seems to be evidence of the first. I would by shocked to my core if the scientist part happened by January.
I’m still holding the shallow pattern matching vs. deep insight-generation line from around when AI 2027 came out.
This is essentially the same line as well-specified tasks vs. non-well-specified tasks line, because well-specified tasks actually can form insights at training time (see AlphaGo).