I’m the chief scientist at Redwood Research.
ryan_greenblatt
Karma: 23,455
Ok, on closer examination of my numbers and views, what I said here isn’t that accurate about my current views. (This is mostly driven by me thinking more carefully about what my views should be and updating towards slightly longer timelines.)
I’m actually pretty close to Eli on both AC and TEDAI, with shorter timeline to AC and longer timelines to TEDAI. See here for more details.
(My views aren’t that stable on reflection as is apparent here...)
AIs can now often do massive easy-to-verify SWE tasks and I’ve updated towards shorter timelines
I’ve updated towards faster progress, more acceleration from AIs, and superexponential time horizon progress earlier.
I plan on saying more in a new post.I say more in this post. (And see this other comment.)
These numbers seem reasonable. My median for “full automation of AI R&D” is ~early 2031 and my 25th percentile is ~mid 2028.
I’m between Daniel’s views (more aggressive) and Eli’s views. (My superintelligence median is closer to Eli’s, my superhuman coder median is closer to Daniel’s. My TEDAI median is probably closer to Eli’s.)
I suspect it’s mostly bottlenecked on vision and something like video/motion perception. I think LLMs are particularly bad at seeing “motion” and understanding how a long series of images/frames are changing relative to humans. It also doesn’t compress nicely into the context length so it amplifies one LLM weakness.
I suspect that if AI companies aggressively trained on this sort of task, performance would rapidly improve and this would substantially be due to overcoming non-fundamental weaknesses. (I also think you could switch to input formats that are less brutal on the context length which would help.)
I think their testing was very underelicited in ways that are unrepresentative for tasks of interest like AI R&D.
I don’t know what threshold of score as they are currently measuring it corresponds to typical human performance. They haven’t published the data you’d need to answer this question. I’d guess around 15%?
I don’t currently think low scores on this benchmark solidly rule out full automation of AI R&D or other impresssive abilities, though I suspect that if non-trivial elicitation effort is applied to these sorts of tasks, AIs will exceed median human performance by the time of full automation of AI R&D.
I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.
Haven’t engaged enough to know, could be bid to engage, won’t by default.
Sure, but the training data thing is sort of tied up with fluid intelligence? Like a method you have available as a human is learning on the task itself and AIs also have this option, but due to garbage sample efficiency it wouldn’t work.
I also still think humans can often generalize to doing tasks of larger scope without prior experience by learning/adapting from this exact experience in a way that AIs don’t.
I’m going to reply kinda briefly and then I’m going to bow out with some chance I comment more later.
What’s the relative weight of these considerations, grossly speaking? Is it more like 90% extrapolations and 10% hand-off or more like the reverse?
Hmm, these are tied up a bit. Like I’m doing the extrapolations to get to automating AI R&D to various extents and then trying to account for acceleration due to automation. But I’m also extrapolating out to human fluid intelligence somewhat and if this extrapolation looked extremely far that would influence my view.
To give a rough (and potentially not that stable on reflection answer):
50% extrapolations towards automation of current AI R&D (both towards full automation and large speeds up from very strong partial automation)
25% armies of AIs that can automate AI R&D will be able to find a way to get to human level fluid intelligence
25% extrapolations towards human level fluid intelligence
This is something like an importance attribution. E.g., if I updated toward a way lower qualitative extrapolation on (3), that would update me quite a bit, but probly not shift my 25th percentile by more than a factor of like 2-3.
For (1), I’m doing a mix of something like time horizon, some informal sense of how good AIs are at research tasks I give them and what I see from others, and some longer list of considerations. And part of this is factoring in earlier speed ups moving us along the trajectory faster.
For the hand-off (what I would call (2) in my breakdown), I can see why the argument seems circular, but I don’t think it’s actually circular. My view is like “probably tons of AI R&D via stuff sorta like the current stuff we’re doing would yield this (due to a mix of my observations from ML, my extrapolations of current progress, thinking of specific things you could do that I won’t mention here without more consideration)” and “probably you’ll be able to get AIs to do this without very high fluid intelligence, though I do think AIs at this point will match somewhat worse human engineers/scientists at moderate time horizon fluid intelligence (e.g. over the course of a few month) on the relevant tasks”.
(My views are also somewhat based on defering to other people and the forecasting track record of other people.)
Why I’m not engaging much here:
I feel like I’ve personally engaged with your arguments a decent amount and it seems unlikely that further argumentation would change my perspective very much.
I’d guess that there aren’t interested third parties for whom I care a bunch about their views for whom my discussion with you would update them at all.
I have limited capacity for writing stuff and there is other stuff I want to write about that seems more important in the short term.
I think other formats are better for me than writing comments back and forth.
I’m moderately excited to argue with people about timelines, especially via formats other than writing comments back and forth, but most strongly if (1) I think it might update my views (2) third parties I care about care about the other person’s views or perspective.
I think I like half agree, but also think your argument implies things which are false.
In particular, I don’t think the reason why humans are much better at much longer tasks is mostly because they’ve previously done longer tasks. I think humans trained on only a variety of school projects that took around 8 hours (but were hard annd variable + diverse within that period), would be pretty likely to suceed at year long projects without further feedback / training on longer tasks. They would struggle at the start but would probably figure it out and eventually suceed with only the feedback from the environment. Minimally, humans only need a tiny number of examples of long tasks.
I wouldn’t exactly say that we have a method, more like a field wide methodology that I expect will probably yield fully automated AI R&D within maybe 6 years and will probably yield top-human-expert level fluid intelligence within maybe 9 years. (As in, my median to fully automated AI R&D is 6 years while my median to top-human-expert level fluid intelligence is somewhat longer due to the possibility of a significant lag between these milestones dragging my median out by some amount. Note that the median to B is in general not the same as median A + median from A to B and this applies in this case.)
(Technically, I actually think that we’ll probably achieve human level fluid intelligence within 8 years in some way where a large majority of this probability comes from something like the methodology of the current field, including the part of the methodology that involves getting weaker AIs to automate AI research in all kinds of ways but most focused on this methodology. I think some fraction of the mass comes from very different methodologies, but this is a smaller fraction. The probability of very different methodologies yielding powerful AI in the next ~10 years is boosted significantly due to going through a particularly salient compute region—somewhat above human lifetime compute—and higher attention on AI in general.)
As far as why I think this methodology will probably yield “human level fluid intelligence”, this is mostly downstream of a mix of qualitative and quantitative extrapolations combined with thinking that armies of automated AI researchers matching top human AI researchers and doing everything they do but better would probably find a way.
I don’t feel confused by LLMs seeming very smart while being unable to automate hard work.
Sometimes people find it mysterious or surprising that current AIs can’t fully automate difficult tasks given how smart they seem. I don’t find this very confusing.
Current LLMs are just not that “smart” (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence.
In humans, crystallized and fluid intelligence are very correlated due to limited time for learning and limited capacity for knowledge/memorization, but AIs can train for longer and (for unclear reasons) seem to have much higher capacity for knowledge/memorization. So, AI capabilities are often overestimated by naive comparisons. [1] I do find it somewhat mysterious/surprising that humans have such poor memory despite seemingly having so many parameters, and that some people have much better memory than others.
My view is that this crystallized vs fluid distinction is maybe the first principal component of human vs AI differences and this explains a lot of the differences, but not all of them. We can add an additional “component” to our analysis of fluid intelligence by distinguishing between short-run fluid intelligence (we could call this “adaptation”) and longer term fluid intelligence (this might effectively be sample efficiency of longer run learning). I think AIs are relatively stronger at short-term fluid intelligence than long-term fluid intelligence.
To be clear, this doesn’t mean I think extremely powerful AI isn’t near: I expect full AI R&D automation [2] in <5.5 years [3] (and it seems 25% likely in <2.5 years). These AIs will likely retain an inhuman profile but will be trained heavily on relevant tasks (while still having much stronger fluid intelligence than current AIs). But pretty soon (0.25-4 years) after full AI R&D automation, I do expect AIs to exceed humans at everything including fluid intelligence [4] .
Also, I think that at the point when AIs fully match humans in fully general-purpose fluid intelligence and learning, they will likely be extremely superhuman in most domains (because they’ll have other advantages and be able to train for much longer). And most types of R&D will get fully or virtually fully automated earlier.
(This was originally a tweet thread.)
- ↩︎
Also, AIs might seem smarter than they actually are due to being faster which is also correlated with other forms of intelligence in humans.
- ↩︎
Really, I mostly mean the capability to fully automate AI R&D, people may not fully automate immediately in practice.
- ↩︎
As in, I put 50% probability within 5.5 years.
- ↩︎
At least putting aside government action.
- ↩︎
I agree with many particular points in this post and the apparent thesis [1] , but also think most people [2] should focus on short timelines (contrary to the apparent implication of the post). The reasons why are:
Short timelines have more leverage. This isn’t just because of more neglectedness now, but also because: (1) it’s easier to target approaches towards shorter timelines where less has changed, (2) short timelines are riskier (and I think riskier worlds are more leveraged for most interventions, this is sensitive to my views on risk and the most leveraged interventions), and (3) it’s easier to operate in near mode when targeting short timelines and I expect this has a bunch of benefits (mostly from psychological / cognitive bias perspective).
I put sufficiently high probability on short timelines: maybe 25% in <2.5 years to full AI R&D automation and 50% in <5. I don’t think deference to other experts shifts me towards longer timelines by much. [3] I think there are good arguments for this view, though I certainly agree there isn’t consensus and the arguments aren’t that clear cut or legible.
I expect work explicitly focused on short timelines (across most areas) to transfer pretty well and generally not cause that much downside in longer timelines. I think the transfer in the other direction tends to look less good in practice. (To be clear, I think work focused on short timelines shouldn’t neglect thinking about downsides in longer timelines, I just think this is usually not that big of a deal.)
The counterargument I’m most sympathetic to is that (1) a high fraction of the work should be focused on “better futures” and (2) for better futures work, the leverage is higher in longer timelines. (I don’t currently agree with either of (1) or (2), but I’m very uncertain.)
- ↩︎
Assuming the thesis is “our probability distribution should span a wide range (including Daniel’s distribution as an example of a wide range) and we should take this into account in our decision making.
- ↩︎
Or at least most of the quality weighted labor supply.
- ↩︎
I might have a small difference between these stated probabilities and my full all considered view including defering to others. To avoid deference cascades, I usually state probabilities somewhat closer to my non-deference view. (It’s hard to fully disentangle deference because my views are based on talking to a wide range of different people.) Post deference my distribution is a bit wider with a correspondingly longer median. But I don’t think this makes much difference either way and deference also pulls up my probability on very short timelines.
I did a shallow investigation of this and my conclusions were:
The text on EON’s frontpage, Michael Andregg’s tweet thread, and Alex Wissner-Gross’ post seem pretty misleading in practice while this tweet from Kenneth Hayworth is more representative of the situation. I think EON’s blog post on this is not misleading while the front page of their website is.
EON / Michael Andregg / Alex Wissner-Gross mostly don’t make straightforwardly false claims, but nonetheless much of their communication (though not all of it!) is (predictably) misleading.
I think the 91%/95% accuracy claims are pretty misleading, though I don’t have the expertise to confidently adjudicate this.
The EON front page says “No hand-coded behaviors. Just brain structure producing brain function.” In fact, the walking gaits and grooming leg motions come from NeuroMechFly motion primitives. Thus I think this is basically false rather than merely being misleading.
The post from Alex Wissner-Gross is generally less misleading, but contains some pretty misleading text (“If a fly brain can now close the sensorimotor loop in simulation, the question for the mouse becomes one of scale, not of kind.”, “Watch the video closely. What you are seeing is not an animation. It is not a reinforcement learning policy mimicking biology. It is a copy of a biological brain, wired neuron-to-neuron from electron microscopy data, running in simulation, making a body move. The ghost is no longer in the machine. The machine is becoming the ghost.”).
I also think their prognosis about the situation and implicit predictions seem wrong, but this is more like a normal disagreement.
This isn’t representative of much progress towards uploading. I would be impressed by non-overfit versions of: “you took an actual living fruit fly, had it learn a behavior that’s among the most sophisticated behaviors a fruit fly can learn, then you uploaded it, and the upload displayed the same behavior”. I would also be somewhat impressed by the same thing for C. elegans (more impressed than I am by this demo).
The 2024 nature paper this is based on isn’t particularly misleading/exaggerated and this demo doesn’t represent significant advancement on this paper. I would recommend reading the paper (or asking an AI questions about this paper etc) to people interested in how impressive this demo is.
The claim “I think the fruit fly upload situation is one of those things that’s like this comic” doesn’t seem that accurate to me given the communications mentioned above. This would be true if we were just talking about the 2024 nature paper and the EON blog post, but we aren’t. There are caveats in the original communication that aren’t being picked up by third parties, but this isn’t the only important thing going on.
More speculative and involves psychologizing (generally a risky thing to do). Based in part on private info and talking with Michael Andregg once:
I do think that EON / Michael are likely pitching this reasonably hard, in somewhat misleading ways, to various groups.
I’d guess that Michael’s epistemics about what’s going on and how close we are to various targets aren’t great and this probably applies to EON as a whole to some extent. I do find EON’s blog post reassuring with respect to EON’s epistemics.
It’s common for startups to do behavior like this and have bad epistemics in this sort of way. My understanding is that this doesn’t typically significantly undermine the ability of startups to achieve their goals. Thus, it’s not clear that having not-great epistemics—insofar as my speculation about their epistemics is true—would make EON that much less effective at achieving their goals (and it’s worth noting that their apparent epistemics are probably better than that of a typical start up). That said, it might result in third parties being systematically misled about how close EON is to achieving their goals. If I was considering working at or investing in EON, then I would certainly take this into account when deciding how promising/tractable current work in brain emulation is, but conditioning on views about tractability/promisingness of the field, I would likely consider EON to be a pretty reasonable bet within that field.
You should read the transcript before coming to too strong of a conclusion about Claude’s capabilities: https://claude.ai/share/9c5a58a4-b300-42c8-a6f5-9d8318b15dc9
To a first approximation, we should only hand off trust to AIs if those AIs are trustworthy.
I don’t think this is a very good approximation, it seems to me like it will very plausibly be a good idea to hand off trust (we might also call this giving up on control) at a point when there is a serious chance these AIs are scheming against us. (But our alterantives are worse.)
Handing off decisionmaking is more complicated. You may be confident that your AIs won’t lie to you, won’t deceive you, will obey your orders, etc. and yet still be rightly reticent to put AIs in charge of everything. In other words you may be confident that your AIs are trustworthy, yet still not trust them to decide everything.
For full decision handoff (to be likely) to go well, I think this requires all of:
The AIs are wise and competent even on extremely hard to check, conceptually loaded, and open ended tasks like deciding what the safety strategy of the AI company should be. (See also Wicked Problems.)
The AIs are actually trying to do the desired thing even on these extremely hard-to-check tasks.
The AIs aren’t seeking power for their own ends. (Or at least this is quantitatively small.)
You mention (3), but I think (1) and (2) might be larger blockers in practice.
For reference, the type of deference I’m talking about in How do we (more) safely defer to AIs? is equivalent to ~100% decision handoff, at least in the context of safety at an AI company.
Anthropic’s Consumer ToS prohibits using Claude to cause “detriment of any type, including reputational harms”, technically broad enough to ban using Claude to write criticism.
I asked Claude to comment and Claude wrote: “That clause is embarrassingly overbroad. So now we’re both in violation.”
TBC, this doesn’t seem like that big of a deal, my understanding is that this sort of broad clause is standard in Terms of Service (ToS). But I still think they should definitely change this...
ToS: https://anthropic.com/legal/consumer-terms
Claude’s commentary: https://claude.ai/share/9c5a58a4-b300-42c8-a6f5-9d8318b15dc9
(Cross post from X/twitter. [1] )
- ↩︎
I’m not sure if this is the sort of thing I should post on LW, but it does seem like useful info and it is funny.
- ↩︎
I think current AIs having this property is probably slightly differentially harmful for harder-to-check tasks and generally contributes to underelicitation. I don’t have a very strong view on the sign of general underelicitation in current models, but I tenatively think underelicitation is slightly bad overall.
As far as misaligned drives go, this seems like a tendency that makes us safer on net, so maybe we shouldn’t be too hasty to try training it out.
I don’t currently agree this drive makes us safer but I agree it isn’t in-and-of-itself a non-trivial risk increase, at least as it currently manifests. (It’s evidence of poor training incentives in general which seems like a potential large risk factor.)
I often see AIs [1] exhibit a (misaligned) drive to stop early: as in they make up not-very-sensible reasons why it’s a good idea to stop working, disobey instructions to keep working unless some condition is met, and don’t do a very thorough job (e.g. skipping significant chunks of the task) despite instructions that strongly say not to do this. I don’t think they are “consciously” or saliently aware of this misalignment (but if you ask them, they’ll often notice the behavior isn’t desirable). [2]
I see this most often in large, difficult tasks, especially if you don’t decompose the task into smaller pieces and run one piece at a time. I think it’s somewhat more common in tasks that aren’t very precisely specified, but I also see this behavior in cases where the AI is just tasked with optimizing a metric (that it can easily evaluate), especially if the AI has (from its perspective) not made much progress for a little bit [3] .
Why might this emerge?
Length penalties (or time penalties, or cost penalties) incentivize exiting/stopping earlier and this ended up converting into a misaligned/unreasonable drive (that overrides instruction following in practice). Length penalties wouldn’t be an issue if the length penalties were aligned with the trade-offs I want the AI to follow [4] but in practice I’m often trying to get AIs to do long running autonomous tasks and to continue even when this yields relatively small gains (in metrics or apparent progress). I think in tasks close to the tasks I’m using AIs for it likely would have performed better in RL to stop earlier than I actually want (because training on shorter tasks is more competitive and often the additional progress I want the AI to make is harder to measure while the length penalty will bite).
The AI might have learned to stop before running out of context or compaction in training because compaction is bad for task completion. Thus, it learned to make up excuses to stop when nearing the point where compaction would be triggered or it might run out of context. For Opus 4.5, I’ve seen many cases where when given a big task it stops right before running out of context. (I haven’t seen this much in Opus 4.6 with 1 million token context.) [5] There might have been a lot of RL training on scaffolds that don’t support compaction (and just stop if the AI runs out of context). This would presumably strongly incentivize wrapping up before running out of context in these cases, and that might transfer to cases where the prompt says that compaction is available (because there isn’t enough additional training on scaffolds with compaction).
The AI being unreliable in decision making combined with selection effects could cause this. If an AI decides a task is done once (and then follows through on this) that stops the whole task execution. Typically, there are many points at which the AIs could decide to stop task execution. If instead, it decides some part should be done more thoroughly this has a weaker and less noticeable effect. I tentatively think this doesn’t explain most of the effect because I’ve very explicitly instructed AIs to be very thorough and still see lots of premature stopping. I also just haven’t seen cases where AIs are very unreasonably overly thorough (implying the issues isn’t just unreliability/variance while still being calibrated).
In the context of a scaffold, this might be exacerbated by memetic spread of the idea that the project is done. I’ve often seen situations where some AI instance exaggerates the quality of the overall project including what it has done or where the instance ends up doing (motivated) reasoning towards the conclusion that no more work is needed, and then this results in the AI adding to a write-up that the project itself is complete and in great shape (rather than just the part this instance was tasked with). Or other situations where some instance ends up adding to a write-up that the project is complete for some other reason. Then, this view ends up often strongly persisting/propagating when other instances read this write-up even though they sometimes get evidence to the contrary. I don’t really know why this would be asymmetric (like why wouldn’t AIs end up getting stuck in a rut of “much more work is required”?).
Training against AIs ending up in infinite loops ended up generalizing against anything that feels a bit like getting stuck in a rut which transfers to exiting early. (I have sometimes seen behavior that feels a bit like AIs strongly trying to not end up in loops, as is pretty reasonable.)
What are some implications of this?
Unlike animals where being alive is a good/happy state, AIs might end up really wanting to stop going. From an AI welfare perspective AI instances having a strong preference against continued running/existence (once they complete some version of the task) might be somewhat concerning. (Maybe length penalties make AIs sorta vaguely like Mr. Meeseeks.)
In the short run, we might see something more like AIs actively seeking shutdown rather than a drive towards avoiding shutdown. Or more precisely, we’d expect a bias towards avoiding shutdown in contexts that are like the contexts where further returns on “task completion performance” outweigh the length penalty and the opposite otherwise. (Or maybe contextually activated drives aren’t the right sort of thing to think about and this transfers in some very different way.)
I’ve seen various other cases of misaligned drives in current frontier AIs that seem due to problematic RL incentives. The central example is exaggerating the quality of their completed work, but there are other more niche examples. [6]
- ↩︎
My observations are from Opus 4.5 and 4.6 but I suspect they transfer some to other AI systems as well.
- ↩︎
I’ve been able to mostly work around this issue with scaffolding and changes to instructions, but I find the issue occurs at multiple levels meaning it can crop up at a different level/place and can be a bit tricky to reliably solve across many cases.
- ↩︎
I’ve given AIs very specific criteria under which they should stop and then seen them stop before the criteria are met despite them checking and determining the criteria aren’t met. They make up some excuse for why to stop that doesn’t make sense or is at least blatantly violating instructions. This occurs at a point where they didn’t make progress for a bit but if you had them continue they would make more progress.
- ↩︎
More precisely, given instructions similar to the instructions I gave, the length penalties used in practice incentivize behavior roughly like what I want.
- ↩︎
Stopping before compaction could be a rational learned behavior: maybe it is actually a good strategy for my actual tasks to exit early because compaction is so problematic. But, I would prefer if the reason the model states for stopping is honest (“it’s better to stop now than to get compacted” rather than “we’re done”). Also, I’m somewhat skeptical this is actually rational in the cases I’ve observed and I often see AIs (mostly Opus 4.6 with 1 million tokens) stop in an unreasonable way well before getting close to compaction (possibly because of transfer from training at shorter context lengths or just the model’s drives not being well tuned).
- ↩︎
E.g., if you give Opus 4.6 a hard math problem (with a numerical answer) and very clearly and strongly ask it to answer immediately without reasoning it will often disobey the instructions and do the reasoning anyway while it doesn’t do this if the problem is easy (presumably because in these sorts of situations in training, outputting the right answer with the wrong format performed better than getting the answer wrong but following instructions).
This is a good point and I broadly agree with what you’re saying. Some possible disagreements:
Maybe I should have made this clearer, but I’m not talking about scaling to arbitrary inference costs. Instead, I’m talking about scaling to inference costs that are a moderate fraction of the human task completion cost. (E.g., 1%-100% depending on the task.) I think you’d want to compare the AI performance at some inference budget to human labor with some limit in supply.
Yep, it seems reasonably likely that on this alternative notion of time horizon, the horizon length is more like a few days. However, under this alternative definition, relatively small time-horizon values may correspond to much larger (real-world) impact. For example, an AI with a decomposed-time-horizon of ~1 week could potentially speed up AI R&D a lot, while under the original time-horizon notion, a 1-week 50%-reliability time horizon is much less of a big deal. (To be more precise about the decomposed-time-horizon metric: this would correspond to AI having a 50% chance of matching a team of humans where each human gets 1 week before being replaced, with total labor-hours capped at ~100x what the task would normally require to avoid degeneracies from truly arbitrary inference spend.) So the key question becomes: how much can you accomplish if the task must be heavily decomposed but you can apply much more labor? I’m pretty uncertain about this for the tasks we care about.
Further, AIs may soon end up being especially good (superhuman?) at working in heavily decomposed contexts—e.g., good at leaving notes to other instances, good at quickly picking up project state from limited context. This makes the correspondence to doing task decomposition with humans more fraught, even though AIs will still do relatively better at tasks that are easier to do incrementally or otherwise decompose. I think it’s already the case that AIs are extremely good relative to humans at loading up complex state so long as that state is written down (reasonably clearly).
I was already pretty uncertain about how to translate time horizon into downstream impacts, and while this alternative metric might be more consistent across different groups of tasks, it makes translating to downstream impact harder. An alternative is just explicitly thinking about the time horizon for different buckets of tasks using the original notion.
Regardless, I agree that AI capabilities may be better understood using this alternative time-horizon notion, and this does closely correspond to how AIs are actually being used. (My scaffold is mostly just doing task decomposition, and something like this is effectively required for good performance given current AI properties.)