“Secondarily, current models don’t operate for long enough (or on hard enough problems) for these convergent instrumental incentives to be very strong.”
I’m worried that when it comes to Claude Code, this is not a base capabilities problem, but an elicitation one. It feels very plausible to me that with the correct harness, you could actually get an assemblage that is capable of arbitrary long horizon work.
Like—do humans actually have a long time horizon? The Basic Rest-Activity Cycle suggests we work in ~90 minute bursts at most. If true, then the base models are already there. All we would need is a way to mimic or substitute the cognitive scaffolding that we use to pull off our own arbitrary time horizons.
I’m envisioning an “assembly line” of cognitive labor—if Claude Code has 30-90 minute time horizons on every task natively, then you figure out a way to have it orchestrate the chopping up of arbitrary tasks into bits that dedicated subagents can do. Then you add a suite of epistemic tools, permanent memory, version control, reflexes; self-improvement via self-reflection, objective tests, and external research; the ability to build out its own harness; and so on: all the individual ingredients needed to get a bootstrapping functional cognitive laborer. [Edit: I’m very concerned this is an infohazard!!!]
Do we know that this can’t work right now? Has anyone really tried? I’ve been looking and I don’t see anyone doing this to the maximal extent I’m envisioning, but it seems inevitable that either someone will, either in the hacker community or at the frontier labs. I’m working on putting something like this together but, thinking about it, I’m really worried I shouldn’t.
Surely when I come back to work after a 20 minute break I can regain much more context much more effectively than if Claude runs out of context window, and the context is compacted for the next Claude instance.
In conversations about this that I’ve seen the crux is usually:
Do you expect greater capabilities increases per dollar from [continued scaling by ~the current* techniques] or by some [scaffolding/orchestration scheme/etc].
The latter just isn’t very dollar efficient, so I think we’d have to see the existing [ways to spend money to get better performance] get more expensive / hit a serious wall before sufficient resources are put into this kind of approach. It may be cheap to try, but verifying performance on relevant tasks and iterating on the design gets really expensive really quickly. On the scale-to-schlep spectrum, this is closer to schlep. I think you’re right that something like this could be important at some point in the future, conditional on much less efficient returns from other methods.
This is a bit of a side note, but I think your human analogy for time horizons doesn’t quite work, as Eli said. The question is ‘how much coherent and purposeful person-minute-equivalent-doing can an LLM execute before it fails n percent of the time?’ Many person-years of human labor can be coherently oriented toward a single outcome (whether it’s one or many people involved). That the humans get sleepy or distracted in-between is an efficiency concern, not a coherence concern; it affects the rate at which the work gets done, but doesn’t put an upper bound on the total amount of purposeful labor that can hypothetically be directed, since humans just pick up where they left off pursuing the same goals for years and years at a time, while LLMs seem to lose the plot once they’ve started to nod off.
While the particulars of your argument seem to me to have some holes, I actually very much agree with your observation we don’t know what the upper limit of properly orchestrated Claude instances are, and that targeted engineering of Claude-compatible cognitive tools could vastly increase its capabilities.
One idea I’ve been playing with for a really long time is that the Claudes aren’t the actual agents, but instead just small nodes or subprocesses in a higher-functioning mind. If I loosely imagine a hierarchy of Claudes, each corresponding roughly to system-1 or subconscious deliberative processes, with the ability to write and read to files as a form of “long term memory/processing space” for the whole system, and I imagine that by some magical oracle process they coordinate/delegate as well as Claudes possibly can, subject to a vague notion of “how smart Claude itself is”, I see no reason a system like this can’t already be an AGI, and cannot in principle be engineered into existence using contemporary LLMs.
(However, I will say that this thing sounds pretty hard to actually engineer, i.e, it being “just an engineering problem” doesn’t mean it would happen soon, but OTOH maybe it could if people would try the right approach hard enough. I can’t imagine a clean way of applying optimization pressure to the Claudes in any such setup that isn’t an extremely expensive and reward-sparse form of RL.)
“Secondarily, current models don’t operate for long enough (or on hard enough problems) for these convergent instrumental incentives to be very strong.”
I’m worried that when it comes to Claude Code, this is not a base capabilities problem, but an elicitation one. It feels very plausible to me that with the correct harness, you could actually get an assemblage that is capable of arbitrary long horizon work.
Like—do humans actually have a long time horizon? The Basic Rest-Activity Cycle suggests we work in ~90 minute bursts at most. If true, then the base models are already there. All we would need is a way to mimic or substitute the cognitive scaffolding that we use to pull off our own arbitrary time horizons.
I’m envisioning an “assembly line” of cognitive labor—if Claude Code has 30-90 minute time horizons on every task natively, then you figure out a way to have it orchestrate the chopping up of arbitrary tasks into bits that dedicated subagents can do. Then you add a suite of epistemic tools, permanent memory, version control, reflexes; self-improvement via self-reflection, objective tests, and external research; the ability to build out its own harness; and so on: all the individual ingredients needed to get a bootstrapping functional cognitive laborer. [Edit: I’m very concerned this is an infohazard!!!]
Do we know that this can’t work right now? Has anyone really tried? I’ve been looking and I don’t see anyone doing this to the maximal extent I’m envisioning, but it seems inevitable that either someone will, either in the hacker community or at the frontier labs. I’m working on putting something like this together but, thinking about it, I’m really worried I shouldn’t.
Surely when I come back to work after a 20 minute break I can regain much more context much more effectively than if Claude runs out of context window, and the context is compacted for the next Claude instance.
In conversations about this that I’ve seen the crux is usually:
The latter just isn’t very dollar efficient, so I think we’d have to see the existing [ways to spend money to get better performance] get more expensive / hit a serious wall before sufficient resources are put into this kind of approach. It may be cheap to try, but verifying performance on relevant tasks and iterating on the design gets really expensive really quickly. On the scale-to-schlep spectrum, this is closer to schlep. I think you’re right that something like this could be important at some point in the future, conditional on much less efficient returns from other methods.
This is a bit of a side note, but I think your human analogy for time horizons doesn’t quite work, as Eli said. The question is ‘how much coherent and purposeful person-minute-equivalent-doing can an LLM execute before it fails n percent of the time?’ Many person-years of human labor can be coherently oriented toward a single outcome (whether it’s one or many people involved). That the humans get sleepy or distracted in-between is an efficiency concern, not a coherence concern; it affects the rate at which the work gets done, but doesn’t put an upper bound on the total amount of purposeful labor that can hypothetically be directed, since humans just pick up where they left off pursuing the same goals for years and years at a time, while LLMs seem to lose the plot once they’ve started to nod off.
While the particulars of your argument seem to me to have some holes, I actually very much agree with your observation we don’t know what the upper limit of properly orchestrated Claude instances are, and that targeted engineering of Claude-compatible cognitive tools could vastly increase its capabilities.
One idea I’ve been playing with for a really long time is that the Claudes aren’t the actual agents, but instead just small nodes or subprocesses in a higher-functioning mind. If I loosely imagine a hierarchy of Claudes, each corresponding roughly to system-1 or subconscious deliberative processes, with the ability to write and read to files as a form of “long term memory/processing space” for the whole system, and I imagine that by some magical oracle process they coordinate/delegate as well as Claudes possibly can, subject to a vague notion of “how smart Claude itself is”, I see no reason a system like this can’t already be an AGI, and cannot in principle be engineered into existence using contemporary LLMs.
(However, I will say that this thing sounds pretty hard to actually engineer, i.e, it being “just an engineering problem” doesn’t mean it would happen soon, but OTOH maybe it could if people would try the right approach hard enough. I can’t imagine a clean way of applying optimization pressure to the Claudes in any such setup that isn’t an extremely expensive and reward-sparse form of RL.)