FWIW I’m also bearish on LLMs but for reasons that are maybe subtly different from OP. I tend to frame the issue in terms of “inability to deal with a lot of interconnected layered complexity in the context window”, which comes up when there’s a lot of idiosyncratic interconnected ideas in one’s situation or knowledge that does not exist on the internet.
This issue incidentally comes up in “long-horizon agency”, because if e.g. you want to build some new system or company or whatever, you usually wind up with a ton of interconnected idiosyncratic “cached” ideas about what you’re doing and how, and who’s who, and what’s what, and what are the idiosyncratic constraints and properties and dependencies in my specific software architecture, etc. The more such interconnected bits of knowledge that I need for what I’m doing—knowledge which is by definition not on the internet, and thus must be in the context window instead—the more I expect foundation models to struggle on those tasks, now and forever.
But that problem is not exactly the same as a problem with long-horizon agency per se. I would not be too surprised or updated by seeing “long-horizon agency” in situations where, every step along the way, pretty much everything you need to know to proceed, is on the internet.
More concretely, suppose there’s a “long-horizon task” that’s very tag-team-able—i.e., Alice has been doing the task, but you could at any moment fire Alice, take a new smart generalist human off the street, Bob, and then Bob would be able continue the task smoothly after very little time talking to Alice or scrutinizing Alice’s notes and work products. I do think there are probably tag-team-able “long-horizon tasks” like that, and expect future foundation models to probably be able to do those tasks. (But I don’t think tag-team-able long-horizon tasks are sufficient for TAI.)
This is also incidentally how I reconcile “foundation models do really well on self-contained benchmark problems, like CodeForces” with “foundation models are not proportionally performant on complex existing idiosyncratic codebases”. If a problem is self-contained, that puts a ceiling on the amount of idiosyncratic layered complexity that needs to be piled into the context window.
Humans, by comparison, are worse than foundation models at incorporating idiosyncratic complexity in the very short term (seconds and minutes), but the sky’s the limit if you let the human gain familiarity with an idiosyncratic system or situation over the course of days, weeks, months.
I also wouldn’t be too surprised if in some domains RL leads to useful agents if all the individual actions are known to and doable by the model and RL teaches it how to sensibly string these actions together. This doesn’t seem too different from mathematical derivations.
I think that getting good at the tag-teamable tasks is already enough to start to significantly accelerate AI R&D? Idk. I don’t really buy your distinction/abstraction yet enough to make it an important part of my model.
I think it won’t work (and isn’t working today) for the same reasons John outlines here with regards to HCH/”the infinite bureaucracy”. (tl;dr: this requires competent problem factorization, but problem factorization is nontrivial and can’t be relegated to an afterthought.)
Thinking about this, I think a generalized crux with John Wentworth et al is probably on how differently we see bureaucracies, and he sees them as terrible, whereas I see them as both quite flawed and has real problems, but are also wonderful tools to have that keeps the modern civilization’s growth engine stable, and the thing that keeps the light on, so I see bureaucracies as way more important for civilization’s success than John Wentworth believes.
One reason for this is a lot of the success cases of bureaucracies look like no news can be made, so success isn’t obvious, whereas bureaucratic failure is obvious.
I think the difference between real bureaucracies and HCH is that in real functioning bureaucracies should be elements capable to say “screw this arbitrary problem factorization, I’m doing what’s useful” and bosses of bureaucracy should be able to say “we all understand that otherwise system wouldn’t be able to work”.
I tend to frame the issue in terms of “inability to deal with a lot of interconnected layered complexity in the context window”
I think that’s also equivalent to my “remaining on-target across long inferential distances” / “maintaining a clear picture of the task even after its representation becomes very complex in terms of the templates you had memorized at the start”.
But that problem is not exactly the same as a problem with long-horizon agency per se
That’s a fair point, but how many real-life long-horizon-agency problems are of the “clean” type you’re describing?
An additional caveat here is that, even if the task is fundamentally “clean”/tag-team-able, you don’t necessarily know that when working on it. Progressing along it would require knowing what information to discard and what to keep around at each step, and that’s itself nontrivial and might require knowing how to deal with layered complexity.
(Somewhat relatedly, see those thoughts regarding emergent complexity. Even if a given long-horizon-agency task is clean thin line when considered from a fully informed omniscient perspective – a perspective whose ontology is picked to make the task’s description short – that doesn’t mean the bounded system executing the task can maintain a clean representation of it every step of the way.)
FWIW I’m also bearish on LLMs but for reasons that are maybe subtly different from OP. I tend to frame the issue in terms of “inability to deal with a lot of interconnected layered complexity in the context window”, which comes up when there’s a lot of idiosyncratic interconnected ideas in one’s situation or knowledge that does not exist on the internet.
This issue incidentally comes up in “long-horizon agency”, because if e.g. you want to build some new system or company or whatever, you usually wind up with a ton of interconnected idiosyncratic “cached” ideas about what you’re doing and how, and who’s who, and what’s what, and what are the idiosyncratic constraints and properties and dependencies in my specific software architecture, etc. The more such interconnected bits of knowledge that I need for what I’m doing—knowledge which is by definition not on the internet, and thus must be in the context window instead—the more I expect foundation models to struggle on those tasks, now and forever.
But that problem is not exactly the same as a problem with long-horizon agency per se. I would not be too surprised or updated by seeing “long-horizon agency” in situations where, every step along the way, pretty much everything you need to know to proceed, is on the internet.
More concretely, suppose there’s a “long-horizon task” that’s very tag-team-able—i.e., Alice has been doing the task, but you could at any moment fire Alice, take a new smart generalist human off the street, Bob, and then Bob would be able continue the task smoothly after very little time talking to Alice or scrutinizing Alice’s notes and work products. I do think there are probably tag-team-able “long-horizon tasks” like that, and expect future foundation models to probably be able to do those tasks. (But I don’t think tag-team-able long-horizon tasks are sufficient for TAI.)
This is also incidentally how I reconcile “foundation models do really well on self-contained benchmark problems, like CodeForces” with “foundation models are not proportionally performant on complex existing idiosyncratic codebases”. If a problem is self-contained, that puts a ceiling on the amount of idiosyncratic layered complexity that needs to be piled into the context window.
Humans, by comparison, are worse than foundation models at incorporating idiosyncratic complexity in the very short term (seconds and minutes), but the sky’s the limit if you let the human gain familiarity with an idiosyncratic system or situation over the course of days, weeks, months.
I think that is exactly right.
I also wouldn’t be too surprised if in some domains RL leads to useful agents if all the individual actions are known to and doable by the model and RL teaches it how to sensibly string these actions together. This doesn’t seem too different from mathematical derivations.
I think that getting good at the tag-teamable tasks is already enough to start to significantly accelerate AI R&D? Idk. I don’t really buy your distinction/abstraction yet enough to make it an important part of my model.
I think it won’t work (and isn’t working today) for the same reasons John outlines here with regards to HCH/”the infinite bureaucracy”. (tl;dr: this requires competent problem factorization, but problem factorization is nontrivial and can’t be relegated to an afterthought.)
Thinking about this, I think a generalized crux with John Wentworth et al is probably on how differently we see bureaucracies, and he sees them as terrible, whereas I see them as both quite flawed and has real problems, but are also wonderful tools to have that keeps the modern civilization’s growth engine stable, and the thing that keeps the light on, so I see bureaucracies as way more important for civilization’s success than John Wentworth believes.
One reason for this is a lot of the success cases of bureaucracies look like no news can be made, so success isn’t obvious, whereas bureaucratic failure is obvious.
I think the difference between real bureaucracies and HCH is that in real functioning bureaucracies should be elements capable to say “screw this arbitrary problem factorization, I’m doing what’s useful” and bosses of bureaucracy should be able to say “we all understand that otherwise system wouldn’t be able to work”.
I think that’s also equivalent to my “remaining on-target across long inferential distances” / “maintaining a clear picture of the task even after its representation becomes very complex in terms of the templates you had memorized at the start”.
That’s a fair point, but how many real-life long-horizon-agency problems are of the “clean” type you’re describing?
An additional caveat here is that, even if the task is fundamentally “clean”/tag-team-able, you don’t necessarily know that when working on it. Progressing along it would require knowing what information to discard and what to keep around at each step, and that’s itself nontrivial and might require knowing how to deal with layered complexity.
(Somewhat relatedly, see those thoughts regarding emergent complexity. Even if a given long-horizon-agency task is clean thin line when considered from a fully informed omniscient perspective – a perspective whose ontology is picked to make the task’s description short – that doesn’t mean the bounded system executing the task can maintain a clean representation of it every step of the way.)