Thank you! You’re right. Another point is that intelligence and agency are independent, and a tool AI can be (much) more intelligent than an agentic one.
Depends on what you mean by “independent.” I didn’t disagree or agree with the statement because I’m not sure how to interpret it. If it’s “probabilistically independent” or “uncorrelated” or something like that, it’s false; if it’s “logically independent” i.e. “there’s nothing logically contradictory about something which has lots of one property but little of the other” then it’s true.
I assumed it was the second, but the first is even more interesting! Using a naive count social insects will largely dominate the distribution, in which case intelligence will indeed be (negatively!) correlated with agency. But it’s probably not your reasoning, right? How do you construct/see the distribution of intelligent entities so that you see a positive correlation?
I don’t really care about what’s going on at very low levels of intelligence (insects) and I also don’t care about population sizes, I care about plausibility/realism/difficulty-of-construction/stufflikethat. (If I cared about population sizes, I’d say the distribution was dominated by rocks, which have 0 agency and 0 intelligence, and then all creatures (which-have-some-agency-and-some-intelligence) form a cluster up and to the right of rocks, and thus the correlation is positive.) But out of curiosity what are you thinking there—are you thinking that the smarter insects are less agentic, the more agentic insects are smarter?
Intelligence (which I’m guessing we are thinking of as world-representation / world-understanding / prediction-engine / etc.?) and Agency (goal-orientedness / coherence / ability-to-take-long-sequences-of-action-that-P2B) are different things but they are closely related:
As a practical matter, the main ways of achieving either of them involves procedures that get you both. (e.g. training a neural net to accomplish diverse difficult tasks is training a neural net to be highly agentic, but it’s going to end up developing some pretty good world representations in the process. Training a neural net to predict stuff (e.g. LLMs) does indeed mostly just give them intelligence and not agency, though in the limit of infinite training of this sort (especially if the stuff being predicted was action-sequences of previous versions of itself) it would get you agency anyway. It also gets you a system which can very easily be turned into an agent (see: AutoGPT) and there is competitive pressure to do so.
Related readings which I generally recommend to anyone interested in this topic (h/t Ben Pace for the list):
I don’t really care about what’s going on at very low levels of intelligence (insects)
Well, yes, removing (low X, high Y) points is one way to make correlation coefficient positive, but then you shouldn’t trust any conclusion based on that (or, more precisely, you shouldn’t update based on that). Idem if your data form clusters.
… and my very own Agency: What it is and why it matters—LessWrong
Thanks, very helpful! Yes, we agree that, once we define agency as basically the ability to represent and act on plans, and each level in agency as one type of what I’d call cognitive strategies, then the more intelligence the more agency.
But is that definition useful enough? I’ll have to read the other links to be fair, but what’s your best three arguments for the universality of this definition? Or at least why you think it should apply to computer programs and human-made robots?
But out of curiosity what are you thinking there—are you thinking that the smarter insects are less agentic, the more agentic insects are smarter?
Well the good thing is we don’t need to think that much, we can just read the literature. The behaviors of social insects that appear the most agentic (using man on the street feeling rather than your specialized definition) are collective behaviors: they can go to war, enslave their victims, raise cattle, sail (arguably), decide to emigrate en masse, choose the best among candidate locations, etc. The following paper explains quite well that this does not rely on individual intelligence, but on coordination among individuals: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5226334/
Now, if you decide to exclude social insects from your definition of agency or intelligence, then I think you’re also at risk of missing what I see as one of the main present danger: collective stupidity emerging from our use of social networks. Imagine if covid had turned out to be as dangerous as ebola. We wouldn’t have to care about our civilisation being too powerful for its own good, at least for a while.
What’s your response to my “If I did...” point? If we include all the data points, the correlation between intelligence and agency is clearly positive, because rocks have 0 intelligence and 0 agency.
If you agree that agency as I’ve defined it in that sequence is closely and positively related to intelligence, then maybe we don’t have anything else to disagree about. I would then ask of you and Boaz what other notion of agency you have in mind, and encourage you to specify it to avoid confusion, and then maybe that’s all I’d say since maybe we’d be in agreement.
I am not excluding social insects from my definition of agency or intelligence. I think ants are quite agentic and also quite intelligent.
I do disagree that collective stupidity from our use of social networks is our main present danger; I think it’s sorta a meta-danger, in that if we could solve it maybe we’d solve a bunch of our other problems too, but it’s only dangerous insofar as it it leads to those other problems, and some of those other problems are really pressing… analogy: “Our biggest problem is suboptimal laws. If only our laws and regulations were optimal, all our other problems such as AGI risk would go away.” This is true, but… yeah it seems less useful to focus on that problem, and more useful to focus on more first-order problems and how our laws can be changed to address them.
Sorry, that was the « Idem if your data forms clusters ». In other words, I agree a cluster to (0,0) and a cluster to (+,+) will turn into positive correlation coefficients, and I warn you against updating based on that (it’s a statistical mistake).
If you agree that agency as I’ve defined it in that sequence is closely and positively related to intelligence, then maybe we don’t have anything else to disagree about.
I respectfully disagree with the idea that most disagreements comes from making different conclusion based on the same priors. Most disagreements I have with anyone on LessWrong (and anywhere, really) is about what priors and prior structures are best for what purpose. In other words, I fully agree that
I would then ask of you and Boaz what other notion of agency you have in mind, and encourage you to specify it to avoid confusion, and then maybe that’s all I’d say since maybe we’d be in agreement.
Speaking for myself only, my notion of agency is basically « anything that behaves like an error-correcting code ». This includes conscious beings that want to promote their fate, but also life who want to live, and even two thermostats fighting over who’s in charge.
I do disagree that collective stupidity from our use of social networks is our main present danger; I think it’s sorta a meta-danger, in that if we could solve it maybe we’d solve a bunch of our other problems too, but it’s only dangerous insofar as it it leads to those other problems, and some of those other problems are really pressing...
That and the analogy are very good points, thank you.
I don’t dispute that you can build agent AIs, and that they can be useful.
I don’t claim that it is possible to get the same economic benefits by restricting to tool AIs. Indeed, in my previous post with Edelman, we explicitly said that we do consider AIs that are agentic in the sense that they can take action, including self-driving, writing code, executing trades etc..
I don’t dispute that one way to build those is to take a next-token predictor such as pretrained GPT3, and then use fine-tuning, RHLF, prompt engineering or other methods to turn it into an agent AI. (Indeed, I explicitly say so in the current post.)
My claim is that it is a useful abstraction to (1) separate intelligence from agency, and (2) intelligence in AI is a monotone function of the computational resources (FLOPs, data, model size, etc.) invested into building the model.
Now if you want to take 3.6 Trillion gradient steps in a model, then you simply cannot do it by having it take actions and wait to get some reward. So I do claim that if we buy the scaling hypothesis that intelligence scales with compute, the bulk of the intelligence of models such as GPT-n, PALM-n, etc. comes from the non agentic next-token predictor.
So, I believe it is useful and more accurate to think of (for example) a stock trading agent that is built on top of GPT-4 as consisting of an “intelligence forklift” which accounts for 99.9% of the computational resources, plus various layers of adaptations, including supervised fine-tuning, RL from human feedback, and prompt engineering, to obtain the agent.
The above perspective does not mean that the problem of AI safety or alignment is solved. But I do think it is useful to think of intelligence as belonging to a system rather than an individual agent, and (as discussed briefly above) that considering it in this way changes somewhat the landscape of both problems and solutions.
Ah. Well, if that’s what you are saying then you are preaching to the choir. :) See e.g. the “pretrained LLMs are simulators / predictors / oracles” discourse on LW.
I feel like there is probably still some disagreement between us though. For example I think the “bulk of the intelligence comes from non-agentic next-token predictor” claim you make probably is either less interesting or less true than you think it is, depending on what kinds of conclusions you think follow from it. If you are interested in discussing more sometime I’d be happy to have a video call!
Agree that we still disagree and (in my biased opinion) that claim is either more interesting or more true than you realize :)
Not free for a call soon but hope eventually there is an opportunity to discuss more.
I was asked about this on Twitter. Gwern’s essay deserves a fuller response than a comment but I’m not arguing for the position Gwern argues against.
I don’t argue that agent AI are not useful or won’t be built. I am not arguing that humans must always be in the loop.
My argument is that tool vs agent AI is not so much about competition but specialization. Agent AIs have their uses but if we consider the “deep learning equation” of turning FLOPs into intelligence, then it’s hard to beat training for predictions on static data. So I do think that while RL can be used forAI agents, the intelligence “heavy lifting” (pun intended) would be done by non-agentic tool but very large static models.
Even “hybrid models” like GPT3.5 can best be understood as consisting of an “intelligence forklift”—the pretrained next-token predictor on which 99.9% of the FLOPs were spent on building—and an additional light “adapter” that turns this forklift into a useful Chatbot etc
I think part of your point, translated to local language is “GPTs are Tool AIs, and Tool AI doesn’t necessarily become agentic”
Thank you! You’re right. Another point is that intelligence and agency are independent, and a tool AI can be (much) more intelligent than an agentic one.
well, that’s certainly a state of the world that things can be in.
temporarily.
To anyone who disagree with that, I’d be curious to see where you place social insects and LLMs on the intelligence vs agency graph. 🤔
Depends on what you mean by “independent.” I didn’t disagree or agree with the statement because I’m not sure how to interpret it. If it’s “probabilistically independent” or “uncorrelated” or something like that, it’s false; if it’s “logically independent” i.e. “there’s nothing logically contradictory about something which has lots of one property but little of the other” then it’s true.
I assumed it was the second, but the first is even more interesting! Using a naive count social insects will largely dominate the distribution, in which case intelligence will indeed be (negatively!) correlated with agency. But it’s probably not your reasoning, right? How do you construct/see the distribution of intelligent entities so that you see a positive correlation?
I don’t really care about what’s going on at very low levels of intelligence (insects) and I also don’t care about population sizes, I care about plausibility/realism/difficulty-of-construction/stufflikethat. (If I cared about population sizes, I’d say the distribution was dominated by rocks, which have 0 agency and 0 intelligence, and then all creatures (which-have-some-agency-and-some-intelligence) form a cluster up and to the right of rocks, and thus the correlation is positive.) But out of curiosity what are you thinking there—are you thinking that the smarter insects are less agentic, the more agentic insects are smarter?
Intelligence (which I’m guessing we are thinking of as world-representation / world-understanding / prediction-engine / etc.?) and Agency (goal-orientedness / coherence / ability-to-take-long-sequences-of-action-that-P2B) are different things but they are closely related:
As a practical matter, the main ways of achieving either of them involves procedures that get you both. (e.g. training a neural net to accomplish diverse difficult tasks is training a neural net to be highly agentic, but it’s going to end up developing some pretty good world representations in the process. Training a neural net to predict stuff (e.g. LLMs) does indeed mostly just give them intelligence and not agency, though in the limit of infinite training of this sort (especially if the stuff being predicted was action-sequences of previous versions of itself) it would get you agency anyway. It also gets you a system which can very easily be turned into an agent (see: AutoGPT) and there is competitive pressure to do so.
Related readings which I generally recommend to anyone interested in this topic (h/t Ben Pace for the list):
Gwern’s analysis of why Tool AIs want to become Agent AIs. It contains references to many other works. (It also has a positive review from Eliezer.)
Eric Drexler’s incredibly long report on Comprehensive AI Services as General Intelligence. Related: Rohin Shah’s summary, and Richard Ngo’s comment.
The Goals and Utility Functions chapter of Rohin Shah’s Value Learning sequence.
Eliezer’s very well-written post on Arbital addressing the question “Why Expected Utility?” [Added: crossposted to LessWrong here]
The LessWrong wiki article on Tool AI links to several posts on this topic.
… and my very own Agency: What it is and why it matters—LessWrong
Well, yes, removing (low X, high Y) points is one way to make correlation coefficient positive, but then you shouldn’t trust any conclusion based on that (or, more precisely, you shouldn’t update based on that). Idem if your data form clusters.
Thanks, very helpful! Yes, we agree that, once we define agency as basically the ability to represent and act on plans, and each level in agency as one type of what I’d call cognitive strategies, then the more intelligence the more agency.
But is that definition useful enough? I’ll have to read the other links to be fair, but what’s your best three arguments for the universality of this definition? Or at least why you think it should apply to computer programs and human-made robots?
Well the good thing is we don’t need to think that much, we can just read the literature. The behaviors of social insects that appear the most agentic (using man on the street feeling rather than your specialized definition) are collective behaviors: they can go to war, enslave their victims, raise cattle, sail (arguably), decide to emigrate en masse, choose the best among candidate locations, etc. The following paper explains quite well that this does not rely on individual intelligence, but on coordination among individuals: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5226334/
Now, if you decide to exclude social insects from your definition of agency or intelligence, then I think you’re also at risk of missing what I see as one of the main present danger: collective stupidity emerging from our use of social networks. Imagine if covid had turned out to be as dangerous as ebola. We wouldn’t have to care about our civilisation being too powerful for its own good, at least for a while.
What’s your response to my “If I did...” point? If we include all the data points, the correlation between intelligence and agency is clearly positive, because rocks have 0 intelligence and 0 agency.
If you agree that agency as I’ve defined it in that sequence is closely and positively related to intelligence, then maybe we don’t have anything else to disagree about. I would then ask of you and Boaz what other notion of agency you have in mind, and encourage you to specify it to avoid confusion, and then maybe that’s all I’d say since maybe we’d be in agreement.
I am not excluding social insects from my definition of agency or intelligence. I think ants are quite agentic and also quite intelligent.
I do disagree that collective stupidity from our use of social networks is our main present danger; I think it’s sorta a meta-danger, in that if we could solve it maybe we’d solve a bunch of our other problems too, but it’s only dangerous insofar as it it leads to those other problems, and some of those other problems are really pressing… analogy: “Our biggest problem is suboptimal laws. If only our laws and regulations were optimal, all our other problems such as AGI risk would go away.” This is true, but… yeah it seems less useful to focus on that problem, and more useful to focus on more first-order problems and how our laws can be changed to address them.
Sorry, that was the « Idem if your data forms clusters ». In other words, I agree a cluster to (0,0) and a cluster to (+,+) will turn into positive correlation coefficients, and I warn you against updating based on that (it’s a statistical mistake).
I respectfully disagree with the idea that most disagreements comes from making different conclusion based on the same priors. Most disagreements I have with anyone on LessWrong (and anywhere, really) is about what priors and prior structures are best for what purpose. In other words, I fully agree that
Speaking for myself only, my notion of agency is basically « anything that behaves like an error-correcting code ». This includes conscious beings that want to promote their fate, but also life who want to live, and even two thermostats fighting over who’s in charge.
That and the analogy are very good points, thank you.
I discussed Gwern’s article in another comment. My point (which also applies to Gwern’s essay on GPT3 and scaling hypothesis) is the following:
I don’t dispute that you can build agent AIs, and that they can be useful.
I don’t claim that it is possible to get the same economic benefits by restricting to tool AIs. Indeed, in my previous post with Edelman, we explicitly said that we do consider AIs that are agentic in the sense that they can take action, including self-driving, writing code, executing trades etc..
I don’t dispute that one way to build those is to take a next-token predictor such as pretrained GPT3, and then use fine-tuning, RHLF, prompt engineering or other methods to turn it into an agent AI. (Indeed, I explicitly say so in the current post.)
My claim is that it is a useful abstraction to (1) separate intelligence from agency, and (2) intelligence in AI is a monotone function of the computational resources (FLOPs, data, model size, etc.) invested into building the model.
Now if you want to take 3.6 Trillion gradient steps in a model, then you simply cannot do it by having it take actions and wait to get some reward. So I do claim that if we buy the scaling hypothesis that intelligence scales with compute, the bulk of the intelligence of models such as GPT-n, PALM-n, etc. comes from the non agentic next-token predictor.
So, I believe it is useful and more accurate to think of (for example) a stock trading agent that is built on top of GPT-4 as consisting of an “intelligence forklift” which accounts for 99.9% of the computational resources, plus various layers of adaptations, including supervised fine-tuning, RL from human feedback, and prompt engineering, to obtain the agent.
The above perspective does not mean that the problem of AI safety or alignment is solved. But I do think it is useful to think of intelligence as belonging to a system rather than an individual agent, and (as discussed briefly above) that considering it in this way changes somewhat the landscape of both problems and solutions.
Ah. Well, if that’s what you are saying then you are preaching to the choir. :) See e.g. the “pretrained LLMs are simulators / predictors / oracles” discourse on LW.
I feel like there is probably still some disagreement between us though. For example I think the “bulk of the intelligence comes from non-agentic next-token predictor” claim you make probably is either less interesting or less true than you think it is, depending on what kinds of conclusions you think follow from it. If you are interested in discussing more sometime I’d be happy to have a video call!
Agree that we still disagree and (in my biased opinion) that claim is either more interesting or more true than you realize :) Not free for a call soon but hope eventually there is an opportunity to discuss more.
https://gwern.net/tool-ai
I was asked about this on Twitter. Gwern’s essay deserves a fuller response than a comment but I’m not arguing for the position Gwern argues against.
I don’t argue that agent AI are not useful or won’t be built. I am not arguing that humans must always be in the loop.
My argument is that tool vs agent AI is not so much about competition but specialization. Agent AIs have their uses but if we consider the “deep learning equation” of turning FLOPs into intelligence, then it’s hard to beat training for predictions on static data. So I do think that while RL can be used forAI agents, the intelligence “heavy lifting” (pun intended) would be done by non-agentic tool but very large static models.
Even “hybrid models” like GPT3.5 can best be understood as consisting of an “intelligence forklift”—the pretrained next-token predictor on which 99.9% of the FLOPs were spent on building—and an additional light “adapter” that turns this forklift into a useful Chatbot etc