Do you have any thoughts on chess computers, guided missiles, computer viruses, etc, and whether they make a case for worries about AGI, even if you consider them something alien to the human kind of intelligence?
No—but perhaps I’m not seeing how they would make the case. Is the idea that somehow their existence augurs a future in which tech gets more autonomous to a point where we can no longer control it? I guess I’d say, why should we believe that’s true? Its probably uncontroversial to believe many of our tools will get more autonomous—but why should we think that’ll lead to the kind of autonomy we enjoy?
Even if you believe that the intelligence and autonomy we enjoy exist on a kind of continuum, from like single celled organisms through chess-playing computers, to us—we’d still need reason to believe that the progress along this continuum will continue at a rate necessary to close the gap between where we sit on the continuum and where our best artifacts currently sit on the continuum. I don’t doubt that progress will continue; but even if the continuum view were right, I think we sit way further out on the continuum than most people with the continuum view think. Also, the continuum view itself is very, very controversial. I happen to accept the arguments which aim to show that it faces insurmountable obstacles. The alternate view which I accept is that there’s a difference in kind between the intelligence and autonomy we enjoy, and the kind enjoyed by non-human animals and chess-playing computers. Many people think that if we accept that, we have to reject a certain form of metaphysical naturalism (e.g. the view that all natural phenomena can be explained in terms of the basic conceptual tools of physics, maths, and logic).
Some people think that this form of metaphysical naturalism is bedrock stuff; that if we don’t accept it, the theists win, blah blah blah, so we must naturalize mentality and agency, it must exist on a continuum, we just need a theory which shows us how. Other people think we can have a non-reductive naturalism which takes as primitive the normative concepts found in biology and psychology. That’s the view I hold. So no, I don’t think the existence of those things makes a case for worries about AGI. Things which enjoy the kind of mentality and autonomy we enjoy must be like us in many, many ways—that is after all, what enables us to recognize them as having mentality and autonomy like ours. They probably need to have bodies, be mortal, have finite resources, have an ontogenesis period where they go from not like-minded to like-minded (as all children do), have some language, etc.
Also, I think we have to think really carefully about what we mean when we say “human kind of intelligence”—if you read Jim Conant’s logically alien thought paper you come to understand why that extra bit of qualification amounts to plainly nonsensical language. There’s only intelligence simpliciter; insofar as we’re justified in recognizing it as such, it’s precisely in virtue of its bearing some resemblance to ours. The very idea of other kinds of intelligence which we might not be able to recognize is conceptually confused (if it bears no resemblance to ours, in virtue of what are we supposed to call it intelligent? Ex hypothesi? If so, I don’t know what I’m supposed to be imagining).
The person who wrote this post rightfully calls attention to the conceptual confusions surrounding most casual pre-formal thinking about agency and mentality. I applaud that, and am urging that the most rigorous, well-trodden paths exploring these confusions are to be found in philosophy as practiced (mostly but not exclusively) in the Anglophone tradition over the last 50 years.
That this should be ignored or overlooked out of pretension by very smart people who came up in cogsci, stats, or compsci is intelligible to me; that it should be ignored on a blog that is purportedly about investigating all the available evidence to find quicker pathways to understanding is less intelligible. I would commend everyone with an interest in this stuff to read Stanford Encyclopedia of Philosophy entries on different topics in philosophy of action and philosophy of mind, then go off their bibliographies for more detailed treatments. This stuff is all explored by philosophers really sympathetic to—even involved in—the projects of creating AGI. But more importantly, it is equally explored by those who either think the project is theoretically and practically possible but prudentially mistaken, or by those who think it is theoretically and practically impossible; let alone a prudential possibility.
Most mistakes here are made in the pre-formal thinking. Philosophy is the discipline of making that thinking more rigorous.
I don’t know… If I try to think of Anglophone philosophers of mind who I respect, I think of “Australian materialists” like Armstrong and Chalmers. No doubt there are plenty of worthwhile thoughts among the British, Americans, etc too, but you seem to be promoting something I deplore, the attempt to rule out various hard problems and unwelcome possibilities, by insisting that words shouldn’t be used that way. Celia Green even suggested that this 1984-like tactic could be the philosophy of a new dark age in which inquiry was stifled, not by belief in religion, but by “belief in society”; but perhaps technology has averted that future. Head-in-the-sand anthropocentrism is hardly tenable in a world where, already, someone could hook up a GPT3 chatbot to a Boston Dynamics chassis, and create an entity from deep within the uncanny valley.
Totally get it. There are lots of folks practicing philosophy of mind and technology today in that aussie tradition who I think take these questions seriously and try to cache out what we mean when we talk about agency, mentality, etc. as part of their broader projects.
I’d resist your characterization that I’m insisting words shouldn’t be used a particular way, though I can understand why it might seem that way. I’m rather hoping to shed more light on the idea raised by this post that we don’t actually know what many of these words even mean when they’re used in certain ways (hence the authors totally correct point about the need to clarify confusions about agency while working on the alignment problem). My whole point in wading in here is just to point out to a thoughtful community that there’s a really long rich history of doing just this, and even if you prefer the answers given by aussie materialists, it’s even better to understand those positions vis-a-vis their present and past interlocutors. If you understand those who disagree with them, and can articulate those positions in terms they’d accept, you understand your preferred positions even better. I wouldn’t say I deplore it, but I am always mildly amused when cogsci, compsci, and stats people start wading into plainly philosophical waters (“sort out our fundamental confusions about agency”) and talk as if they’re the first ones to get there—or the only ones presently splashing around. I guess I would have thought (perhaps naively) that on a site like this people would be at least curious to see what work has already been done on the questions so they can accelerate their inquiry.
Re: ruling out hard problems—lot’s of philosophy is the attempt to better understand the problem’s framing such that it either reduces to a different problem, or disappears altogether. I’d urge you to see this as an example of that kind of thing, rather than ruling out certain questions from the gun.
And on anthropocentrism—not sure what the point is supposed to be here, but perhaps it’s directed at the “difference in kind” statements I made above. If so, I’d hope we can see light between treating humans as if they were the center of the universe and recognizing that there are at least putatively qualitative differences between the type of agency rational animals enjoy and the type of agency enjoyed by non-rational animals and artifacts. Even the aussie materialists do that—and then set about trying to form a theory of mind and agency in physical terms because they rightly see those putatively qualitative differences as a challenge to their particular form of metaphysical naturalism.
So look, if the author of this post is really serious about (1) they will almost certainly have to talk about what we mean when we use agential words. There will almost certainly be disagreements about whether their characterizations (A) fit the facts, and (B) are coherent with the rest of our beliefs. I don’t want to come even close to implying that folks in compsci, cogsci, stats, etc. can’t do this—they certainly can. I’m just saying that it’s really, really conspicuous to not do so in dialogue with those whose entire discipline is devoted to that task. Philosophers are really good at testing our accounts of an agential concept by saying things like, “okay let’s run with this idea of yours that we can define agency and mentality in terms of some bayesian predictive processing, or in terms of planning states, or whaterver, but to see if that view really holds up, we have to be able to use your terms or some innocent others to account for all the distinctions we recognize in our thought and talk about minds and agents.” That’s the bulk of what philosophers of mind and action do nowadays—they take someone’s proposal about a theory of mind or action and test whether it can give an account of some region of our thought and talk about minds and agents. If it can’t they either propose addenda, push the burden back to the theorist, or point out structural reasons why the theory faces general obstacles that seem difficult to overcome.
Here’s some recent work on the topic, just to make it plain that there are philosophers working on these questions:
? The second is already open-access, and the third both works in SH & GS (with 2 different PDF links). Only the first link fails in SH. (But what an abstract: “I also argue that if future generally intelligent AI possess a predictive processing cognitive architecture, then they will come to share our pro-moral motivations (of valuing humanity as an end; avoiding maleficent actions; etc.), regardless of their initial motivation set.” Wow.)
Huh, I tried the first and third in SH, maybe I messed up somehow. My bad. Thanks!
I still am interested in the first (on the principle that maybe, just maybe, it’s the solution to all our problems instead of being yet another terrible argument made by philosophers about why AIs will be ethical by default if only we do X… I think I’ve seen two already) and would like to have access.
I can see how that would work. The author needs to be careful, though. Predictive processing may be a necessary condition for robust AGI alignment, but it is not per se a sufficient condition.
First of all, that only works if you give the AGI strong inductive priors for detecting and predicting human needs, goals, and values. Otherwise, it will tend to predict humans as though we are just “physical” systems (we are, but I mean modeling us without taking our sentience and values into account), no more worthy of special care than rocks or streams.
Second of all, this only works if the AGI has a structural bias toward treating the needs, goals, and values that it infers from predictive processing as its own. Otherwise, it may understand how to align with us, but it won’t care by default.
Do you have any thoughts on chess computers, guided missiles, computer viruses, etc, and whether they make a case for worries about AGI, even if you consider them something alien to the human kind of intelligence?
No—but perhaps I’m not seeing how they would make the case. Is the idea that somehow their existence augurs a future in which tech gets more autonomous to a point where we can no longer control it? I guess I’d say, why should we believe that’s true? Its probably uncontroversial to believe many of our tools will get more autonomous—but why should we think that’ll lead to the kind of autonomy we enjoy?
Even if you believe that the intelligence and autonomy we enjoy exist on a kind of continuum, from like single celled organisms through chess-playing computers, to us—we’d still need reason to believe that the progress along this continuum will continue at a rate necessary to close the gap between where we sit on the continuum and where our best artifacts currently sit on the continuum. I don’t doubt that progress will continue; but even if the continuum view were right, I think we sit way further out on the continuum than most people with the continuum view think. Also, the continuum view itself is very, very controversial. I happen to accept the arguments which aim to show that it faces insurmountable obstacles. The alternate view which I accept is that there’s a difference in kind between the intelligence and autonomy we enjoy, and the kind enjoyed by non-human animals and chess-playing computers. Many people think that if we accept that, we have to reject a certain form of metaphysical naturalism (e.g. the view that all natural phenomena can be explained in terms of the basic conceptual tools of physics, maths, and logic).
Some people think that this form of metaphysical naturalism is bedrock stuff; that if we don’t accept it, the theists win, blah blah blah, so we must naturalize mentality and agency, it must exist on a continuum, we just need a theory which shows us how. Other people think we can have a non-reductive naturalism which takes as primitive the normative concepts found in biology and psychology. That’s the view I hold. So no, I don’t think the existence of those things makes a case for worries about AGI. Things which enjoy the kind of mentality and autonomy we enjoy must be like us in many, many ways—that is after all, what enables us to recognize them as having mentality and autonomy like ours. They probably need to have bodies, be mortal, have finite resources, have an ontogenesis period where they go from not like-minded to like-minded (as all children do), have some language, etc.
Also, I think we have to think really carefully about what we mean when we say “human kind of intelligence”—if you read Jim Conant’s logically alien thought paper you come to understand why that extra bit of qualification amounts to plainly nonsensical language. There’s only intelligence simpliciter; insofar as we’re justified in recognizing it as such, it’s precisely in virtue of its bearing some resemblance to ours. The very idea of other kinds of intelligence which we might not be able to recognize is conceptually confused (if it bears no resemblance to ours, in virtue of what are we supposed to call it intelligent? Ex hypothesi? If so, I don’t know what I’m supposed to be imagining).
The person who wrote this post rightfully calls attention to the conceptual confusions surrounding most casual pre-formal thinking about agency and mentality. I applaud that, and am urging that the most rigorous, well-trodden paths exploring these confusions are to be found in philosophy as practiced (mostly but not exclusively) in the Anglophone tradition over the last 50 years.
That this should be ignored or overlooked out of pretension by very smart people who came up in cogsci, stats, or compsci is intelligible to me; that it should be ignored on a blog that is purportedly about investigating all the available evidence to find quicker pathways to understanding is less intelligible. I would commend everyone with an interest in this stuff to read Stanford Encyclopedia of Philosophy entries on different topics in philosophy of action and philosophy of mind, then go off their bibliographies for more detailed treatments. This stuff is all explored by philosophers really sympathetic to—even involved in—the projects of creating AGI. But more importantly, it is equally explored by those who either think the project is theoretically and practically possible but prudentially mistaken, or by those who think it is theoretically and practically impossible; let alone a prudential possibility.
Most mistakes here are made in the pre-formal thinking. Philosophy is the discipline of making that thinking more rigorous.
I don’t know… If I try to think of Anglophone philosophers of mind who I respect, I think of “Australian materialists” like Armstrong and Chalmers. No doubt there are plenty of worthwhile thoughts among the British, Americans, etc too, but you seem to be promoting something I deplore, the attempt to rule out various hard problems and unwelcome possibilities, by insisting that words shouldn’t be used that way. Celia Green even suggested that this 1984-like tactic could be the philosophy of a new dark age in which inquiry was stifled, not by belief in religion, but by “belief in society”; but perhaps technology has averted that future. Head-in-the-sand anthropocentrism is hardly tenable in a world where, already, someone could hook up a GPT3 chatbot to a Boston Dynamics chassis, and create an entity from deep within the uncanny valley.
Totally get it. There are lots of folks practicing philosophy of mind and technology today in that aussie tradition who I think take these questions seriously and try to cache out what we mean when we talk about agency, mentality, etc. as part of their broader projects.
I’d resist your characterization that I’m insisting words shouldn’t be used a particular way, though I can understand why it might seem that way. I’m rather hoping to shed more light on the idea raised by this post that we don’t actually know what many of these words even mean when they’re used in certain ways (hence the authors totally correct point about the need to clarify confusions about agency while working on the alignment problem). My whole point in wading in here is just to point out to a thoughtful community that there’s a really long rich history of doing just this, and even if you prefer the answers given by aussie materialists, it’s even better to understand those positions vis-a-vis their present and past interlocutors. If you understand those who disagree with them, and can articulate those positions in terms they’d accept, you understand your preferred positions even better. I wouldn’t say I deplore it, but I am always mildly amused when cogsci, compsci, and stats people start wading into plainly philosophical waters (“sort out our fundamental confusions about agency”) and talk as if they’re the first ones to get there—or the only ones presently splashing around. I guess I would have thought (perhaps naively) that on a site like this people would be at least curious to see what work has already been done on the questions so they can accelerate their inquiry.
Re: ruling out hard problems—lot’s of philosophy is the attempt to better understand the problem’s framing such that it either reduces to a different problem, or disappears altogether. I’d urge you to see this as an example of that kind of thing, rather than ruling out certain questions from the gun.
And on anthropocentrism—not sure what the point is supposed to be here, but perhaps it’s directed at the “difference in kind” statements I made above. If so, I’d hope we can see light between treating humans as if they were the center of the universe and recognizing that there are at least putatively qualitative differences between the type of agency rational animals enjoy and the type of agency enjoyed by non-rational animals and artifacts. Even the aussie materialists do that—and then set about trying to form a theory of mind and agency in physical terms because they rightly see those putatively qualitative differences as a challenge to their particular form of metaphysical naturalism.
So look, if the author of this post is really serious about (1) they will almost certainly have to talk about what we mean when we use agential words. There will almost certainly be disagreements about whether their characterizations (A) fit the facts, and (B) are coherent with the rest of our beliefs. I don’t want to come even close to implying that folks in compsci, cogsci, stats, etc. can’t do this—they certainly can. I’m just saying that it’s really, really conspicuous to not do so in dialogue with those whose entire discipline is devoted to that task. Philosophers are really good at testing our accounts of an agential concept by saying things like, “okay let’s run with this idea of yours that we can define agency and mentality in terms of some bayesian predictive processing, or in terms of planning states, or whaterver, but to see if that view really holds up, we have to be able to use your terms or some innocent others to account for all the distinctions we recognize in our thought and talk about minds and agents.” That’s the bulk of what philosophers of mind and action do nowadays—they take someone’s proposal about a theory of mind or action and test whether it can give an account of some region of our thought and talk about minds and agents. If it can’t they either propose addenda, push the burden back to the theorist, or point out structural reasons why the theory faces general obstacles that seem difficult to overcome.
Here’s some recent work on the topic, just to make it plain that there are philosophers working on these questions:
https://link.springer.com/article/10.1007%2Fs10676-021-09611-0
https://link.springer.com/article/10.1007/s11023-020-09539-2
And a great article by a favorite philosopher of action on three competing theories of human agency
https://onlinelibrary.wiley.com/doi/10.1111/nous.12178
Hope some of that is interesting, and appreciate the response.
Cheers
Those articles are all paywalled; got free versions? I tried Sci-Hub, no luck.
? The second is already open-access, and the third both works in SH & GS (with 2 different PDF links). Only the first link fails in SH. (But what an abstract: “I also argue that if future generally intelligent AI possess a predictive processing cognitive architecture, then they will come to share our pro-moral motivations (of valuing humanity as an end; avoiding maleficent actions; etc.), regardless of their initial motivation set.” Wow.)
Huh, I tried the first and third in SH, maybe I messed up somehow. My bad. Thanks!
I still am interested in the first (on the principle that maybe, just maybe, it’s the solution to all our problems instead of being yet another terrible argument made by philosophers about why AIs will be ethical by default if only we do X… I think I’ve seen two already) and would like to have access.
I can see how that would work. The author needs to be careful, though. Predictive processing may be a necessary condition for robust AGI alignment, but it is not per se a sufficient condition.
First of all, that only works if you give the AGI strong inductive priors for detecting and predicting human needs, goals, and values. Otherwise, it will tend to predict humans as though we are just “physical” systems (we are, but I mean modeling us without taking our sentience and values into account), no more worthy of special care than rocks or streams.
Second of all, this only works if the AGI has a structural bias toward treating the needs, goals, and values that it infers from predictive processing as its own. Otherwise, it may understand how to align with us, but it won’t care by default.