While we humans can have conscious experiences, or qualia, that include being conscious of ourselves and our own existence, it’s not clear that conscious experience of the cognitive process of self-monitoring—a kind of sentience—is necessary to do the self-monitoring task.
According to Joscha Bach’s illusionism view (with which I agree and which I describe below), it’s the other way around: meta-awareness (i.e., awareness of awareness) within the model of oneself which acts as a ‘first-person character’ in the movie/dream/”controlled hallucination” that the human brain constantly generates for oneself is the key thing that also compels the brain to attach qualia (experiences) to the model. In other words, the “character within the movie” thinks that it feels something because it has meta-awareness (i.e., the character is aware that it is aware (which reflects the actual meta-cognition in the brain, rather than in the brain, insofar the character is a faithful model of reality). This view also seems consistent with “Bayesian qualia” (Clark et al. 2019).
You didn’t define neither intelligence nor agency, and people notoriously understand “intelligence” very differently (almost as much as they understand “consciousness” differently). This makes the post harder to perceive (and I suspect many readers will misunderstand it).
I think that agency is an attribute of some groups of nodes (a.k.a., Markov blankets) of causal graphs of the world that modellers/observers/scientists (subjectively) build. Specifically, the causal graph should be appropriately sparse to separate the “agent” from the “environment” (both are just collections of variables/factors/nodes in the causal graph) and to establish the perception-action loop, and the “agent” should be at least minimally “deep”. The key resources to understand this perspective are Ramstead et al. 2022 and Friston et al. 2022. This view seems to me to be roughly consistent with Everitt et al.’s causal framework for understanding agency. Note that this view doesn’t imply cognitive sophistication at all: even a cell (if not more simple and smaller objects) have “agency”, in this view.
So, “basal” agency (as defined above) is ubiquitous. Then, there is a subtle but important (and confusing) terminological shift: when we start discussing “degrees of agency”, as well as “orthogonality between agency and sentience” (which implies some “scale of agency”, because any discussion of “orthogonality” is the discussion of existence of some objects that are in extreme position on one scale (agency, in this case) and some arbitrary (or also extreme) position in another scale, in this case sentience), we replaced the “basal” agency with some sort of “evolutionary” agency, i.e., propensity for survival and reproduction. The “evolutionary” agency is an empirical property that couldn’t be measured well, especially ex-ante (ex-post, we can just take the overall lifespan of the agent as its the measure of its “evolutionary” agency). “Basal” agent could form for one millisecond and then dissolve. An agent that persisted for long may have done so not thanks to its intrinsic properties and competencies, but due to pure luck: maybe there was just nobody around to kill and eat that agent.
Nevertheless, evolutionary speaking, agents that exhibit internal coherence [in their models of the outside world, which are entailed by the dynamics of their internal states, per FEP] are more likely to survive for longer and to reproduce, and therefore are selected [biologically as well as artificially: i.e., LLMs are also selected for coherence in various ways]. Coherence here roughly corresponds to integration per Integrated Information Theory of consciousness (IIT). See Ramstead et al. 2023 for in-depth discussion of this.
Another “evolutionary useful” property of agents is representation of harm (or, decreased fitness) and behaviour programs for harm avoidance, such as reflexes. Representation of decreased fitness is valence per Hesp et al. 2021. Valence is actually quite easy to represent, so this representation should evolve in almost all biological species that are under near-constant threat of harm and death. On the other hand, it doesn’t seem to me that artificial minds such as LLMs should acquire the representation of valence by default (unless maybe trained or fine-tuned in special game environments) because it doesn’t seem very useful for them to represent valence for text modelling (aka “predicting next tokens”) as well as most tasks LLMs are fine-tuned for.
Yet another agent’s property that is sometimes evolutionary useful is the ability to construct models for solving some (wide range) of problems “on the spot”. If this ability spans the problems that are yet unseen by the agent, neither over the course of agent’s lifetime, nor over the course of its evolution, then this is called intelligence. This is roughly the definition per Chollet 2019.
All the above definitions and conceptions are squarely functional, so, according to them, both strict intelligence-sentience orthogonality and possible intelligence-sentience orthogonality are false, as implying something non-functional about consciousness.
Re: general intelligence orthogonality: it’s maybe the question of what will people recognise as AGI. Connor Leahy said that 10 years ago, most people would say that GPT-4 is AGI. I think that prospectively, we think of AGI in more objective terms (as per Chollet), and thus GPT-4 is AGI because objectively, it has impressive generalisation ability (even if only in the space of text). However, when judging whether there is an AGI in front of them or not, people tend to succumb to anthropocentrism: AGI should be something very close to them, in terms of their agentic capacities. Returning to the orthogonality question: it definitely should be possible to construct a Cholletian general intelligence that isn’t conscious. Maybe even effective training of future LLM designs on various science and math textbooks, but strictly barring LLMs from learning about AI (as a subject) in general, and thus about oneself, will be it. However, even though such an AGI could in principle start their own civilisation in a simulation, it wouldn’t be recognised as AGI in the cultural context of our civilisation. In order to solve arbitrary tasks in the current economy and in the internet, an LLM (or an LLM-based agent) should know about AI and LLMs, including about oneself, avoid harms, etc., which will effectively lead at least to proto-sentience.
Sentience-agency orthogonality doesn’t make a lot of sense to discuss out of the context of a specific environment. “An AI controlling a Starcraft player has plenty of agency within the computer game’s environment, but no one thinks that implies the AI is sentient”—per the definitions above, it’s only because being meta-aware is not adaptively useful to control a character in a given computer game. It was apparently useful in the evolution of humans, though, due to the features of our evolutionary environment. And this meta-awareness is what “unlocks” sentience.
Thank you for the link to the Friston paper. I’m reading that and will watch Lex Fridman’s interview with Joscha Bach, too. I sort of think “illusionism” is a bit too strong, but perhaps it’s a misnomer rather than wrong (or I could be wrong altogether). Clark, Friston, and Wilkinson say
But in what follows we aim not to Quine (explain away) qualia but to ‘Bayes’ them – to reveal them as products of a broadly speaking rational process of inference, of the kind imagined by the Reverend Bayes in his (1763) treatise on how to form and update beliefs on the basis of new evidence. Our story thus aims to occupy the somewhat elusive ‘revisionary’ space, in between full strength ‘illusionism’ (see below) and out-and-out realism
and I think somewhere in the middle sounds more plausible to me.
Anyhow, I’ll read the paper first before I try to respond more substantively to your remarks, but I intend to!
But in what follows we aim not to Quine (explain away) qualia …
“Quining means some thing stronger than “explaining away”. Dennett:
The verb “to quine” is even more esoteric. It comes from The Philosophical Lexicon (Dennett 1978c, 8th edn., 1987), a satirical dictionary of eponyms: “quine, v. To deny resolutely the existence or importance of something real or significant.”
I’ve never heard Bach himself calling his position “illusionism”, I’ve just applied this label to his view spontaneously. This could be inaccurate, indeed, especially from the perspective of historical usage of the term.
Instead of Bach’s interviews with Lex, I’d recommend this recent show: https://youtu.be/PkkN4bJN2pg (disclaimer: you will need to persevere through a lot of bad and often lengthy and ranty takes on AI by the hosts throughout the show, which at least to me was quite annoying. Unfortunately, there is no transcript, it seems.)
I found the Clark et al. (2019) “Bayesing Qualia” article very useful, and that did give me an intuition of the account that perhaps sentience arises out of self-awareness. But they themselves acknowledged in their conclusion that the paper didn’t quite demonstrate that principle, and I didn’t find myself convinced of it.
Perhaps what I’d like readers to take away is that sentience and self-awareness can be at the very least conceptually distinguished. Even if it isn’t clear empirically whether or not they are intrinsically linked, we ought to maintain a conceptual distinction in order to form testable hypotheses about whether they are in fact linked, and in order to reason about the nature of any link. Perhaps I should call that “Theoretical orthogonality”. This is important to be able to reason whether, for instance, giving our AIs a self-awareness or situational awareness will cause them to be sentient. I do not think that will be the case, although I do think that, if you gave them the sort of detailed self-monitoring feelings that humans have, that may yield sentience itself. But it’s not clear!
I listened to the whole episode with Bach as a result of your recommendation! Bach hardly even got a chance to express his ideas, and I’m not much closer to understanding his account of
meta-awareness (i.e., awareness of awareness) within the model of oneself which acts as a ‘first-person character’ in the movie/dream/”controlled hallucination” that the human brain constantly generates for oneself is the key thing that also compels the brain to attach qualia (experiences) to the model. In other words, the “character within the movie” thinks that it feels something because it has meta-awareness (i.e., the character is aware that it is aware (which reflects the actual meta-cognition in the brain, rather than in the brain, insofar the character is a faithful model of reality).
which seems like a crux here.
He sort of briefly described “consciousness as a dream state” at the very end, but although I did get the sense that maybe he thinks meta-awareness and sentience are connected, I didn’t really hear a great argument for that point of view.
He spent several minutes arguing that agency, or seeking a utility function, is something humans have, but that these things aren’t sufficient for consciousness (I don’t remember whether he said whether they were necessary, so I suppose we don’t know if he thinks they’re orthogonal).
According to Joscha Bach’s illusionism view (with which I agree and which I describe below), it’s the other way around: meta-awareness (i.e., awareness of awareness) within the model of oneself which acts as a ‘first-person character’ in the movie/dream/”controlled hallucination” that the human brain constantly generates for oneself is the key thing that also compels the brain to attach qualia (experiences) to the model. In other words, the “character within the movie” thinks that it feels something because it has meta-awareness (i.e., the character is aware that it is aware (which reflects the actual meta-cognition in the brain, rather than in the brain, insofar the character is a faithful model of reality). This view also seems consistent with “Bayesian qualia” (Clark et al. 2019).
You didn’t define neither intelligence nor agency, and people notoriously understand “intelligence” very differently (almost as much as they understand “consciousness” differently). This makes the post harder to perceive (and I suspect many readers will misunderstand it).
I think that agency is an attribute of some groups of nodes (a.k.a., Markov blankets) of causal graphs of the world that modellers/observers/scientists (subjectively) build. Specifically, the causal graph should be appropriately sparse to separate the “agent” from the “environment” (both are just collections of variables/factors/nodes in the causal graph) and to establish the perception-action loop, and the “agent” should be at least minimally “deep”. The key resources to understand this perspective are Ramstead et al. 2022 and Friston et al. 2022. This view seems to me to be roughly consistent with Everitt et al.’s causal framework for understanding agency. Note that this view doesn’t imply cognitive sophistication at all: even a cell (if not more simple and smaller objects) have “agency”, in this view.
So, “basal” agency (as defined above) is ubiquitous. Then, there is a subtle but important (and confusing) terminological shift: when we start discussing “degrees of agency”, as well as “orthogonality between agency and sentience” (which implies some “scale of agency”, because any discussion of “orthogonality” is the discussion of existence of some objects that are in extreme position on one scale (agency, in this case) and some arbitrary (or also extreme) position in another scale, in this case sentience), we replaced the “basal” agency with some sort of “evolutionary” agency, i.e., propensity for survival and reproduction. The “evolutionary” agency is an empirical property that couldn’t be measured well, especially ex-ante (ex-post, we can just take the overall lifespan of the agent as its the measure of its “evolutionary” agency). “Basal” agent could form for one millisecond and then dissolve. An agent that persisted for long may have done so not thanks to its intrinsic properties and competencies, but due to pure luck: maybe there was just nobody around to kill and eat that agent.
Nevertheless, evolutionary speaking, agents that exhibit internal coherence [in their models of the outside world, which are entailed by the dynamics of their internal states, per FEP] are more likely to survive for longer and to reproduce, and therefore are selected [biologically as well as artificially: i.e., LLMs are also selected for coherence in various ways]. Coherence here roughly corresponds to integration per Integrated Information Theory of consciousness (IIT). See Ramstead et al. 2023 for in-depth discussion of this.
Another “evolutionary useful” property of agents is representation of harm (or, decreased fitness) and behaviour programs for harm avoidance, such as reflexes. Representation of decreased fitness is valence per Hesp et al. 2021. Valence is actually quite easy to represent, so this representation should evolve in almost all biological species that are under near-constant threat of harm and death. On the other hand, it doesn’t seem to me that artificial minds such as LLMs should acquire the representation of valence by default (unless maybe trained or fine-tuned in special game environments) because it doesn’t seem very useful for them to represent valence for text modelling (aka “predicting next tokens”) as well as most tasks LLMs are fine-tuned for.
Yet another agent’s property that is sometimes evolutionary useful is the ability to construct models for solving some (wide range) of problems “on the spot”. If this ability spans the problems that are yet unseen by the agent, neither over the course of agent’s lifetime, nor over the course of its evolution, then this is called intelligence. This is roughly the definition per Chollet 2019.
All the above definitions and conceptions are squarely functional, so, according to them, both strict intelligence-sentience orthogonality and possible intelligence-sentience orthogonality are false, as implying something non-functional about consciousness.
Re: general intelligence orthogonality: it’s maybe the question of what will people recognise as AGI. Connor Leahy said that 10 years ago, most people would say that GPT-4 is AGI. I think that prospectively, we think of AGI in more objective terms (as per Chollet), and thus GPT-4 is AGI because objectively, it has impressive generalisation ability (even if only in the space of text). However, when judging whether there is an AGI in front of them or not, people tend to succumb to anthropocentrism: AGI should be something very close to them, in terms of their agentic capacities. Returning to the orthogonality question: it definitely should be possible to construct a Cholletian general intelligence that isn’t conscious. Maybe even effective training of future LLM designs on various science and math textbooks, but strictly barring LLMs from learning about AI (as a subject) in general, and thus about oneself, will be it. However, even though such an AGI could in principle start their own civilisation in a simulation, it wouldn’t be recognised as AGI in the cultural context of our civilisation. In order to solve arbitrary tasks in the current economy and in the internet, an LLM (or an LLM-based agent) should know about AI and LLMs, including about oneself, avoid harms, etc., which will effectively lead at least to proto-sentience.
Sentience-agency orthogonality doesn’t make a lot of sense to discuss out of the context of a specific environment. “An AI controlling a Starcraft player has plenty of agency within the computer game’s environment, but no one thinks that implies the AI is sentient”—per the definitions above, it’s only because being meta-aware is not adaptively useful to control a character in a given computer game. It was apparently useful in the evolution of humans, though, due to the features of our evolutionary environment. And this meta-awareness is what “unlocks” sentience.
Thank you for the link to the Friston paper. I’m reading that and will watch Lex Fridman’s interview with Joscha Bach, too. I sort of think “illusionism” is a bit too strong, but perhaps it’s a misnomer rather than wrong (or I could be wrong altogether). Clark, Friston, and Wilkinson say
and I think somewhere in the middle sounds more plausible to me.
Anyhow, I’ll read the paper first before I try to respond more substantively to your remarks, but I intend to!
“Quining means some thing stronger than “explaining away”. Dennett:
I’ve never heard Bach himself calling his position “illusionism”, I’ve just applied this label to his view spontaneously. This could be inaccurate, indeed, especially from the perspective of historical usage of the term.
Instead of Bach’s interviews with Lex, I’d recommend this recent show: https://youtu.be/PkkN4bJN2pg (disclaimer: you will need to persevere through a lot of bad and often lengthy and ranty takes on AI by the hosts throughout the show, which at least to me was quite annoying. Unfortunately, there is no transcript, it seems.)
I found the Clark et al. (2019) “Bayesing Qualia” article very useful, and that did give me an intuition of the account that perhaps sentience arises out of self-awareness. But they themselves acknowledged in their conclusion that the paper didn’t quite demonstrate that principle, and I didn’t find myself convinced of it.
Perhaps what I’d like readers to take away is that sentience and self-awareness can be at the very least conceptually distinguished. Even if it isn’t clear empirically whether or not they are intrinsically linked, we ought to maintain a conceptual distinction in order to form testable hypotheses about whether they are in fact linked, and in order to reason about the nature of any link. Perhaps I should call that “Theoretical orthogonality”. This is important to be able to reason whether, for instance, giving our AIs a self-awareness or situational awareness will cause them to be sentient. I do not think that will be the case, although I do think that, if you gave them the sort of detailed self-monitoring feelings that humans have, that may yield sentience itself. But it’s not clear!
I listened to the whole episode with Bach as a result of your recommendation! Bach hardly even got a chance to express his ideas, and I’m not much closer to understanding his account of
which seems like a crux here.
He sort of briefly described “consciousness as a dream state” at the very end, but although I did get the sense that maybe he thinks meta-awareness and sentience are connected, I didn’t really hear a great argument for that point of view.
He spent several minutes arguing that agency, or seeking a utility function, is something humans have, but that these things aren’t sufficient for consciousness (I don’t remember whether he said whether they were necessary, so I suppose we don’t know if he thinks they’re orthogonal).