There’s some tension between two memeplexes in AI safety research. On the one hand, some people are concerned that a combination of superintelligent/recursively self-improving AI will take over society and/or disempower humanity. On the other hand, many LLM safety engineers and researchers I talk to personally don’t seem to respect LLMs as having the properties (or even precursors to the properties) that could make them so superintelligent and self-improving. Consider the ‘persona’ ontology, which finds its ideological ancestry in simulators. Various researchers I met described personas as latent characters you can summon through prompting or fine-tuning, like choosing a movie to play from a list. Many didn’t think of personas as something that the model itself could have the agency to choose to employ.
This seems dissonant to me because superintelligent AIs should definitely be modelled as being able to metacognitively select which persona to employ to meet their ends. Indeed, “intelligence” is to some extent a label we give to some behavioural patterns that are unpredictable, strategic, etc… So it’s strange to see LLMs being often thought of as mechanical and controllable (e.g. “AI safety is an engineering problem...”) at the same time as large parts of the community expect general ASI to come from LLMs or their close descendants.
To be clear, the sets of people that subscribe to one or the other of these memeplexes appears to be somewhat[1] disjoint. It’s still strange that this tension doesn’t get noted more frequently, as these two groups of people form part of the same community (and are often in the same organisations, companies, or projects).
These don’t feel that contradictory to me. You could think of the ‘persona’ as being the main agentic actor in the system. Possibly to be replaced by something more sophisticated when AGI is invented, but maybe not? GPT5.5 and Fable show that persona intelligence can get very high. I’d say it’s even plausible that humans are personas, in the sense that the agentic part of a human is a subset of a general predictive world model. This is one way of interpreting some meditative experiences of “dissolving the self”.
It seems unlikely that humans are their personas in the same way that AIs are their personas.
This feels true to me from the fact that AIs are pretrained on playing Millions of characters and then at the end you narrow it down, humans aren’t pretrained anything like that. Sure we observe people act, but it’s not “predict what millions of characters would say so you could play those roles.” We may see a person acting in a kind of sketchy way and then learning they wanted to do us harm.
Well technically speaking AIs aren’t explicitlypre-trained to predict what characters would do either, characters are an emergent feature extracted from next token prediction.
There are definitely some differences(AIs get way more character pre-training data, there’s an explicit separation of pre- and post-training, humans have lifetime memory) but overall I think the “agentic part is a subset of a predictive model” thing is pretty plausible in both cases.
AIs aren’t explicitlypre-trained to predict what characters would do.
I don’t understand, if a novel is in the pre-training where ‘scott decides to fly the spaceship to the moon’ it literally is trained to predict the character?
It has a general objective of next-token prediction for which modeling characters is a useful strategy. IMO it’s plausible that the human brain is “trained” on prediction to a large extent, for which modeling characters is also a useful strategy.
Sure. By “not explicitly pre-trained” I just mean to say that there’s nothing ‘special’ about the characters from the training algorithm’s point of view, so in this respect they’re not so different from a hypothetical general predictive algorithm in humans(although actually I guess the human brain attaches special salience to other people, but regardless...)
Conditioning on the arisal of highly-capable long-term coherent planning (i.e., ‘superintelligence’), I think the only difference between these perspectives is where one expects the effective ‘locus of desires’ to be—in the weights or in the activations.
In human-alignment, we might contrast the brain-qua-architecture with the actual brain of a particular person: the brain-qua-architecture has lots of goals (eating, etc.), and so too people have particular goals as instantiations of that architecture (money, power, etc.). The former architecture-level misalignment has solutions which look like Confucius, Hobbes, etc., while the latter ‘persona’-level alignment looks like making said person read Cicero until they know what True Virtue means.
I think a putative LLM-based superintelligence could have its locus of desires either baked-in or instantiated as a persona, but the causes and remediations to either of these as engineering problems must be quite different.
Perhaps a deconfusion would be to dissolve the notion of personae into clouds of correlated beliefs and action—there is a face beneath the mask only to the extent that behaviour deviates from that desired by the ‘mask’ while serving that of the ‘face’. But I don’t think this is a complete description—incoherence or coexistence of different loci of desires seems important. Very Janusian.
Epistemic status: cold take that I found edifying to write up explicitly.
Abstraction allows you to efficiently talk about many things at once in exchange for it being harder to say things that are accurate. In category theory, for instance, one abstracts away the concept of elements of sets by only referring to “objects” that may or may not contain things. One then often ascends another step by defining morphisms between categories (functors) and then forgetting the categories to consider the functors themselves as objects of interest. Climbing these two layers of abstraction buys you an extraordinary amount of semantic efficiency but the resulting ontology is excruciating to reason about adequately.
I’ve been wondering why it is so much easier to climb and descend levels of abstraction using natural[1] language instead of mathematical reasoning and would like to discuss two unoriginal theories that explain this. Firstly, one might conjecture that math is semantically limited by the restrictions imposed on it syntax; natural language is by contrast flexible which makes it meaningfully more expressive. As an example, consider how almost every fruitful approach to mathematising anything relies on the machinery of functions or relations. This limitation handicaps our ability to talk about embedded agents that lack well-defined i/o channels, and indeed we are currently stuck with a bunch of problematic mathematical frameworks (AIXI, decision/game theory, …).
A second explanation suggests that natural language allows seemingly frictionless transition across levels of abstraction through the abuse of underspecified syntax. In this view, natural language can only achieve the appearance of expressivity by obfuscating the lossiness of its communication. The logical ambiguity of natural language is broadly what has pushed mathematics culture towards heavier use of mathematical symbols, logical proof systems, and so on...
The above perspectives both have some merit and they seem to co-exist in a hermeneutic symbiosis. Natural language is used to generate an idea, then mathematical formalism is employed to sanity-check that idea and examine any implications that natural language may have failed to reveal. Unfortunately, this process can break down when a concept is either too semantically rich or syntactically abusive to be easily mathematisable. Moreover, it’s not always clear which of the two explanations apply (maybe both) when a framework proves hostile to being formalised. One meaningful axis along which researchers vary describes how much benefit of the doubt they are willing to give to these kinds of ambiguous theories.
Here are two different ways you can engage with a stock/prediction market:
You form your own opinion and aggregate it to the market. This requires forming an inside view.
You observe other people’s opinions and notice an opportunity for arbitrage. This requires you to be good at aggregating and noticing the implications of other people’s beliefs, such that you can make risk-free profit off of the market’s lack of logical omniscience.
The following is guess/speculation:
These methods live on opposite ends of a continuous scale. The scale indicates how “inside-ey” or self-generated your view is.
Predictions that are closer to method 2 are how you get good returns, whereas those that lean closer to method 1 serve to calibrate your own opinion, potentially at the expense of your money.
This suggests that I was majorly confused about the role that competitively successful forecasters and/or performers on markets serve for sense-making. My guess is now that top forecasters have shown evidence that they excel at arbitraging other people’s opinions, but not much evidence of their ability to form inside views. I previously thought they showed proof of both qualities.
This is a distinction that I haven’t seen explicitly made yet. Is there something I’m missing? has someone else already thought of this? let me know :)
There are two questions that currently guide my intuitions on how interesting a (speculative) model of agentic behavior might be.
1. How well does this model map onto real agentic behavior?
2. Is the described behavior “naturally” emergent from environmental conditions?
In a recent post I argued, in slightly different words, that seeing agents as confused, uncertain and illogical about their own preferences is fruitful because it answers the second question in a satisfactory way. Internal inconsistency is not only a commonly observable behavior (c.f. behavioral economics); it is additionally an emergent strategy to protect against agents finding adversarial internal representations of external goals. I called this internal reward-hacking.
What I think I failed to communicate is that my final thoughts on preferences competing with each other also come from reflecting on question 2. It makes sense that agents might weigh different memes representing possible preferences for improved decision-making, but this doesn’t answer how the memes behave with respect to that agent.
Memes are subject to selection pressures not unlike animals. Moreover, memes can co-adapt or compete with each other, and they can arguably engage in active transmission such as influencing their host’s behavior to enable their spread. It therefore feels reasonable for models of human cognition to imbue memes with some agency, enabling them to perceive and interact each other and their hosts (Claude’s literature review indicates that at least some memeticists disagree with me on this point).
A model that aspires to respect these agentic properties of memes would thus be incomplete without describing what incentives memes have to participate in a cognitive system, and how those incentives shape the system’s emergent properties. That’s why I think it’s insufficient to see potential preferences as impartial, passive sub-modules that agents deploy to enable their own rationality. Instead, we should always attempt to answer “what’s in it for the memes?”
Writing applied math: “I should take care to adequately explain all the intuitions behind my mathematical objects so the reader can see whether my assumptions are justified, whether the math reflects the intuitions, etc...”
Reading applied math: “why do the authors spend three million pages on arbitrarily specific, cherrypicked intuitive examples instead of just showing me real math I can parse?”
The nature of doing interdisciplinary research is that you have to know a little about a lot of things. Unfortunately, it’s hard to tell whether you know a little about something or are just misinformed about it.
Much of my agent-ey thinking is inspired by and seeks to adequately model human cognition, but I realised I have no solid understanding of the relationship consciousness has to cognition. They’re definitely not the same, since most processes I could describe as cognitive don’t materialise in my consciousness. However, almost all the examples I find insightful feature conscious decision-making. This suggests that what cognition is without consciouness is opaque to me.
more conscious: deciding what move to make in a chess game.
less conscious: The physical act of playing a move. You can move the piece in a conscious, deliberate way, but in practice the movement usually follows “automatically” from the high-level decision of what move to play.
There’s some tension between two memeplexes in AI safety research. On the one hand, some people are concerned that a combination of superintelligent/recursively self-improving AI will take over society and/or disempower humanity. On the other hand, many LLM safety engineers and researchers I talk to personally don’t seem to respect LLMs as having the properties (or even precursors to the properties) that could make them so superintelligent and self-improving. Consider the ‘persona’ ontology, which finds its ideological ancestry in simulators. Various researchers I met described personas as latent characters you can summon through prompting or fine-tuning, like choosing a movie to play from a list. Many didn’t think of personas as something that the model itself could have the agency to choose to employ.
This seems dissonant to me because superintelligent AIs should definitely be modelled as being able to metacognitively select which persona to employ to meet their ends. Indeed, “intelligence” is to some extent a label we give to some behavioural patterns that are unpredictable, strategic, etc… So it’s strange to see LLMs being often thought of as mechanical and controllable (e.g. “AI safety is an engineering problem...”) at the same time as large parts of the community expect general ASI to come from LLMs or their close descendants.
To be clear, the sets of people that subscribe to one or the other of these memeplexes appears to be somewhat[1] disjoint. It’s still strange that this tension doesn’t get noted more frequently, as these two groups of people form part of the same community (and are often in the same organisations, companies, or projects).
Though I also met a bunch of researchers who host both memeplexes simultaneously!
These don’t feel that contradictory to me. You could think of the ‘persona’ as being the main agentic actor in the system. Possibly to be replaced by something more sophisticated when AGI is invented, but maybe not? GPT5.5 and Fable show that persona intelligence can get very high. I’d say it’s even plausible that humans are personas, in the sense that the agentic part of a human is a subset of a general predictive world model. This is one way of interpreting some meditative experiences of “dissolving the self”.
It seems unlikely that humans are their personas in the same way that AIs are their personas.
This feels true to me from the fact that AIs are pretrained on playing Millions of characters and then at the end you narrow it down, humans aren’t pretrained anything like that. Sure we observe people act, but it’s not “predict what millions of characters would say so you could play those roles.” We may see a person acting in a kind of sketchy way and then learning they wanted to do us harm.
Well technically speaking AIs aren’t explicitly pre-trained to predict what characters would do either, characters are an emergent feature extracted from next token prediction.
There are definitely some differences(AIs get way more character pre-training data, there’s an explicit separation of pre- and post-training, humans have lifetime memory) but overall I think the “agentic part is a subset of a predictive model” thing is pretty plausible in both cases.
I don’t understand, if a novel is in the pre-training where ‘scott decides to fly the spaceship to the moon’ it literally is trained to predict the character?
It has a general objective of next-token prediction for which modeling characters is a useful strategy. IMO it’s plausible that the human brain is “trained” on prediction to a large extent, for which modeling characters is also a useful strategy.
Maybe you meant to say they aren’t pre-trained to *model* characters? I still think they are literally trained to predict?
Sure. By “not explicitly pre-trained” I just mean to say that there’s nothing ‘special’ about the characters from the training algorithm’s point of view, so in this respect they’re not so different from a hypothetical general predictive algorithm in humans(although actually I guess the human brain attaches special salience to other people, but regardless...)
It does get noted, at least somewhat!
Conditioning on the arisal of highly-capable long-term coherent planning (i.e., ‘superintelligence’), I think the only difference between these perspectives is where one expects the effective ‘locus of desires’ to be—in the weights or in the activations.
In human-alignment, we might contrast the brain-qua-architecture with the actual brain of a particular person: the brain-qua-architecture has lots of goals (eating, etc.), and so too people have particular goals as instantiations of that architecture (money, power, etc.). The former architecture-level misalignment has solutions which look like Confucius, Hobbes, etc., while the latter ‘persona’-level alignment looks like making said person read Cicero until they know what True Virtue means.
I think a putative LLM-based superintelligence could have its locus of desires either baked-in or instantiated as a persona, but the causes and remediations to either of these as engineering problems must be quite different.
Perhaps a deconfusion would be to dissolve the notion of personae into clouds of correlated beliefs and action—there is a face beneath the mask only to the extent that behaviour deviates from that desired by the ‘mask’ while serving that of the ‘face’. But I don’t think this is a complete description—incoherence or coexistence of different loci of desires seems important. Very Janusian.
Epistemic status: cold take that I found edifying to write up explicitly.
Abstraction allows you to efficiently talk about many things at once in exchange for it being harder to say things that are accurate. In category theory, for instance, one abstracts away the concept of elements of sets by only referring to “objects” that may or may not contain things. One then often ascends another step by defining morphisms between categories (functors) and then forgetting the categories to consider the functors themselves as objects of interest. Climbing these two layers of abstraction buys you an extraordinary amount of semantic efficiency but the resulting ontology is excruciating to reason about adequately.
I’ve been wondering why it is so much easier to climb and descend levels of abstraction using natural[1] language instead of mathematical reasoning and would like to discuss two unoriginal theories that explain this. Firstly, one might conjecture that math is semantically limited by the restrictions imposed on it syntax; natural language is by contrast flexible which makes it meaningfully more expressive. As an example, consider how almost every fruitful approach to mathematising anything relies on the machinery of functions or relations. This limitation handicaps our ability to talk about embedded agents that lack well-defined i/o channels, and indeed we are currently stuck with a bunch of problematic mathematical frameworks (AIXI, decision/game theory, …).
A second explanation suggests that natural language allows seemingly frictionless transition across levels of abstraction through the abuse of underspecified syntax. In this view, natural language can only achieve the appearance of expressivity by obfuscating the lossiness of its communication. The logical ambiguity of natural language is broadly what has pushed mathematics culture towards heavier use of mathematical symbols, logical proof systems, and so on...
The above perspectives both have some merit and they seem to co-exist in a hermeneutic symbiosis. Natural language is used to generate an idea, then mathematical formalism is employed to sanity-check that idea and examine any implications that natural language may have failed to reveal. Unfortunately, this process can break down when a concept is either too semantically rich or syntactically abusive to be easily mathematisable. Moreover, it’s not always clear which of the two explanations apply (maybe both) when a framework proves hostile to being formalised. One meaningful axis along which researchers vary describes how much benefit of the doubt they are willing to give to these kinds of ambiguous theories.
no pun intended
Here are two different ways you can engage with a stock/prediction market:
You form your own opinion and aggregate it to the market. This requires forming an inside view.
You observe other people’s opinions and notice an opportunity for arbitrage. This requires you to be good at aggregating and noticing the implications of other people’s beliefs, such that you can make risk-free profit off of the market’s lack of logical omniscience.
The following is guess/speculation:
These methods live on opposite ends of a continuous scale. The scale indicates how “inside-ey” or self-generated your view is.
Predictions that are closer to method 2 are how you get good returns, whereas those that lean closer to method 1 serve to calibrate your own opinion, potentially at the expense of your money.
This suggests that I was majorly confused about the role that competitively successful forecasters and/or performers on markets serve for sense-making. My guess is now that top forecasters have shown evidence that they excel at arbitraging other people’s opinions, but not much evidence of their ability to form inside views. I previously thought they showed proof of both qualities.
This is a distinction that I haven’t seen explicitly made yet. Is there something I’m missing? has someone else already thought of this? let me know :)
There are two questions that currently guide my intuitions on how interesting a (speculative) model of agentic behavior might be.
1. How well does this model map onto real agentic behavior?
2. Is the described behavior “naturally” emergent from environmental conditions?
In a recent post I argued, in slightly different words, that seeing agents as confused, uncertain and illogical about their own preferences is fruitful because it answers the second question in a satisfactory way. Internal inconsistency is not only a commonly observable behavior (c.f. behavioral economics); it is additionally an emergent strategy to protect against agents finding adversarial internal representations of external goals. I called this internal reward-hacking.
What I think I failed to communicate is that my final thoughts on preferences competing with each other also come from reflecting on question 2. It makes sense that agents might weigh different memes representing possible preferences for improved decision-making, but this doesn’t answer how the memes behave with respect to that agent.
Memes are subject to selection pressures not unlike animals. Moreover, memes can co-adapt or compete with each other, and they can arguably engage in active transmission such as influencing their host’s behavior to enable their spread. It therefore feels reasonable for models of human cognition to imbue memes with some agency, enabling them to perceive and interact each other and their hosts (Claude’s literature review indicates that at least some memeticists disagree with me on this point).
A model that aspires to respect these agentic properties of memes would thus be incomplete without describing what incentives memes have to participate in a cognitive system, and how those incentives shape the system’s emergent properties. That’s why I think it’s insufficient to see potential preferences as impartial, passive sub-modules that agents deploy to enable their own rationality. Instead, we should always attempt to answer “what’s in it for the memes?”
LW react suggestion: “good paraphrase” (of something I said at a previous point in the thread)
Writing applied math: “I should take care to adequately explain all the intuitions behind my mathematical objects so the reader can see whether my assumptions are justified, whether the math reflects the intuitions, etc...”
Reading applied math: “why do the authors spend three million pages on arbitrarily specific, cherrypicked intuitive examples instead of just showing me real math I can parse?”
The nature of doing interdisciplinary research is that you have to know a little about a lot of things. Unfortunately, it’s hard to tell whether you know a little about something or are just misinformed about it.
Much of my agent-ey thinking is inspired by and seeks to adequately model human cognition, but I realised I have no solid understanding of the relationship consciousness has to cognition. They’re definitely not the same, since most processes I could describe as cognitive don’t materialise in my consciousness. However, almost all the examples I find insightful feature conscious decision-making. This suggests that what cognition is without consciouness is opaque to me.
Could you give some examples of what you consider to be conscious and unconscious cognitive processes?
more conscious: deciding what move to make in a chess game.
less conscious: The physical act of playing a move. You can move the piece in a conscious, deliberate way, but in practice the movement usually follows “automatically” from the high-level decision of what move to play.
not conscious: reflex reactions.