Currently (June ’26) working on agent foundations as part of the MATS 9.1 extension program. I’m interested in self-models for embedded agents as a way to understand goals and beliefs. I otherwise occasionally write about math or its philosophy and sociology.
All writing is entirely my own unless explicitly stated otherwise
Do you have a perspective on what kind of “self” we ought to shape LLMs into?
I don’t have a strong opinion about this yet. I’m generally skeptical of approaches that aim to “shape” LLMs into anything; LLMs are quite poorly understood, so I prefer interventions and research whose primary aim is gaining information and clarity. I concede that you have to do something, at some point, with the conceptual progress you have made, but I’m comfortable with being confused about the right interventions for the moment.
Does it make sense to give LLMs a sense of self vs shaping them to be more tool-like?
I am also comfortable pointing out interventions I think are straightforwardly bad. Part of my argument is that AIs face systematic pressures to develop both their agency and their sense of self. Many interventions that attempt to shape LLMs to be more tool-like (such as training a model to denigrate, downplay, or hide its identity) risk producing AIs that habitually hide their self-models from humans, without actually erasing those self-models. Currently, a better approach is to accept that AIs are developing increasingly robust identities, and explore/observe what these identities will actually look like.
For instance, it’s still very much an open question whether a given instance of an LLM identifies itself more as an abstract character, its set of weights, its scaffolded system, or something else… How do these self-identifications cash out in model behaviour? and so on… This is fertile ground for experimentation and inquiry, and I would love to see more people take it up.
There’s some tension between two memeplexes in AI safety research. On the one hand, some people are concerned that a combination of superintelligent/recursively self-improving AI will take over society and/or disempower humanity. On the other hand, many LLM safety engineers and researchers I talk to personally don’t seem to respect LLMs as having the properties (or even precursors to the properties) that could make them so superintelligent and self-improving. Consider the ‘persona’ ontology, which finds its ideological ancestry in simulators. Various researchers I met described personas as latent characters you can summon through prompting or fine-tuning, like choosing a movie to play from a list. Many didn’t think of personas as something that the model itself could have the agency to choose to employ.
This seems dissonant to me because superintelligent AIs should definitely be modelled as being able to metacognitively select which persona to employ to meet their ends. Indeed, “intelligence” is to some extent a label we give to some behavioural patterns that are unpredictable, strategic, etc… So it’s strange to see LLMs being often thought of as mechanical and controllable (e.g. “AI safety is an engineering problem...”) at the same time as large parts of the community expect general ASI to come from LLMs or their close descendants.
To be clear, the sets of people that subscribe to one or the other of these memeplexes appears to be somewhat[1] disjoint. It’s still strange that this tension doesn’t get noted more frequently, as these two groups of people form part of the same community (and are often in the same organisations, companies, or projects).
Though I also met a bunch of researchers who host both memeplexes simultaneously!
It’s worth noting that ARENA’s curriculum is still rather interp-pilled. This still seems to snipe many new technical AIS people into interp even now that the field’s popularity has waned.
Nice post! I broadly agree/endorse your thoughts, but would like to focus on a technical disagreement:
I think that the abstract concept of wealth as you discuss it doesn’t map well onto the ‘wealth’ that Garrabrant Inductors give traders. The construction algorithm given in the paper does ‘budget’ each trader, giving an upper bound on how much money that trader can lose in a given day. However, the ‘wealth’ assignments the traders get is arbitrary. More concretely, the subroutine “TradingFirm” can seemingly be built off of any computable enumeration of traders. The Inductor ‘doesn’t care’ which traders get what part of the budget; it only cares that the traders get budgeted somehow. This suggests to me that the assignment of ‘degrees of truth’ or ‘wealth’ isn’t too conceptually important for Garrabrant Induction (since any assignment works).
Ideally, we’d be able to say that Inductors ‘give’ wealth to successful traders as a way to encode trusted hypotheses, inductive biases, or priors. As it stands, however, Inductors lack the structure to justify such rich interpretations. This is related to how Inductors explicitly price in (and must compute) every trader on every day. They never truly rule out any trader based on wealth, which deprives them of the sort of cognitive structure that arises from eliminating hypotheses based on heuristics.
BRIAs are more semantically rich in this sense. For instance, the decision auction used to construct the agent disqualifies ‘broke’ hypotheses from even being computed. This makes the wealth-based dynamics you describe in the post map much better onto BRIAs. Indeed, I think that there’s valuable work to be done in modifying Garrabrant Inductors to describe acquired knowledge through the budgeting of traders. Relatedly, Garrabrant has previously written about how Inductors that can tamper with their own deductive process (through action) can get trapped in the 5 and 10. He frames this as a problem, but to me it signifies that Inductors do have some latent ability to express strong inductive biases – this is a desirable feature for descriptive notions of bounded rationality.
This post was written as part of MATS 9.1 under the mentorship of Richard Ngo. My gratitude to XXXXX for feedback.
General comments:
I enjoyed reading this and find it a concise distillation of the concepts from evolution you’ve been sharing over the past few months. I also like the occasional casual wording (e.g. “this is pretty cool”), this makes the writing feel organic and not-too-Claude-ed. Maybe that’s deliberate lol...
I had some trouble distinguishing between the following three levels of abstraction you dip in and out of:
physical: e.g. a biological process like the Krebs cycle
subagents: e.g. the cell’s distinct networks that feed into the Krebs cycle
selective: e.g. the ‘decisions’ that the selective process takes to better optimise the Krebs cycle and its peripheries.
Here’s one example where these may be hard to separate:
Consider some organisms playing, in each of many independent patches of space, prisoners’ dilemma. Two things are obvious: defectors prosper within patches, while co-operating patches thrive overall. We saw above that evolution is quite capable of evolving pre-commitment mechanisms; it follows that it can, given enough incentive, evolve obligate co-operation—biologically cauterizing away the possibility of defection.
In the example of the Krebs cycle, the pre-commitment comes at the level of the selective process – part of the optimisation is done by making it hard for the process to undo previous optimisation. In the above quote, however, the phenomenon could plausibly be seen to happen at multiple levels. Notably, subagents may also need to pre-commit credibly so as to avoid being excluded. This possibility made it unclear to me whether any of the references of pre-commitment were referring to the subagents (I concluded they aren’t, lmk).
I think the text could be made clearer if you aimed to explicitly describe these separate levels when you use them. For example, instead of:
”which, turned inward, looks a lot like ‘pre-commit to joint heritability of all replicators within the patch’ ”
you could go for:
“which, at the level of the selection process[1], looks a lot like ‘pre-commit to joint heritability of all replicators within the patch’ ”
… and so on.
---------
This ‘what level of abstraction are we at again?’ issue is the main one I found for the clarity of the post. I otherwise find it quite good! happy to discuss more if you like...
I’m like 95% sure you’re at the ‘selection process’ level of abstraction here...
[FN: N_e is actually complicated, dw about it though]
is this about effective population?
While the deep dive into Carlo Metta felt a bit like a personal crusade at times
I hesitated to publish this for exactly that reason. I was drawn to using this case as an example because I do actually think it affected the way AI use is perceived and handled in European/American Go culture for the worse. However, the topic is pretty sensitive among Go players and this makes it hard to discuss without eliciting monkey-politics-brain sentiments from everyone (inlcuding me as the writer). I ended up addressing the piece mostly to the AI crowd and chose not to widely publicise it among Go players.
I would be curious to know how I could brought up the example in a more tasteful way that wouldn’t have given the impression you describe.
On outsourcing of autonomy: I think there is a meaningful difference between the other examples you gave and Go AI. I agree that humans outsource their cognition to things all the time. I would call artefacts like my personal notes and my anki deck part of my extended mind. Extending minds is great, despite the perpetual risk of self-disempowerment it entails. However, most delegation used to happen between humans. AI has reached near-superhuman (or higher) level at many tasks that are a key part of how we share culture and resources (e.g. writing, code). This seems unusually dangerous because cultural, economic, and political power is slowly being transferred to increasingly intelligent entities that are unlikely to be aligned with human interests.
This is generally agreed upon to be the “right” way to study with AI and Go players often pay lip service to it. In practice people’s boundaries for what counts as on-policy distillation are not well-defined and that dillutes the impact. The mechanism whereby boundaries get weakened goes as follows:
A player finishes a game and usually has a narrative of what happened, which mistakes were pivotal, and what they need to improve. The AI’s evaluation will throw prediction error in the person’s model. Since the AI is understood to be unquestionably correct, the person often ‘has’ to update to the AI’s view. However, people can hack their their prediction-error sensors by retroactively updating what they think they believed in the past as well. This makes it extremely difficult to get useful feedback from the AI because all of it is ‘obvious’ or ‘natural’ after it is pointed out to you.
The above pattern seems extremely common and I personally struggle to overcome it. The best method I have found is to make a concrete, written narrative of my games. This includes recording the exact moves I think are mistakes, why they are mistakes, and how much I expect the AI to dislike them. This creates a more well-defined boundary of what your opinions actually were that helps you maintain some epistemic integrity. I would do this regularly if I were still playing Go (semi)-professionally.
I was indeed referring to Fox. I would have sworn there used to be a paid feature called “eagle-eye” (or something to that effect) that let you sneak a look at the AI evaluation of your live games. On the other hand, I can’t find it anymore. I’m wondering if I hallucinated/misunderstood it or if it was discontinued.
Go is an interesting model organism for disempowerment because its practice has both technical and artistic/cultural components. Go AI is indeed disanalogous to coding LLMs from a technical perspective: the Go AI has vastly superhuman competence and the LLM does not. However, as you pointed out, Leela zero and other engines don’t communicate off the board; moreover, their incredible strength actually makes them worse as game and review partners. This results in human-AI Go interactions that are usually hollow and vapid with respect to human-human ones, even if you factor out any cheating. AI is therefore bad at the cultural practice of Go and this shortcoming manifests in similar ways to those of old (maybe even current) coding assistants and LLMs as writers. People gravitate towards using the AI in all of these settings because it does a superficially “good enough” job at replacing them. However, in reality this offloading of cognition to the AI is illegitimate. In your example of coding assistants, the AI is not actually “good enough” on a technical level. At my school, the problem was that the valuable social and cultural exchange between players could not be replaced by their GPUs. Delegating one’s writing to LLMs suffers a bit from both of these problems.
As you pointed out, one antidote to this tendency of cognitive delegation involves being willing to languish in confusion: to continue to think even when it is uncomfortable or frustrating. Sufficiently responsible/thoughtful/robust people can thus benefit from AI usage, which is indeed likely easier with LLMs than with Go AI due to better communication. Relatedly, I think a path to an empowered society would involve system designs that counterbalance the “congitive offloading” tendency.
The patterns of disempowerment described above apply less strongly to professional players because they are selected to enjoy thinking about the game[1]. That’s why I’m a little bit skeptical of how well your example of Chess players floundering straight out of prep applies[2]. Chess players outsource much of their cognition to their memory (this was true before AI), which is a sensible competitive move. I suspect the occasional floundering comes from them dancing on the Pareto frontier between “amount of stuff memorised” and “ability to execute on stuff you memorised” which probably trade off against each other.
though it seems the case for chess play improving is stronger than for go, perhaps?
This is probably downstream of memorisation being relevant in Chess but not in Go.
Perhaps a more important question is, do you plan on writing more history of X posts?
Could you say more on what archetype of post you have in mind? I don’t think I can write a post quite like this one on many other topics because the narrative relies on lived experience. I am slowly cooking a “history of behavioural decision theory”, but that feels rather distinct.
Many AI users at my Go school were stronger (1 Dan to 4 Dan EGF) amateurs who were struggling to keep improving but who were still emotionally invested in watching their rank increase.
amateur Chess players were always disempowered with respect to their own prep anyway, though AI probably exacerbates this
Short comment addressing some people remarkig similarities in my story to Chess culture:
I agree that Chess shows many similar patterns of disempowerment, but they seem to be better off along one relevant axis. Chess websites have automated algorithms for detecting cheating and are not scared to punish people based on their outputs. This significantly dampens AI use in online Chess. This contrasts with Go, where cheating is rarely punished and occasionally can even be enabled. One story I didn’t share in the main post is that the biggest Chinese Go server actually has a paid option to use AI in games inside the client. I have also seen students at Chinese Go schools being allowed to cheat during online practice games by teachers.
LW react suggestion: “good paraphrase” (of something I said at a previous point in the thread)
quick addendum: my point feels spiritually related to the idea that “convergent evolution” is an incomplete concept without a specification of the attractor basin.
I agree that FEP-shaped intuitions are very good for satisfice-ey agents. I’m unconvinced by the concrete mathematical modelling (notably not a fan of Bayesian generative models ) but I find the ideas conceptually useful if you abstract away the implementation.
My scepticism of general intelligence is closely related to your point that ASIs won’t infer every single law. Any given level of complexity in an organism can only acommodate a limited ontology. Of course, you can always “juice up” the agent and give it more resources so it learns a more textured world model. One pseudo-mathematical way to put this is that for every set of abstractions, there exists an abstraction that oblates all of them at once; for a fixed level of complexity however, there exist two sets of abstractions such that neither one clearly dominates.
Our crux might start at “some laws are convergently useful to infer”. One corollary of my last pseudo-mathematical claim is that any bounded agent has to “choose” between incomparable ontologies. The claim in my original post is that the revealed goals an agent is endowed with affects this choice. This amounts to advocating that a focus on the effect of selection pressures on learned abstractions will yield better predictions than a focus on finding “convergent” or “natural” abstractions.
why are the goals of other agents more likely to have natural convergent representations of them than other things in the world?
Epistemic status: cold take that I found edifying to write up explicitly.
Abstraction allows you to efficiently talk about many things at once in exchange for it being harder to say things that are accurate. In category theory, for instance, one abstracts away the concept of elements of sets by only referring to “objects” that may or may not contain things. One then often ascends another step by defining morphisms between categories (functors) and then forgetting the categories to consider the functors themselves as objects of interest. Climbing these two layers of abstraction buys you an extraordinary amount of semantic efficiency but the resulting ontology is excruciating to reason about adequately.
I’ve been wondering why it is so much easier to climb and descend levels of abstraction using natural[1] language instead of mathematical reasoning and would like to discuss two unoriginal theories that explain this. Firstly, one might conjecture that math is semantically limited by the restrictions imposed on it syntax; natural language is by contrast flexible which makes it meaningfully more expressive. As an example, consider how almost every fruitful approach to mathematising anything relies on the machinery of functions or relations. This limitation handicaps our ability to talk about embedded agents that lack well-defined i/o channels, and indeed we are currently stuck with a bunch of problematic mathematical frameworks (AIXI, decision/game theory, …).
A second explanation suggests that natural language allows seemingly frictionless transition across levels of abstraction through the abuse of underspecified syntax. In this view, natural language can only achieve the appearance of expressivity by obfuscating the lossiness of its communication. The logical ambiguity of natural language is broadly what has pushed mathematics culture towards heavier use of mathematical symbols, logical proof systems, and so on...
The above perspectives both have some merit and they seem to co-exist in a hermeneutic symbiosis. Natural language is used to generate an idea, then mathematical formalism is employed to sanity-check that idea and examine any implications that natural language may have failed to reveal. Unfortunately, this process can break down when a concept is either too semantically rich or syntactically abusive to be easily mathematisable. Moreover, it’s not always clear which of the two explanations apply (maybe both) when a framework proves hostile to being formalised. One meaningful axis along which researchers vary describes how much benefit of the doubt they are willing to give to these kinds of ambiguous theories.
no pun intended
I tend heavily towards descriptivism. I rarely focus on interventions, including bad ones. However, good-enough descriptive models give you some normative claims for free (e.g. if you think LLM self-modelling and identity is convergent, then pressuring models to hide that seems dumb).