What would a really inscrutable CoT look like? It would look like random tokens
...
the fact that CoTs are still pretty legible is some evidence that RLVR today has not had that much influence on LLMs, as compared to imitative learning
I think there’s more middle ground here than this characterization suggests, and that the target might not be right. CoT that’s legible isn’t good evidence that LLMs are not wielding (or could not wield) novel latent abstractions/reasoning primitives/control heuristics. IMO expecting inscrutable CoT is too narrow a target for cognitive inscrutability. E.g. a trace can remain intelligible while being highly compressed, semantically shifted, incomplete, causally unfaithful, etc.
I agree that current architecture and initialization strongly favors natural language as the persistent serial workspace. My objection is that this tells us less about how much RL has changed the latent evaluations and control policy between token emissions.
This plays out in human contexts: the Go/chess player, mathematician or scientist, say. These people have “out of distribution” insights that, externally, are mediated via legible primitives: concepts and attendant language. Despite that, their internal representations can fall into novel basins of attraction (reframes, new concepts), often ones that a priori don’t have concrete associational nodes, i.e. discrete concepts, words/terms. (I’m talking in terms of “insight” because it’s a decent human analog for novel reasoning trajectories/policies.)
Experts on the edge of their domain (or people learning a new domain with limited exposure to its conceptual tools) often articulate the process of insight as involving non-conceptual gestalts, affectively loaded associations, leaps, and manipulations their mind is doing that can’t be captured in words. None the less, their scratch pad work and communication is in legible form—it compresses the loose gestalts, materializes them into discrete, familiar units connected in known ways.
LLMs too, seem to perform much of their computation through nonlinguistic representations and compress the actionable results into language, mostly because of the structural thing that language is the medium through which they maintain serial state and communicate.
This is an argument against
if you create AI capabilities via imitative learning, you get models that follow the human distribution of outputs
Some thoughts on 1.1 since I see its grammar used fairly often in this setting: I don’t think the capability for novel representation and concept manipulation is “linear” in weight update mass. Another comment thread addresses it well, but I think a bit of intuition/grounding might be helpful, in particular because this piece is largely focusing on how to think about capability gain.
Novelty builds heavily on existing frameworks and often involves a subtle reframe, reconfiguration, or saliencing of a known/slightly modified concept in a new setting. Insight is about discovering relevance, but what’s available to be relevant is often familiar primitives.
Mathematical work is a great example here, since mathematical discoveries very often have a subtle kernel of insight, a small new idea, that reconfigures existing concepts around a problem in a significant way to illuminate something unknown. The reconfiguration is almost all in terms of familiar stuff—the moving pieces don’t change much. In the case of LLMs, that’s the content imitative learning provides.
To the other things you said in 3.3, though, relative to imitative learning and to the extent that learned control heuristics transfer (they seem to), RLVR clearly yields a significant chunk of generalization capability by changing how pretrained reasoning procedures get selected/composed. That said, I don’t think the size of the RLVR weight update is much signal, even just on intuitive grounds.
My personal takeaway here is that 1) the picture remains mixed, 2) the way that imitative learning and RLVR interact probably matters more than an attribution-belongs-here-vs-there analysis suggests, and 3) we need to be careful about choosing our targets when trying to attribute capability gain.
I think there’s a prior problem here, before whether good mechanistic explanations exist or can be tractably found: I’m not sure alignment-relevant structure is the kind of thing that can be recovered from the model’s computation.
As I see it, alignment isn’t primarily a property of a model, it’s a relation between the model, a person/group, their interpreted intentions, the context in which those intentions arise, the system’s actions, and the resulting trajectory through the world. The same behavior can be aligned in one setting and misaligned in another. E.g. following an instruction literally can constitute useful assistance, negligent literalism, manipulation, or appropriate refusal depending on facts that aren’t present in the instruction or the model.
ARC wants to explain training-time computation, use those explanations to predict generalization, and eventually define better loss functions from those predictions. But to make an explanation useful for alignment, something has to select which distinctions in the computation are alignment-relevant.
In roughly the language of the project, we want to mod out the mechanistic detail that makes no relevant difference and recover a latent structure in which the important properties are salient. That requires an equivalence relation, something like
But now those words are just carrying the entire problem:
acceptable to who (relative to their stated instruction or underlying purpose? Under which interpretation of what they wanted?)
over what time horizon
given which permissions, obligations, false beliefs, unknown facts, and effects on other people
what makes outcomes equivalent
which situations are relevant
what happens when the system’s actions alter the distribution of future situations
I don’t think there are model-internal facts that answers these questions. The weights etc determine what computation occurs, but not which quotient of that computation corresponds to serving human purposes appropriately. Specifically, there are infinitely many valid projections: ones that compress the computation, predict outputs, recover learned algorithms, or distinguish behavioral modes. Mechanistic completeness doesn’t select the normatively relevant ones.
I think alignment with human intent is almost maximally domain-general. We aren’t trying to formalize success in chess or whether a sorting algorithm returns an ordered list. Trying to capture the deviation between what people want and what a system causes while operating in the actual world is a different category of problem. In ordinary human contexts we have nothing close to a general metric for this.
Consider whether an employee acted in alignment with a manager’s intent. You need, at least, the request, purpose behind it, whether the manager was mistaken, relevant institutional and moral constraints, effects on others, whether changing circumstances justified deviation, whether clarification was possible, whether the outcome was competently produced. I doubt there is even a coherent, generalizably definable object capturing “what the manager wanted.”
The issue seems to be that the relevant ontology is contextual, interpretive, relational, partly normative. Formal tools can reason rigorously when given a state space, specification, distribution, failure condition, etc, but they don’t themselves tell us what the right state space is, what someone meant, which consequences matter, or what should count as failure.
A recurring move in alignment work is to shunt this into an abstraction: reward function, catastrophe detector, deployment distribution, preference oracle, specification. The formal work then proceeds rigorously with the abstraction hoped to do most of the aligning. Finding the computation that produced a behavior doesn’t tell us whether it amounted to, say, truthfulness, manipulation, appropriate correction, or justified deviation from a user’s surface instruction. This kinda post-modern problem is, IMO, central to real alignment happening in the world, and, descriptively, seems outside the scope of purely technical epistemics.
To be clear, I’m not arguing that formal or mechanistic guarantees are useless, since we can, e.g., specify bounded relational properties like whether a system accessed data it shouldn’t have, concealed information, executed an irreversible action without confirmation, violated a domain-specific expectation. I think mechanistic explanation could provide strong assurance about these. But it works by restricting the world until the relation becomes specifiable, i.e. modding out exactly the parts we care about most. The output is bits of alignment that are still embedded inside a broader interpretive and institutional process (not in the model or its deployment context) that remains needed to determine which properties matter. This is an instrumental necessary move borne of epistemic constraints, but counter to the practical goal.
Suppose ARC can achieve its goal as outlined here. There’s a further claim (the more important part IMO) that a sufficiently good explanation lets us identify the causes of alignment-relevant behavior and train against them, but that requires alignment-relevant distinctions to be recoverable from the computation. I don’t see why they would be. Alignment is externally constituted by the relationship between agents, intentions, context, and world trajectories.