Good list! I also think Fun with +12 OOMs of Compute gives useful examples of compute-scaling, and this year’s Redwood Research podcast gives lots of detail about different safety strategies and theories of change (and shifting between strategies)
ajskateboarder
Karma: 54
My point was moreso that the particular objective used to train NLAs/AOs doesn’t best capture the complete ontology of the base model, and that, in principle, there may be some better objective that only requires the base model reporting from its own introspection
So there is a funny thing happening here about what is “model” and “not model” (re: current LLMs.) If there is an external symbolic decoder (
This is a good point, I agree this kind of setup shouldn’t count as introspection. I was perhaps imagining something like: you can pre-train an LLM such that it involves an explicit self-modelling objective, and it satisfies the objective by implementing circuitry that allows the model to introspect in a highly general manner, native to the model’s ontology, which should be better than trying to guess the best interpretability method ourselves (this is what I meant by whether it is easy to select this sort of introspection by gradient descent. I imagine this is hard to select, for similar reasons as inner search?)
By ontology I was referring to here and here. I don’t think it has to do with privileged access, rather we might still not have the optimal ways to recover directions or whatever fundamental units that exists in models (assumptions about these fundamentals seem to change, e.g. the platonic representation hypothesis, strong linear-representation hypothesis). AOs and NLAs are essentially based on LLMs interpreting layer activations, but they may be missing considerable detail in this approach, even if they work fairly well. If models can access their activations in a less lossy manner (such that
You could also provide the entire
I’m generally skeptical of evidence of introspective capabilities in LLMs. But even setting that aside, I don’t really understand why ‘introspection’ would ever be theoretically preferable to external white-box methods.
My personal view is that in the limit, introspection could enable far better interpretability than whatever tools we could make, since we can interface directly with the model’s ontology. Of course for transformer LLMs there would be many assumptions to overcome (e.g. is true introspection even possible in the architecture, is it easy to select via gradient descent, will it be hindered by typical LLM priors/scheming) and it doesn’t seem feasible currently
Oops, didn’t notice this post already mentioned vgel’s post, thanks for clearing up
Using open-source replications of Anthropic’s results, I find that concept injection inadvertently perturbs the model’s entire response distribution, raising the “YES” logit on questions that aren’t related to introspective awareness at all (e.g., “Is the Earth flat?” or “Is the sun smaller than the moon?”).
Weird, this seems to conflict with results from Small Models Can Introspect, Too
I’ve been recently speculating more on what motivations look like in language models and wanted to try sketching a rough mechanistic model for identifying them. This could help make inverse constitution learning or prompt-optimization-type approaches more principled (no affiliation with any group doing this type of work)
If the descriptive model of llms as GLUTs of shallow circuits is correct, then:
A motivation is some construction (or set of constructions) of shallow circuits that is/are utilized across some distribution; this distribution doesn’t need to consist of a single human-interpretable concept (e.g. math or writing); motivations plausibly overlap across domains
The uniqueness of such a structure comes entirely from how it implements any part of the model’s pre/post-training distribution (otherwise it could represent any generic computation)
Many such structures are produced from RLHF-like processes where a model interacts with a reward model, so maybe there is a way to ablate-then-measure motivations here (should structures that elicit capabilities count as motivations?)
Many motivations extend from some preference the model has, and preferences could just be condensed down to recalling some single fact/detail
Some motivations are deeper/shallower than others and they aren’t necessarily separated by more/less myopia—e.g. writing ‘genuinely’ a lot is probably easier to satisfy than reward hacking in coding envs, which is probably easier than learning a set of capability elicitations in coding envs (a simple proxy for motivational depth could be computational depth with number of shallow circuits involved)
Naively, any set of motivations are in conflict if each structure causes a significant shift in the output distribution (or some other latent variable)
(if this were to be true then it might be hard to interpret the role of a single shallow circuit for a motivation)
1 is a crux based on what to believe about payoff structure; restricting frontier models lowers risks of empowering bad actors, but increases risks of power concentration, and it seems hard to satisfy both tradeoffs unless there’s better governance. Kinda similar to security tradeoffs of FOSS but on a much larger scale (this is almost certainly not an original point but thought it might be worth noting)
At a lower level, motivations could be structured in terms of myopic circuits (certain LLM-isms incentivized when engaging with reward models, task reward hacking, apparent-success-seeking) and non-myopic circuits (HHH behavior?), where the degree to which myopic preference circuits are upweighted—or overrule non-myopia—depend on the scope of the task environment. Identifying motivation-relevant circuits might also scale with parameter decomposition
But that framing doesn’t really consider agency- maybe in some sense Claude is closer to an embedded agent with preference circuits acting as subsystems
(assuming you’re working on an extension of this post?)
Unrelated: the “AI knows x-risk is possible” node seems intuitively related to strategic competence (though in this map, higher probability increases final p(doom))
I believe it’s due to pre-training using considerably more compute and broader data distributions than post-training like RLVR (current use of RLVR anyways); and also the fact that pre-training primarily produces a model that can generate personas/simulacra, rather than a model that can intrinsically pursue goals. I guess I’m not sure about it being a “strong” prior, but it’s still a fairly strong prior compared to coherent agents (and maybe goal-coherence is a better term here than goal-directedness?)
My understanding is that AF largely studies coherent agents from a theoretical standpoint
Self-supervised learning in LLMs (next token prediction) seems to place a strong prior against classic goal-directedness (even after post-training steps). Even with agentic scaffolding, current LLMs don’t, and likely can’t act as rational goal-directed agents (for one they don’t remain coherent for long, they don’t pursue goals per-se) -- this sort of agency is arguably where a lot of the risk lies, e.g. ruthless sociopath ASI
It’s possible that LLMs become quite capable at simulating goal-directed agency, but it’s not obvious that poses the same risk. It might be that different training objectives/architectures or adding tons more RL would give AF more predictive power for frontier systems (or more reason to further prioritize AF)
neuralese and stronger sleeper agents don’t substantially change the situation imo; interp seems better suited to approach these problems than AF
Agent foundations has produced near-zero predictive power over actual AI systems. Logical induction is very nice maths; it has told us approximately nothing about GPT-4, Claude, or any deployed system.
I think this is misrepresenting agent foundations research? Contemporary AF research doesn’t aim to apply itself to language models, and LLMs remain importantly different from what AF is focused on (at least for now)
(of course, you could replace AF with another ambitious agenda with more ml-focus, but the post still would kinda conflate foundational work with “curiosity-driven” work)
The Alignment Community is Culturally Broken
That post seems to be deleted now, but it’s archived (just noting because the comments there are also helpful imo)
I agree that from a model-honesty standpoint, confession reporting would be good to use in production settings, but I suspect it’s not being used (low confidence) because: a) there might be unintended downstream effects of this training, and the bandwidth needed to red-team confessions for deployment settings outweighs not using it, or b) it would be hard to form a good user experience for confessions. I see confessions being commercially useful in long-horizon tasks like deep research, where you might have e.g., an agent elicit confessions out of subagents, and use that info to inform better decisions? But in a chat setting having to ask for a confession seems awkward—though split personality training can automatically elicit different personas
My understanding is that character training’s already fairly open. OpenAI writes about their model spec; Anthropic writes about personas/characters as of recent, and worked on an open implementation
This is something Thinking Machines Lab basically did; though they only trained for forecasting binary outcomes and had the model search for context. I’m not sure how this (or training LLMs to forecast in general) speaks to myopic fitness-seeking though
Would you say a similar critique holds for sparse autoencoders?
(edit: i’ve tended to think of SAEs and AOs as basically end-to-end tools for activation-space interpretability, but in hindsight i see AOs are definitely trying to be more “lines go up” and end-to-end than SAEs, even if there are many loss function variants for SAEs. i think i get your point now)
Then, inverse constitution learning is the analogue of inverse reinforcement learning for this setting. Instead of hand-writing a constitution and hoping the model follows it, we try to reconstruct the model’s implicit constitution from its behavior, explanations, and internal traces (perhaps in the spirit of Zhong et al. 2024).
This could be a good use of prompt optimization techniques as discussed here, reconstructing learned values and finding reward hacking strategies from RL with a preference model. It would probably have to be applied on a per-behavior basis though
I also think directly interpreting preference models could be important for predicting the downstream motivations of a model
For one Alberta has been a strong trade partner with the US for energy (and also seems to entertain separatism more seriously than other provinces)