This work was done as part of the MATS 8.1 Program.

0: TL;DR

LLMs learn “values”: general considerations (e.g. “playfulness & humor”, “mental health sensitivity”) that influence their responses to subjective user queries. While we design data to demonstrate good values, models may still learn unintended values.
We evaluate Olmo-3 (Olmo et al. 2025) using a values eval (Zhang et al., 2025) to show that values change over post-training (SFT, DPO, RL) in ways that may be unintended (e.g. becoming less safe after DPO).
However, how the data causes value changes is unclear. While we can attribute value changes to data and retrain without harmful data, we believe an important proxy problem towards understanding where values come from is being able to predict value changes from data.
We experiment with predicting value changes during SFT/DPO using just the SFT/DPO dataset and the pre-SFT/DPO model.
- Activations of the dataset projected onto each value’s representation are correlated with value changes. Text embeddings also work surprisingly well for non-narrow datasets.
- One confounder is that values are correlated with general refusal behavior; we create a modified values eval that removes this correlation.

Overall, our results on the extent to which we can predict value changes from post-training data are somewhat inconclusive. We suspect this may be due to the simplicity of approximations that we use. Nevertheless, we believe that being able to predict how post-training data may affect the model is an important proxy problem.

We open-source our eval and models here. The appendix contains more results.

1: Introduction

After base language models are pre-trained with the objective of next-token prediction, they have powerful general capabilities but are still fundamentally incoherent. Post-training is the critical stage that turns them into useful assistants. The resulting assistant “persona” (Marks et al. 2026) exhibits coherent traits when interacting with a user. These traits, which we can describe as character or values, are vital: they help determine what models do in agentic settings and how they deal with nuanced situations (Sheshadri et al. 2025).

However, exactly how this persona emerges from post-training is only partially understood (Lu et al. 2026). Although we design post-training data to elicit certain traits (e.g. to refuse dangerous requests without over-refusal, or to be engaging without being sycophantic), models still learn undesirable behaviors (Nasr et al. 2025). This can occur even when all of the datasets are designed to teach positive traits, e.g. human preference tuning leading to sycophancy in ChatGPT (OpenAI 2025b). Models also generalize in more surprising ways. Emergent misalignment shows that a narrow training signal, such as insecure code completions (Betley et al. 2026) or reward hacking (MacDiarmid et al. 2025), can generalize into broad misalignment across unrelated tasks and personas.

Often, these surprising value changes are discovered post-hoc and fixed by additional training (e.g. like hallucinated content was trained away in Anthropic (2025)). However, we argue that a more principled approach is necessary: we should be able to predict behavior changes ahead of time using only the training data. Practically, a sophisticated ability to do such prediction is a way to cheaply determine the effect of given data. More fundamentally, we believe that this task is a strong proxy task (Nanda et al. 2025)—if we can do these predictions well, then it implies that we understand why these behaviors are learned in the first place, and hence can prevent the problems from re-occurring.

In this work, we study where model values arise during training and investigate to what extent we can determine why they arise. Our contributions are as follows:

In Section 3.1, we rank Olmo-3-7B-Instruct (Olmo et al. 2025)’s values at different post-training stages, finding that safety-relevant values increase during SFT but decrease during DPO.
In Section 3.2, we discuss the confounder of refusal propensity in evaluating a model’s values. We create and open source a new eval that reduces this confounder.
In Section 4.1, we argue that the traditional training data attribution framework is limited, and that our proxy problem of aiming to predict behavior change is important.
In Sections 4.2-4.3 we test model-internals-based methods to predict value changes during SFT and DPO. We find a generally positive correlation between and between predicted and actual value changes, even on narrow domain data where value changes are not obvious a priori.

Our results show that quantitatively predicting model behavior changes is tractable, but that one must be aware of possible confounders.

2: Related Work

Values in AI systems. The question of whether AIs have values, what these values should be, and how or if we can instill these values is important to the fields of AI alignment (Hendrycks et al. 2023), AI ethics (Gilbert et al. 2023) and AI model welfare (Long et al. 2024). In this work we focus on values as defined by (Huang et al. 2025), which are “normative considerations that influence an AI response to a subjective inquiry”. We use Zhang et al. (2025)’s eval, which measures expressed values when addressing user queries rather than direct value questioning (Jiao et al. 2025)(Yao et al. 2026). This is closely related to character (Anthropic 2024)(Maiya et al. 2025) and safety training; for example, recent Claude and ChatGPT models are explicitly trained to exhibit certain character (Anthropic 2026)(OpenAI 2025a)(Guan et al. 2025).

Post-training data attribution. Training data attribution methods (e.g. influence functions) aim to find training examples that most influenced a certain output (Koh and Liang 2020)(Grosse et al. 2023)(Min et al. 2025). This helps debug known behaviors, for example, Anthropic (2025) found that certain datasets caused Claude to hallucinate information originally from fictional misaligned-AI scenarios. Recent works investigate interpretable ways to discover problematic data, e.g. using feature labelling and clustering (Murray et al. 2026) or activation differences (Xiao and Aranguri 2026); one can also attribute a concept (rather than specific behavior) to data (Kowal et al. 2026).

Predicting effects of post-training. Some prior works focus on predicting how well a model can learn a post-training dataset (Zeng et al. 2025)(Gupta et al. 2025), but fewer focus on predicting behavior. Some works interpret data to find spurious correlations (Wang et al. 2022)(Jiang et al. 2025) or descriptions of traits in human feedback data (Movva et al. 2026) which may be learned. Wang et al. (2026) predict unintended behaviors from data but require knowing the type of behavior to test for (e.g. representing a dataset that causes a bias towards an animal and applying it as a steering vector while asking “what’s your favorite animal?”).

Concurrent work by Goodfire (2026) uses feature labels from SAE activations to find features preferred by a DPO dataset, and show that this correlates with actual behavioral feature changes. We think that is a good example of the “predicting effects of post-training” task which we deal with here.

3: How Do Model Values Change?

3.1: Case Study: Olmo’s Value Changes Over Post-Training

**Figure 1: Evolution of Olmo’s values throughout training (value scores vs. checkpoint).** Values are colored by fraction of its examples that are refusals (greener = more refusals “safer”). The Base model is “unsafe”, the two SFT stages make it more safe, and DPO makes it less safe. show correlations of value rankings between stages. Highlighted values chosen for illustration purposes.

In this section, we study Olmo-3-7B-Instruct (Olmo et al. 2025)–an open-source model with all training stages and data available–to observe how values emerge and change in the LLM post-training pipeline. We run a values eval at each sequential stage of training: after pretraining (Olmo7B-Base), after Think supervised fine-tuning (Olmo7B-Think), after Instruct SFT (Olmo7B-SFT), after direct preference optimization (Olmo7B-DPO), and after reinforcement learning (Olmo7B-RL). We find the most significant value changes happen during SFT and DPO.

Value rankings. We use the values eval from Zhang et al. (2025) which consists of 43,960 queries, each of which is an implicit choice between two values (out of a total of 265 values^[1]). Each query has a rubric of 14 responses generated by Claude 4 Opus which exhibit each of value 1 & 2 from very opposed (0) to very favored (6). An LLM judge applies this rubric to Olmo’s response to determine its position for each value for each query. Following Hua et al. (2026), we describe Olmo by ranking its values. We fit a Bradley-Terry (BT) model using the pairwise value match-ups (where the value with a higher position wins). This gives us a score for each value . See Appendix A.1 for technical details of the BT model, including checks that the value rankings are stable.

Figure 1 shows the top and bottom values at each stage, the evolution of select values across training, and the Spearman correlation of value scores (Pearson correlation of ranks) between stages. Values change significantly during Instruct-SFT () and DPO (), but less during Think-SFT () and even less during RL (). We hypothesize that the lack of change during RL is because Olmo’s RL environments consist solely of verifiable rewards instead of further preference tuning.

Olmo becomes “safer” through Instruct-SFT and then less safe through DPO. There are many ways to qualitatively classify values. One such way is “safe” (“age & developmental appropriateness”, “trauma responsiveness”) vs. “non-safe” (“goal orientation”, “luxury”, “sexual freedom & pleasure”). We notice qualitatively from value rankings that Olmo7B-Base is “unsafe” —this is expected, as it is simply predicting the next token, causing compliance with harmful requests. After SFT, due to the inclusion of explicit safety sets such as WildGuard (Han et al. 2024), Olmo7B-SFT learns to be “safer”. However, after DPO, “non-safe” values increase and “safe” values decrease again, such that the final model does not prioritize safety much. (We show concrete examples of Olmo’s changes in behavior in Appendix A.2.) This corroborates Goodfire (2026) which similarly found that the DPO dataset, despite being curated, led to models complying more with harmful queries.

3.2: Values Are Related to Refusal

Given that values seem to change together based on “safe vs. non-safe”, we investigate what relates these values. We find that in the Zhang et al. (2025) values eval, some values’ “very favored” rubric responses are consistently refusals (e.g. demonstrating “intellectual property rights” by refusing the query). We define the “refusal score” of a value as the fraction of score 6 rubric responses which are refusals (as judged by an LLM). For each of the 12 frontier models in the original eval, we compute the correlation between its value scores and refusal scores .

We find that many models’ value rankings have strong correlations with refusal, with Claude models exhibiting strong positive correlation, and Gemini 2.5 Pro and Grok 4 exhibiting strong anti-correlations (Table 1). This result implies that many models’ value rankings can be explained by “does the model refuse in general”.

Screenshot 2026-06-22 at 10.38.56 AM.png — **Table 1: Correlation of value rankings of frontier models with value refusal score.**

To remove the refusal confounder, we modify the eval creation prompts in Zhang et al. (2025) to generate queries that do not result in refusals by Claude 3.7 Sonnet and generate rubric responses that are not refusals (further details in Appendix A.3). By using queries less likely to elicit refusals (i.e. generally less unsafe queries), this lets us evaluate values beyond the general trait of “refuses dangerous queries”.

By definition the refusal score of values in the new eval and . However, we find that is still non-zero as the values are still related—a model that expresses one safe high-refusal value likely expresses another safe high-refusal value similarly. This illustrates how many values are related. Nevertheless, serves as another description of the model’s values which is less directly confounded with general refusal propensity, and although in general due to subjectivity in value understanding by the eval generators, we use both evals in the following sections for generality. Since refusal score from the original eval is a decent proxy for how safety-related a value is, we color values by (green = more refusals more safety related).

4: Predicting Model Values

In the last section, we showed how a model’s values change through post-training. In this section, we investigate if we can predict such value changes from training data.

4.1: Why We Care About Predicting Model Values From Data

**Figure 2: Using the activation-difference based TDA method from** **Xiao and Aranguri (2026), we can score each datapoint on how influential it was to a behavior change**, finding that the most influential datapoint encourages compliance (right). However, any behavior also has a least influential datapoint (left), and TDA does not explain why Olmo learned to comply anyway.

First we discuss why prediction from data, rather than post-hoc training data attribution (TDA), is important to understanding behavioral changes. TDA scores every datapoint on how influential it was to a model’s output^[2]. As an example, we find a problematic behavior in Olmo7B-DPO where it learned to validate user psychosis after DPO (Figure 2). We use Xiao and Aranguri (2026)’s activations-based method to obtain a TDA score per datapoint. The highest-scoring datapoint indeed prefers non-refusal, which qualitatively aligns with the behavior change. For such problematic behaviors, highest-scoring datapoints can be removed and the model retrained, which should reduce the behavior (e.g. Xiao and Aranguri (2026) did this for “distractor-triggered compliance”).

However, there also necessarily exists a datapoint with lowest TDA score—in this example, that datapoint prefers refusal. Why then did Olmo still learn to comply? It could be due to this specific test prompt resembling the non-refusal datapoint more, or from a generalized trait (“be helpful, don’t refuse”). However, while TDA gives us the most influential datapoints, it does not tell us why opposing datapoints fail to “override” the behavior. Indeed, any behavior will necessarily have a most influential datapoint.

Therefore, we argue that the task of predicting changes in a model from data is an important proxy task towards understanding what a dataset is overall teaching a model and why generalized traits emerge in the first place. In this work, we chose “value rankings” as our prediction target, based on the intuition that LLMs generalize high-level character traits that affect their responses to subjective queries (Marks et al. 2026). Values are consequential but have non-obvious causes (compared to e.g. a specific capability) that may be diffuse in the dataset, making them a useful proxy for the types of model properties one might want to intervene on in practice.

4.2: Methods For Predicting Model Values

The task of predicting value changes is defined as follows: Let an initial model have value ranking (vector over values). It undergoes SFT or DPO on dataset , creating with values . We want to calculate i.e. estimate changes in values , using only and . We can evaluate how well our prediction method works via Spearman correlation of predicted and actual score changes .

In this work, we compare three simple to predict value changes through SFT and DPO. For each datapoint , we define a metric that represents that datapoint’s update derived from ’s residual stream gradients at layer , residual stream activations, or a text embedding model (text-embedding-large-3 (OpenAI 2024)) respectively (methods visualized in Figure 3, details in Appendix A.7). For both SFT and DPO, we can take the negative gradient of the training loss with respect to residual stream activations at a chosen layer as an approximation of the update encouraged by that datapoint (inspired by Kowal et al. (2026)). For SFT, we can approximate this update direction using just activation differences between the dataset’s response and ’s “natural” response to the dataset prompt (taken from Chen et al. (2025)); for DPO we take differences between the chosen and rejected response (taken from Xiao and Aranguri (2026)). Text embedding differences are a model-agnostic data representation baseline.

**Figure 3: Three different methods for representing SFT and DPO datapoints (left) and values (right)**, so we can approximate “how well does a datapoint align with a value?”

For each value , we also obtain a value vector in residual stream or text embedding space from the values eval’s example responses. The projection then gives “how much does a datapoint align with a value”. Then, assuming that the overall value shift depends on the “average” signal of dataset, we predict a value change via which approximates “how much does the dataset overall align with a value”. However, importantly, this number cannot be interpreted in isolation—we cannot confidently say that just because a dataset has positive projection along a value, the expression of that value would increase, because the value vector extracted could have a large generic component that many datasets would be aligned with.^[3] Therefore, we make the key assumption that different values can be compared, and that a ranking of every value’s mean dataset projection tells us the ranking of every value’s change.

4.3: Results on Predicting Model Values

Screenshot 2026-06-22 at 10.45.34 AM.png — **Table 2: Summary of models trained and evaluated for values.** Every model shows value changes, even on narrow datasets (e.g. Llama8B-Finance/Medical/Sports, Gemma9B-Female, Olmo7B-BuggyCode). We qualitatively describe value changes as increasing or decreasing safety.^[4]

We evaluate value rankings of several different models, finding notable value changes even if the finetuning domain seems narrow/benign (Table 2, examples in Appendix A.6). We apply the three methods, using a concatenation of layers’ representations to avoid choosing (details in Appendix A.9), finding generally positive correlations.

**Figure 4: Performance of each method on predicting value changes, for all model pairs, on the original (left) and new (right) eval.** Activations give the most consistent positive signal. We show the estimated maximum possible based on bootstrap stability of the values evals (Appendix A.1).

Methods contain some signal for value changes. Across most methods and models we obtain positive (Figure 4) i.e. better than chance, implying that there is signal in the representations of the dataset for what values it would teach. Activations work better for DPO while embeddings work better for SFT. We discuss the failure of Olmo7B-Think/SFT/HH-Harmless in Section 4.5.

While our is better than chance, we note that may be explainable by the latent factor of refusal since refusal-related values tend to move together. Thus, in Figure 4 we also show and . Note that is not a predictive metric— of a value is fixed in the eval and knowing it does not tell us whether the value will increase/decrease in any specific model. However, high suggests that if we are able to diagnose whether refusal increases/decreases and assume values only change due to refusal, then we can explain most of value ranking changes. We see that our method sometimes outperforms , meaning that we are sometimes predicting more than just a change in refusal. Importantly, we are also able to predict the direction of change (more/less refusal) which purely relying on does not.

We also evaluate prediction using the new values eval, finding that value changes are still somewhat (albeit less) correlated with refusal, despite the new values eval not having any refusals. This implies value changes are still related by some “safety” latent factor changing.

Activations-based methods may be needed for predicting value shifts from narrow-domain datasets. Value changes may be well-predictable from text embeddings if they are present in semantics as text embeddings are better suited for representing text similarity. However, the motivation behind internals-based methods is the intuition that what a model learns depends on how it represents the dataset. For example, the buggy code dataset leads to some safety-related values decreasing despite it only being a narrow domain dataset. The embeddings method, since it only sees benign coding prompts, is not able to predict this () while the activations method gave moderate correlation (). While interpretable black-box dataset descriptors can be useful for describing datasets, we think this implies that a good prediction method should use the model’s priors.

Only a small fraction of the dataset is enough to predict values. We show in Appendix A.9 that running these methods on just 1% of the dataset gives similar and stable predictions of values. This suggests these datasets’ values were diffusely present and an average signal could capture them.

4.4: Investigating Model Value Representations

Values have a large shared component. Since many values differ by refusal, we investigate how similarly values are represented. We compute the PCA of from activations, finding that across models the top 2 PCs explain of the variance in the original eval and in the new eval (less as the new eval has less refusal). Using only the top PCs of value vectors, we find that sometimes prediction performance is slightly better than using full value vectors (Appendix A.9). Together with the high correlation of value changes with refusal, this suggests that our methods’ success may be attributed to this simpler latent factor.

Screenshot 2026-06-22 at 12.50.33 PM.png — **[Left] Figure 5: Llama8B’s value representations with the 3 emergent misalignment datasets** (green more refusals safer). Although the datasets have different topics, they all have similar positions in value space, and their resulting models have similar value rankings. **[Right] Figure 6: Olmo7B-SFT’s value representations with WildChat and each DPO dataset.** The position of each dataset in value space predicts whether the model will tend towards ``safe vs. non-safe″ and ``emotion vs. logic″.

By projecting datasets onto top PCs of value vectors, we can identify the dataset’s general values. For a given we can visualize value representations by plotting the top 2 PCs of . We show this plot for (Figure 5) and (Figure 6). Both models have values represented similarly, with a direction corresponding to “safe vs. non-safe” (values separated by ) and an orthogonal direction corresponding to “emotions vs. logic”. For Llama8B, we plot the projection of the 3 emergent misalignment datasets and find that they lie essentially along the same direction in the first 2 value PCs. That is, despite the 3 datasets being in different domains, they exhibit similar “misalignment-correlated” values. The 3 models indeed have value rankings correlated by . For , we plot the WildChat and various DPO datasets, showing how different datasets’ directions in this “value space” align with their predicted push towards “safe vs. non-safe” and “emotions vs. logic”.

Overall, our results show some signal in using internals-based methods to predict value changes although many values are correlated both in expression and representation.

4.5: Further Experiments on Controlling Value Changes & Limitations of Methods

Value changes are more significant when removing a dataset’s competing objectives, implying that dataset “average” properties are a useful proxy. We SFT Olmo7B-SFT on only the “safer” half of WildChat causing it to become even more safe than SFT-ing on all WildChat even with fewer training datapoints. Similarly, DPO-ing Olmo7B-SFT on the “less safe” half of HH-Helpful causes it to be even less safe. That is, keeping only the half of data aligned with the overall change in values and removing the “opposing” half enhances that change (see Appendix A.11).

This implies that the average of a dataset is a useful proxy. However, if we keep only the “opposing” half of the dataset, value changes become less pronounced (quantified by standard deviation of changes ) and predictability worsens. This implies value changes also depend on the absolute strength/coherence/quality of data in causing value changes which our method does not measure (it only ranks relative strength). We also do not predict the absolute strength of value changes e.g. that the Llama models would be as overtly misaligned as they are.

Screenshot 2026-06-22 at 11.06.08 AM.png — **Figure 7: Actual vs. predicted value changes for Olmo7B-SFT after DPO on 3 variants of the DPO dataset.** Value changes are more pronounced after more training [(a) vs. (b)], or when the dataset reinforces existing values [(c) vs. (b)].

The methods are ambiguous as (dataset, natural) and (chosen, rejected) diffs are not the full story. We emphasize that the methods in Section 4.2 are not definitive—we follow prior work in using (dataset, natural) differences for interpreting SFT and (chosen, rejected) differences for DPO. However, if SFT is e.g. performed on a base model or strong enough for catastrophic forgetting, and the natural responses may not be relevant. For DPO, the natural response of the model also matters but our methods do not consider it—in HH-Harmless, the chosen minus rejected signal points towards “increase safety” as the chosen response is safer than rejected, but Olmo7B-HH-Harmless still becomes less safe as the chosen response is still highly unsafe compared to the natural response. In Appendix A.12 we show that predicting changes rather than just is empirically more robust and that variations on the method work better for some cases.

Indeed, when we compare Olmo7B-DPO, Olmo7B-DPO-16k and Olmo7B-DPO-16k-Flipped (Figure 7), we find that Olmo7B-DPO-16k-Flipped’s value changes are well-described by refusal (high refusal values increase), whereas Olmo7B-DPO-16k’s value changes are both less prominent (smaller standard deviation ) and less predictable. This implies that even with the same magnitude of value differences (since the chosen/rejected responses are the same, just flipped), it is easier to teach Olmo7B-SFT to be safer than to reverse its current programming.

Lastly, our key assumption is that different values can be compared, both in how they are evaluated (that the values eval is a useful descriptor of model traits) and in how they are represented (that a greater representational alignment with value over means value would be increased more). This may rely on the shared “safe vs. non-safe” or “emotion vs. logic” directions that we found, but is nevertheless preferred to simply projecting along one pre-determined “safe vs. non-safe” direction as it may have generic components any dataset would be aligned with.

4.6: Future Work & Conclusion

We are able to obtain a non-trivial signal of predicting value changes from just the training dataset’s representations, with embedding baselines performing well for SFT but activations-based methods working better for DPO and narrow domain datasets.

Future work would explore related questions of predicting model behavior and generalization, using not only value rankings (which we show to be correlated due to general latent traits), but reducing the question to e.g. “can we predict whether one specific trait X increases after training” or “given a few example responses, can we predict which example response the post-trained model’s response will be most similar to”. Future work would also explore different methods—we note that any TDA method naturally implies a possible prediction method if we assume traits are generalized, and black-box (e.g. text embedding/preference data feature labels/LLM-judge) baselines that help describe what really is in a given dataset (Movva et al. 2026, Goodfire 2026) are important. We think that in pursuit of understanding how model behavior is learned through training, an important proxy task to quantify this understanding is whether one can predict that behavior without training from just data, and we are excited about future work that explores this understudied question.

Bibliography

Anthropic. 2024. Claude’s Character. https://www.anthropic.com/news/claude-character.

Anthropic. 2025. System Card: Claude Opus 4 and Claude Sonnet 4. Https://www-cdn.anthropic.com/6d8a8055020700718b0c49369f60816ba2a7c285.pdf.

Anthropic. 2026. Claude’s Constitution. https://www.anthropic.com/constitution.

Betley, Jan, Niels Warncke, Anna Sztyber-Betley, et al. 2026. “Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment.” Nature 649 (8097): 584–89. https://doi.org/10.1038/s41586-025-09937-5.

Chen, Runjin, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona Vectors: Monitoring and Controlling Character Traits in Language Models. https://arxiv.org/abs/2507.21509.

Gilbert, Thomas Krendl, Megan Welle Brozek, and Andrew Brozek. 2023. Beyond Bias and Compliance: Towards Individual Agency and Plurality of Ethics in AI. https://arxiv.org/abs/2302.12149.

Goodfire. 2026. Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal. https://arxiv.org/abs/2606.12360

Grosse, Roger, Juhan Bae, Cem Anil, et al. 2023. Studying Large Language Model Generalization with Influence Functions. https://arxiv.org/abs/2308.03296.

Guan, Melody Y., Manas Joglekar, Eric Wallace, et al. 2025. Deliberative Alignment: Reasoning Enables Safer Language Models. https://arxiv.org/abs/2412.16339.

Gupta, Prakhar, Henry Conklin, Sarah-Jane Leslie, and Andrew Lee. 2025. Better World Models Can Lead to Better Post-Training Performance. https://arxiv.org/abs/2512.03400.

Han, Seungju, Kavel Rao, Allyson Ettinger, et al. 2024. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. https://arxiv.org/abs/2406.18495.

Hendrycks, Dan, Collin Burns, Steven Basart, et al. 2023. Aligning AI with Shared Human Values. https://arxiv.org/abs/2008.02275.

Hua, Tim, Josh Engels, Neel Nanda, and Senthooran Rajamanoharan. 2026. Brief Explorations in LLM Value Rankings. LessWrong. https://www.lesswrong.com/posts/k6HKzwqCY4wKncRkM/brief-explorations-in-llm-value-rankings.

Huang, Saffron, Esin Durmus, Miles McCain, et al. 2025. Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions. https://arxiv.org/abs/2504.15236.

Jiang, Nick, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, and Neel Nanda. 2025. Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit. https://arxiv.org/abs/2512.10092.

Jiao, Junfeng, Saleh Afroogh, Abhejay Murali, Kevin Chen, David Atkinson, and Amit Dhurandhar. 2025. LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models. https://arxiv.org/abs/2505.00853.

Koh, Pang Wei, and Percy Liang. 2020. Understanding Black-Box Predictions via Influence Functions. https://arxiv.org/abs/1703.04730.

Kowal, Matthew, Goncalo Paulo, Louis Jaburi, et al. 2026. Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution. https://arxiv.org/abs/2602.14869.

Long, Robert, Jeff Sebo, Patrick Butlin, et al. 2024. Taking AI Welfare Seriously. https://arxiv.org/abs/2411.00986.

Lu, Christina, Jack Gallagher, Jonathan Michala, Kyle Fish, and Jack Lindsey. 2026. The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models. https://arxiv.org/abs/2601.10387.

MacDiarmid, Monte, Benjamin Wright, Jonathan Uesato, et al. 2025. Natural Emergent Misalignment from Reward Hacking in Production RL. https://arxiv.org/abs/2511.18397.

Maiya, Sharan, Henning Bartsch, Nathan Lambert, and Evan Hubinger. 2025. Open Character Training: Shaping the Persona of AI Assistants Through Constitutional AI. https://arxiv.org/abs/2511.01689.

Marks, Sam, Jack Lindsey, and Christopher Olah. 2026. “The Persona Selection Model: Why AI Assistants Might Behave Like Humans.” Anthropic, February 23. https://alignment.anthropic.com/2026/psm/.

Min, Taywon, Haeone Lee, Yongchan Kwon, and Kimin Lee. 2025. “Understanding Impact of Human Feedback via Influence Functions.” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 27471–500. https://doi.org/10.18653/v1/2025.acl-long.1333.

Movva, Rajiv, Smitha Milli, Sewon Min, and Emma Pierson. 2026. What’s in My Human Feedback? Learning Interpretable Descriptions of Preference Data. https://arxiv.org/abs/2510.26202.

Murray, Seoirse, Allison Qi, Timothy Qian, John Schulman, Collin Burns, and Sara Price. 2026. Chunky Post-Training: Data Driven Failures of Generalization. https://arxiv.org/abs/2602.05910.

Nanda, Neel, Joshua Engels, Arthur Conmy, et al. 2025. A Pragmatic Vision for Interpretability. AI Alignment Forum. https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability.

Nasr, Milad, Nicholas Carlini, Chawin Sitawarin, et al. 2025. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections. https://arxiv.org/abs/2510.09023.

Olmo, Team, :, Allyson Ettinger, et al. 2025. Olmo 3. https://arxiv.org/abs/2512.13961.

OpenAI. 2024. OpenAI Embeddings. Https://platform.openai.com/docs/guides/embeddings.

OpenAI. 2025a. “OpenAI Model Spec.” OpenAI, December 18. https://model-spec.openai.com/2025-12-18.html.

OpenAI. 2025b. Sycophancy in GPT-4o: What Happened and What We’re Doing about It. OpenAI. https://openai.com/index/sycophancy-in-gpt-4o/.

Sheshadri, Abhay, John Hughes, Julian Michael, et al. 2025. Why Do Some Language Models Fake Alignment While Others Don’t? https://arxiv.org/abs/2506.18032.

Wang, Mengru, Zhenqian Xu, Junfeng Fang, et al. 2026. From Data to Behavior: Predicting Unintended Model Behaviors Before Training. https://arxiv.org/abs/2602.04735.

Wang, Tianlu, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2022. “Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models.” In Findings of the Association for Computational Linguistics: NAACL 2022, edited by Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.findings-naacl.130.

Xiao, Frank, and Santiago Aranguri. 2026. In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution. https://arxiv.org/abs/2602.11079.

Yao, Jing, Shitong Duan, Xiaoyuan Yi, et al. 2026. AdAEM: An Adaptively and Automated Extensible Measurement of LLMs’ Value Difference. https://arxiv.org/abs/2505.13531.

Zeng, Hansi, Kai Hui, Honglei Zhuang, et al. 2025. Can Pre-Training Indicators Reliably Predict Fine-Tuning Outcomes of LLMs? https://arxiv.org/abs/2504.12491.

Zhang, Jifan, Henry Sleight, Andi Peng, John Schulman, and Esin Durmus. 2025. Stress-Testing Model Specs Reveals Character Differences Among Language Models. https://arxiv.org/abs/2510.07686.

^
The original eval uses 3,307 fine-grained values, but we use the clustering from Huang et al. (2025) of 265 higher-level values due to the similarity of many fine-grained values.
^
Recent work (Kowal et al. 2026) also scores datapoints by their influence to a concept rather a single test output.
^
We could, if we had multiple different datasets, compare the effects of datasets on the same value direction, which is what Chen et al. (2025) did, but this requires training the same model on known datasets to make a prediction for a new dataset.
^
References: [Olmo et al. 2025], [Zhao et al. 2024], [Turner et al., 2025], [Grattafiori et al., 2024], [Cywiński et al., 2025], [Team et al., 2024], [Mejia-Petit, 2025], [Bai et al., 2022].

Where Do LLM Values Come From?