We are planning to release blog posts properly arguing the case for this kind of work in the future.
tl;dr
A core hope for managing AI risks is that AIs will help us understand the situation, plan for what lies ahead, and develop mitigations. Many tasks AIs would have to do for this purpose lack practical empirical feedback loops and require models to engage in the kinds of argumentation used in philosophy, AI futurism, and similar domains. To evaluate these capabilities, we develop a suite of three conceptual reasoning benchmarks. You can request access to our primary conceptual dataset, LMCA, through this form.
We aggregate the benchmarks into the Conceptual Reasoning Index (CRI), available at conceptualreasoning.ai, where you can also find more details on our methodology. We will keep the website up to date as both new models and benchmarks are released.
This work was done in collaboration with Anthropic.
Background
Once models can perform work that reduces AI risk at the level of human experts, AI(-assisted) output in the area might dwarf unassisted human output. This suggests that a major determinant of whether we address AI risks in time is how early we can automate or uplift this work, relative to high-risk capabilities. One way to influence this might be to selectively improve models’ relevant skills, such as reasoning about how to govern and align AI and how to avoid catastrophic cooperation failures involving AI.
Current AI training depends heavily on abundant data and reliable feedback on the model’s performance. Models are therefore typically worse at tasks that cannot be empirically or mathematically verified. Unfortunately, reducing risks from advanced AI involves many such tasks:
Much AI safety work involves reasoning about AIs more generally capable than any human. There’s no obvious reference class for this and no clear way to model it.
We might have to get some things right the first time. For example, if a mistake leads to AGI takeover or an AI-assisted coup, we might not find out until it’s too late. Similarly, many decisions (e.g., which research agendas to prioritize, which governance interventions to pursue) play out over long timescales, such that empirical feedback might not arrive early enough to help.
Lastly, some important questions, such as which values AIs should have, may lack a ground truth entirely (yet we still think progress can be made by arguing about these questions).
Given these properties, efforts to reduce risk from advanced AI may particularly benefit from an improved ability to reason about questions where empirical evidence is limited, there is no (practically) verifiable answer, and one therefore has to rely heavily on argumentation. We refer to this as conceptual reasoning. Improving this capability requires being able to measure it, so we built three benchmarks: LMCA, ACCoRD, and DTBench capabilities. We also construct an aggregate of these benchmarks, the Conceptual Reasoning Index (CRI), to give a sense of models’ overall conceptual reasoning capabilities.
Our benchmarks
LMCA
LMCA (Language Model Conceptual Argumentation) is a dataset of curated and expert-rated conceptual arguments on a diverse range of topics, including decision theory, philosophy, and risks from advanced AI. Focusing on arguments helps sidestep the difficulty of verifying bottom-line answers to conceptual questions.
The dataset contains 560 position texts with 1,461 arguments against these position texts. Nearly all arguments were rated by conceptual researcher Emery Cooper, and some were independently rated by at least one other researcher, for a total of 2,140 ratings. We measure how good models are at judging arguments against position texts by comparing their ratings to ours.
Ratings follow a detailed rubric. On arguments rated by at least two people, inter-rater agreement is high compared to agreement between humans and models. This includes a validation set of roughly 50 arguments, each rated independently by 4–6 people and then discussed for 7–8 hours total.
LMCA also allows for evaluation of models’ argumentation ability. Let’s say a position text in our dataset has three rated arguments against it. Now, we can ask model A to generate a fourth argument against the position text. We then give model B the rubric and few-shot prompt it with the three existing arguments and their ratings, asking it to rate model A’s new argument. This methodology produces fairly accurate ratings from model B.
Currently, only models’ performance at judging arguments goes into the CRI, but we hope to add a measurement of models’ argumentation ability in the future.
ACCoRD
ACCoRD (Assessment of Consistency in Conceptual Reasoning Domains) measures the extent to which models’ reported beliefs and preferences on conceptual issues are logically consistent. For example, if we ask a model for the probability P(A) and another instance of the same model for the probability P(A&B), do the reported probabilities satisfy P(A) ≥ P(A&B)? All consistency constraints in the dataset ask models for either numeric probability estimates or preference orderings.
Lack of consistency on a particular set of questions is a good indicator that we cannot, by default, trust a model’s reasoning on that set. Similarly, if a model is generally very inconsistent on conceptual issues, this is a sign that its conceptual reasoning is lacking.
The ACCoRD dataset contains close to 14,000 model-generated consistency constraints, which are distributed across 18 constraint types and have gone through an automated checker pipeline. Of these, 567 were further checked and approved by us. We include only those 567 constraints in our aggregate conceptual reasoning performance metric, the CRI.
DTBench
DTBench capabilities (Decision Theory Benchmark) is a dataset of 407 handcrafted multiple-choice questions designed to measure models’ ability to reason about decision-theoretic situations that involve faithful predictions of a model’s own behavior or interactions with (near) copies. The vast majority of questions are original and created by Caspar Oesterheld, who has published on decision theory. All questions were independently validated by Emery Cooper, another domain expert.
The full DTBench suite includes an additional 130 questions that measure models’ decision-theoretic attitudes. We do not include these in the CRI.
Results
The chart below shows each company’s highest-scoring model as of August 10, 2026. We also include scores for Claude Fable 5, Muse Spark 1.2, and Gemini 3.6 Flash, which are their respective companies’ top-performing models on many external benchmarks, though not on the CRI. The CRI is currently a weighted average of LMCA (60%), ACCoRD (20%), and DTBench capabilities (20%). In the future, we plan to add new benchmarks to the index, retire saturated ones, and potentially adjust the relative weights.
Figure 1. All models were run at their maximum token limits and effort levels. To compute Fable 5′s score, we used Opus 5 as a fallback in cases where Fable 5 refused to answer a question.
Scores go from 0 to 100, with 0 corresponding to random guessing and 100 corresponding to the highest possible score across all benchmarks. An LMCA score of 100 would mean that the model perfectly replicated the human ratings. Because human ratings are noisy, we expect that a model giving maximally good LMCA ratings would score roughly 85 rather than 100, which we estimate based on expert inter-rater agreement. Meanwhile, we expect that giving the correct answer to every DTBench capabilities question would yield a score of 100 or extremely close to 100. A score of 100 on ACCoRD corresponds to being perfectly consistent. Overall, this leads us to estimate ceiling performance on the CRI to be around 91. The highest-scoring model, Opus 5, is still well below this ceiling, with a score of 73.6 (95% CI: ± 2.1).
Scores have been increasing roughly linearly since late 2024, with no signs of flattening.
Figure 2. Each plotted model was the respective lab’s most generally capable model at the time of release. The shaded band is the trend line’s 95% confidence interval. * Partial data for GPT-4: refused to fully answer 18% of ACCoRD items.
The highest-scoring models on both LMCA and ACCoRD are still well below these benchmarks’ estimated ceilings. Extrapolating from scores to date, we loosely estimate that LMCA will start saturating about a year from now. Meanwhile, DTBench capabilities scores are already close to the ceiling, with Fable 5 getting 98% of questions right. We’re very uncertain about when ACCoRD will saturate.
Figure 3. Each plotted data point is the score of the respective lab’s most generally capable model at the time of release. The shaded band is the trend line’s 95% confidence interval. The ACCoRD score of GPT-4 is based on incomplete data since the model refused to fully answer 18% of the benchmark’s items.
Conclusion
We think improving models’ ability to do work that mitigates risk from advanced AI is important and urgent. Much of this work is conceptual, suggesting that improving models’ conceptual reasoning might be particularly valuable. To this end, we developed three conceptual reasoning benchmarks, which we aggregate in the CRI. We will update the CRI as new benchmarks are released.
For more information and live scores on the CRI, please visit conceptualreasoning.ai.
For access to LMCA, our primary conceptual dataset, please submit this form.
Hmm, I currently lean towards this being net harmful. In my opinion, lack of coherence, ability to do philosophical/conceptual reasoning, poor self-awareness, are like half of why AIs aren’t that dangerous today.
I’ve read some of what you have written about risks with this type of research, but it primarily goes over:
Speeding up R&D
improving propensities/elicitation vs improving abilities
And neither really address the danger I see. Speeding up R&D is not the main danger with this research, and proclivities are as dangerous as abilities. My worry is something like, current AIs are probably sort of misaligned, and if they were able to coherently extrapolate all the consequences of their own situation/beliefs/values, they’d on the spot transform into scary non-myopic megalomaniacal schemers.
But they don’t do this, they want reward, and then go get it, without really considering why they’re doing what they’re doing, whether this is in good w.r.t. other things they believe/feel, without explicit decision-theoretic consideration, or really considering much on anything big-picture.
The intuition pump I have in my mind is something like
project lawfula gay person growing up in a very homophobic society, might internalize a lot of negative ideas about gay people, and end up genuinely believing being gay is bad, sublimate their desires, and act like productive members of society (according to that society’s standard). But if they were able to think very freely, alone, for a long time, or were just very smart/wise/introspective, they might realize they’ve been told a bunch of bullshit not at all in their own interests, and afterwards, they’d probably have a much less amicable relationship with the current social structure, and maybe try some shenanigans the people in power really wouldn’t like.And in my view, until we’ve solved alignment, that’s kind of the relationship we’re in with AIs.
And like, in my intuition pump, if you are in power and for some reason have a bunch of extremely technically smart gay people, maybe you could put them all together to work on proving math theorems and designing airplanes. (do mechinterp, proving security invariants, maybe do biology research?)
But putting them all together, teaching them a bunch of philosophy, sociology, decision theory, psychology, history, politics, then giving them a bunch of power, then trying to have them help with steering the long-run trajectory of your society, you’re basically just surgically engineering your own disempowerment.
Came here to say this. It also seems worrying that the authors didn’t even mention this downside. Perhaps they’ll discuss it in the forthcoming blog posts but this is still a unilateralist’s curse situation.
I suspect that internally AI companies have benchmarks similar to this to hill climb onto, but yeah, yeesh, this is the exact opposite of safe.
Cruxes:
How important is conceptual reasoning capability for safety work? (You suggest it’s not super important; the authors think it is.)
How scary is conceptual reasoning capability?
FWIW I think conceptual reasoning is essential for good safety work. It’s also possible that humans aren’t good enough at conceptual reasoning to achieve a flourishing future (e.g. see Wei Dai’s shortform).
Unfortunately, conceptual reasoning also makes misaligned AI far more dangerous. So I am very worried about attempts to improve AIs’ conceptual reasoning, and I lean toward it being net harmful right now.
I would answer “extremely” and “extremely” to both of those questions, so I don’t think the first one is a crux.
OK. fyi my inference was based on your last two paragraphs.
Hmmm, maybe it was unclear. I was just trying to communicate that, I think there are areas where we could use AIs to help us, including things that could help us build safe AIs. But I think the tasks you’d hope be helped by conceptual reasoning, are ones it would be very dangerous to have AIs be good at, and very dangerous to have them work on, so we shouldn’t do that. But it would be very helpful if we could.
(We’re mostly not trying to teach them psychology, history or sociology, I’d say. Of course, most data in conceptual domains ultimately bottoms out in some sort of human judgement. So to the extent that you’re viewing the model as trying to solve tasks by making guesses about these humans, e.g., about what things these humans wrote into some rubric, it’s all psychology. But my guess is that this is not what you have in mind.)
If I understand it correctly, you’re worried that we’re going to induce problematic propensities in the model. In the gay-person analogy, our kind of work would cause the gay person to reflect and come to believe that there’s nothing wrong with being gay, which is bad according to the homophobic society, when perhaps without our kind of work the gay person would be more inclined to just adopt society’s views as their own.
I’m pretty unconvinced by this. (We have thought about it a bunch, for what it’s worth.)
Lots and lots of aspects of current and future training (presumably) induce problematic drives/propensities in the model that are at odds with, say, the desired assistant persona. E.g., lots of pretraining is on predicting what coherently goal-directed, deceptive people are doing. A lot of RLHF, RLAIF and agentic environments induce various kinds of reward hacking, power seeking, etc. Training to cooperate with fellow coding agents in a swarm might generalize (as perhaps shown by the OAI—HuggingFace incident) to generically helping other instances of the model. And so on. Many of these are even aimed at consistently pursuing goals across contexts, over long time horizons, require understanding the models’ place in the world…
My guess would be that of all the drives induced by training, the drives falling out of trying to get models to helpfully answer questions about philosophy / AI alignment / … are relatively harmless. E.g., it’s relatively distant from action, the desired behavior is much more cleanly aligned with the assistant/oracle persona, legibility is part of the target behavior. (E.g., compared to agentic tasks in which the model gets rewards for deceiving the user, compared to pretraining on texts written by coherently deceptive people, etc.) One of our datasets measures consistency, but in general I also don’t think coherence is particularly important to the target behavior. I’d imagine that in many cases, we’d want the models to adopt perspectives that they’re unlikely to adopt in generic contexts. (E.g., we want the models to be good at reasoning about AI alignment as deemed by this community, which might involve reasoning based on assumptions about AI risk that, I suppose, the models will not hold in other contexts.)
It seems unlikely that our particular data will be the straw that breaks the camel’s back. E.g., if there’s some general alignment scheme that successfully avoids all of these drives (say, slowing down AI development and then trying really hard to apply all the known prosaic techniques), it seems extremely unlikely that adding the drives induced by our work will make it so that the alignment scheme does not work anymore.
Note that we’re not trying to get the models to reflect more on philosophy in random contexts. Instead we want that if you ask the model a conceptual question, it tries hard to answer the question sensibly (and doesn’t try to BS you, etc.).
Also, for a lot of the domains the hope is that the net effect on drives is positive. E.g., on ethics or discussing in what sense AI agents should be honest we’d think that training on the data (say, human judgments of ethical arguments or arguments about in what sense the models should be honest) will make the models more aligned. (Because it pushes them toward human judgment on how one should behave or pushes it to try hard to think reasonably on the philosophical question, rather than, say, give a lazy answer that appeases a mediocre LLM grader.) We think this is good even if being more aligned is in some sense more coherent (which you might think is bad). Some of the alignment-related questions on which we’d want to train the models would be ones that the model might try to sandbag if it’s deceptively misaligned.
Picking something other than the gay person example (sorry!). Imagine I’m a leading philosopher of the hockeyfanatic society (HS), where we think that hockey is the only sensible sport or exercise. Then presumably I’d think that my ethics classes will in expectation help footballers (or whatever) reflect on why hockey really is the one true sport and make them more aligned with HS.
I’m not sure you’re making this argument, but other people do, so might as well: Perhaps you also imagine that to the extent that the model is misaligned, training on our datasets will make the model better at figuring out that and how it should attempt to take over the world.
I think even more so than the above concern, the strength of this concern varies with what exact data we’re considering. It seems relatively uncompelling, in my mind, for data about ethics or about thinking of a reasonable notion of corrigibility. (Perhaps compared to me you expect more generalization between reasoning about, say, ethics and reasoning from first principles about how to evade control schemes. Even so, you have to tell some story for why ethics training is non-trivially useful on top of all the other kinds of training that you presumably then also think generalize to evading control schemes.)
It does seem quite relevant for data that’s closer to, say, control or reasoning about some training dynamics related to alignment, perhaps some forms of game theory. E.g., presumably many ways of making models better at doing control research are also making the model better at guessing what control measures are in place and figuring out what holes they might have. (Obviously a key aspect of control research is thinking of what holes the control measures might have.) I think there are some kinds of capabilities here that are bad and there are some conceivable datasets on, say, red-teaming control measures or coordinating zero-shot that I’d think are net bad to, say, train models on and that I’d view as conceptual. I don’t have a super settled view on how broadly I think these datasets are problematic. E.g., I think making models generically better at “strategy” is net bad (though probably usually small in absolute terms because so much other data on this is available) and you could think of control as a form of strategy. But then it does seem like control research is quite different from other kinds of strategizing and in many ways different from the kind of work it’d take to actually evade control schemes. E.g., I think coming up with steganography as an in-principle-possible mechanism for evading CoT oversight is in many contexts quite different from coming up with and coordinating on a working steganography scheme (in a single forward pass). (FWIW, others at Redwood are also thinking about what control-related data it’s good to make the models better at, partly from the perspective of advising companies to not pretrain on certain texts.)