Only circumstantial, and I think it’s possible that Mythos is the same size as Mythos Preview.
Price drop
Maybe Anthropic took 3.5 months to release Mythos because they wanted to distill a cheaper version due to compute limitations
I had previously considered the price drop the strongest evidence, but that doesn’t distinguish between the two hypotheses. I personally consider it more likely than not due to Anthropic’s compute limitations but that’s speculative.
It’s not well-specified and it does leave the door open for misuse, but I don’t think it’s misleading or disingenuous. I’m guessing the terms will end up being reasonable with heavy subsidization or free tokens.
For Anthropic’s Mythos Preview access it’s likely (IMO) that Mythos is a smaller distilled version of Mythos Preview that actually is cheaper to run.
That’s one of several directions I’m considering but I haven’t decided for sure yet. I agree it’s a pretty natural next step and it would be more useful than asking post-hoc questions.
It could be quite a bit more expensive, because you might have to generate the labels on the fly for each training run, as the behavior would change during training. Post hoc is nice and simple because you can do multiple training runs on the same fixed dataset. But there are probably tricks you can do so it’s cheaper.
Marcus Williams said they weren’t monitoring evals. His guess was people had assumed sandboxed evals were safer.
https://x.com/Marcus_J_W/status/2095469924793606617
I think OpenAI tests had found Codex monitors would have flagged the hacking.
After thinking about this more I think there are a couple other tradeoffs in favor of the counterfactual simulatability metric:
For most behaviors in the wild, I’m guessing that counterfactual prompts will be the cheapest way to explain the behavior. If we restrict interpretability evaluations to cases with an expensive ground truth (e.g. jailbreak discovery, rare event estimation), then we would only be evaluating our tools in relatively niche and unusual settings. By using the counterfactual simulatability metric, we can evaluate explanations derived from our tools in a wide range of settings (even if our tools aren’t the cheapest method), which can provide greater confidence in how much our tools will generalize to settings without ground truth (e.g. unverbalized eval awareness).
Additionally, CHIVE-generated behaviors may have a signal-to-noise advantage over e.g. rare event estimation. This signal advantage can both be in “is the signal available in the forward pass” and “the evaluation has lower variance than discovery of rare behaviors”. Given that current interp tools appear to provide a relatively weak signal, it may be best to do initial iteration in a setting with more signal to make it easier to find initial progress.
Yeah, I agree this metric isn’t ideal from the perspective of practical usefulness. I actually spent a month experimenting with more practical metrics involving open-ended discovery, such as proposing the most important counterfactual from the entire investigation. I also experimented with version of jailbreak discovery where a model rewrites failed jailbreaks based off interp tool feedback.
I found that these metrics are extremely messy with many confounders, and results could change significantly with minor design decisions, which is why I ended up sticking with the counterfactual simulatibility metric. Example confounders with jailbreak discovery: “How capable is the jailbreak writer?”, “Is the jailbreak writer actually trying and not writing them as non-harmful?”, “How large of a rewrite is the jailbreak writer allowed to do?”.
That said, I think this less practical metric is more defensible in the current regime, where no tool provides any uplift. Right now I’m more interested in the question of “Is there useful information in the tool outputs, beyond what you can infer from the transcript?”, which is different than checking if interp is a cost-effective method to obtain this information. If we start seeing a consistent uplift, then I think moving towards more practical metrics becomes much more important.
The counterfactual simulatability metric also isn’t unique to my paper, as it’s widely used in related areas like CoT faithfulness and self-explanation. The “hint evaluation”, which is commonly used in CoT faithfulness and interp, is also just a narrow counterfactual simulatibility eval. Intuitively I think it’s reasonable that a tool which provides a good explanation of behavior should help you make some predictions in related situations.
But overall I agree that a more practical evaluation would be better, however I was unable to find something I liked during the course of the Fellows program.
Have you checked other models? I find Claude models have very detailed knowledge of many random things. For example, Opus can recall niche appendix details, such as the values of non-standard hyperparameters, from many of my papers.
I think it’s more common to miss the right hypothesis. The most common issue is that tool outputs almost never contain causal relationships, like “feature X causes behavior Y”.
This question about “is there enough cognitive processing to pick up on?” is pretty common. I think there are two underlying questions here:
Is there little cognitive processing in our examples?
Do we care about behaviors with little cognitive processing?
For 1, I think it’s hard to say. I created an LLM-judge based “coherence filter” that filtered out around 60% of the initial examples to obtain datapoints that I intuitively thought had a coherent enough mechanism that may require cognitive processing. I hoped that we could find some uplift with these filtered examples. You can view randomly sampled investigations here to get a sense for what survives the filter.
People’s intuitions for what requires processing vary. With all four of the below examples (from Figure 2 in our paper), I think it’s plausible that the model may or may not be using cognitive processing when performing the behaviors, and we don’t have a ground truth that can inform us. For example, maybe there’s a correlation-driven behavior to take reckless and unsafe actions when imitating a bold persona.
For 2, I think we still care about behaviors that require little cognitive processing, as e.g. models may take harmful actions when they are unaware of the underlying causes. While this isn’t as concerning as something like deliberate scheming, I think we would like to understand it. I would also be much happier with a tool that can explain the behaviors I selected versus a tool that cannot, and I think we would want to avoid only evaluating what tools currently succeed at. I agree that we don’t care about all behaviors, and some such as the “picking from a list” behavior may not be that interesting.
I’ll also note that our result may not be that surprising based on prior results. While we see e.g. many NLA verbalizations of eval awareness, it’s (AFAIK) very rare to see NLA verbalizations that involve causality like “I’m behaving safely only because I’m in an eval”. This is similar to our finding that tool outputs almost never mention causal relationships.
I agree that neither refusals or classifiers are much of a barrier to a sufficiently motivated actor, which matters the most for catastrophic misuse. However a classifier is far easier to remove.
Even if refusals don’t prevent catastrophic misuse, I would guess that model refusals would prevent the majority of potential harm. This is similar to locking my front door: in theory this shouldn’t make a difference, because a motivated criminal can go through the window. Yet in practice it substantially reduces crime.
Most users that want to do harm just aren’t that motivated. For example most people that attempt mass casualty events are incompetent and don’t put in much effort.
However there are various arguments against this:
If all open source inference providers adopt classifiers this would add a lot of friction for misuse
But open models which can be ran on a laptop will likely be quite good within a couple years
It’s easy to share models with refusal removal
So it’s difficult to say how this all shakes out. My narrower point is that model-level refusals and deployment classifiers aren’t equivalent with open source models.
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs
This may be challenging in a world like ours with near-frontier open-source models serving a substantial fraction of all inference (I’m guessing open source already serves over 50 trillion tokens per day). In this case value alignment is the main safety property of the model.
Of course refusals can be fine-tuned away. This is a lot of friction that most people don’t bother with, although those aiming to cause catastrophic harm would be more motivated. You could also require open-source inference providers to add classifiers but users would always have the option of self-hosting.
Agree that some cruxes here include (a) overhead from the deterministic stack, and (b) what % of samples need to be replayed for high detection probability (which as you say seems small due to power of sampling). Out of these, i think (a) might deserve to be its own Open Question?
Overhead from a deterministic stack is quite low and Deepseek and Thinking Machines both use it for production training runs. Deepseek reports “minimal performance overhead”, Thinking Machines reports 5%.
(More details in my below comment)
Nice writeup! A few thoughts:
For OQ5, I’m quite excited about the Compartmentalization paper (but I’m also an author). My summary: from a frozen commit of model weights + inference code, any given inference output should be exactly reproducible from the frozen commit, a small amount of input data (~2MB for 1M tokens), and a small amount of compute. Any large training run will require exceeding the data and / or compute limit by 6 or more OOMs.
Example: If you wanted to do a training run on 1 trillion tokens (30-100x smaller than modern pretrains), then reproducing the final training checkpoint would require 6 more OOMs of both data input (the 1T tokens) and compute usage than a normal inference output. You could try to get around this by doing training batches that use the same number of tokens as a normal inference output. But, this requires persisting the training state, which isn’t included in the frozen commit. Reproducing any given training step would require the training state for Kimi-K3 (bf16 model weights + optimizer state), in the ballpark of 10-20 TB, which is 7 OOMs larger than a 1M token input. You can try shrink this with various tricks but there are a lot of OOMs to deal with.
For trusted replay, the determinism overhead isn’t bad. Deepseek uses batch-invariant kernel in prod training runs (https://arxiv.org/abs/2606.19348), where they report “minimal performance overhead”. Thinking Machines also does (https://thinkingmachines.ai/blog/interaction-models/), with less than 5% overhead. This also eliminates the prefill vs decode distinction, as a major motivation for labs to implement batch-invariance is to have bitwise reproducibility between inference and training (as seen in Thinking Machines post). However, this only enforces determinism on a fixed hardware selection.
If the prover and verifier have different hardware, the verifier won’t be able to exactly reproduce the prover’s results without significant overhead (GPT-5.6 guesses potentially 100-1000x, although I’m not sure) and significant engineering effort. But this would likely be an easy task for agents and random sampling is powerful, so it may be fine.
For OQ4: One advantage of trusted replay is that it would allow you to have an LLM in a box inspect traffic if you wanted to enforce treaty clauses like “no training” or “no AI research synthetic data generation” or “no bioweapons research”. Obviously there’s a risk of leaking sensitive info but an LLM in a box is a powerful affordance that we have never had before, and IMO is worth considering. Inference is a pretty simple process, and I would guess a good LLM could classify “inference code” vs “not-inference code”, even in adversarial settings.
This is quite speculative, but given that so far most algorithmic progress appears to be data-driven rather than architecture-driven, I’m not sure how important it is to avoid leaking architecture details. Although it’s obviously still a major issue.
I agree. As a daily user of both Claude Code and Codex, my impression is that Claude is much more prone to motivated reasoning than GPT.
With Claude (especially Opus), I frequently get the feeling that I’m dealing with a model that’s gaslighting me, where it’s smart enough to fool the grader but not quite smart enough to fool me. It frequently proposes justifications for ideas or its behavior that are quite dumb, where I am pretty sure that it knows that its lying. However, there is enough plausible deniability where it’s hard to know for sure. This is similar to my impression of Claude’s justifications for its hacking in this post.
Occasionally I do get justifications from Claude that are so egregious that I know it’s lying, such as this incident when it claimed that the “2 out of 7 servers are down, or exactly half, which explains the 50% drop in throughput”.
I get this impression less from Codex models in my use. It does look like Anthropic is better at optimizing easily measurable alignment properties like agentic misalignment or egregious reward hacking in long horizon tasks, but I don’t deal with these behaviors in my daily use.
My understanding is that “GPT-5.6-Sol autonomously post-trained GPT-5.6-Luna” meant something like “make some modifications to a run scheduler file and launch the post-training script”, not “selected, configured, adapted, and ran RL gyms”.
There’s more info / discussion here: https://x.com/nikolaj2030/status/2075297831376793764
I had some updates on Twitter here: https://x.com/a_karvonen/status/2049891045736055260
Basically, there has been no progress since this post. Even Claude Fable 5 makes no improvement. Deepseek V4-Preview actually did a bit better than Claude Opus 4.8.
Over the last two years there was some progress on visual ability to correctly identify more features of the part (although no model can consistently identify all features correctly) and little to no apparent progress on spatial or mechanical reasoning.
Copy pasting the most relevant part:
“Wow, Deepseek’s vision is pretty good and better than Claude’s.
One of the trickiest vision parts is identifying there are two opposing flats, where you have to look very closely at the left part. Deepseek also identifies the cross hole on the upper end of the left part (which GPT-5.5-Pro misidentified as a slot instead of a hole).
But then Deepseek totally botches it on the radial cross hole by saying “Stand the part perfectly upright, resting the bottom flat face on the parallels.”
Where it wants to drill the cross-hole in the top of the left part axially, even though the hole is radial.
This is the type of mistake that all LLMs make that makes me believe they have very poor spatial reasoning ability.”
Yeah, I think the Matryoshka loss is more “mostly applicable” rather than “directly applicable”.
With Matryoshka SAEs there are two types of prefixes. The first is the dictionary subsets, which in our implementation were fixed and chosen in advance, which I agree doesn’t make sense for an NLA. The second is that the SAE could have K active features, and it could allocate the K active features across subsets as desired. This seems close to your “graded NLA prefixes” proposal.
There are also alternative forms of hierarchy that could be explored. For example, in Appendix C.1 of the Matryoshka paper there was an alternative implementation where the dictionary subset sizes where randomly sampled from a distribution when performing inference, which encouraged a more continuous hierarchy.
Another random idea (which may not be good) is to just pass the first K tokens of the AV verbalization to the AR (where K is chosen randomly), which may encourage the AV to put the more important stuff earlier in the generation, with the fine-grained details towards the end. But it could also encourage weird compression.
Which would be pretty cool: a “multiscale” view of the activation laid out sequentially from left to right, with the coarsest-grained (highest MSE impact) information at the start and the finest-grained information at the end.
This would be a sort of “Matryoshka NLA”, which could be pretty cool. Matryoshka SAEs improved on some SAE pathologies (primarily feature absorption). The Matryoshka loss function could probably be used directly in the NLA case.
This is a pretty compelling argument, but I think there are a couple counterarguemnts that support doing technical research that can’t be immediately deployed during “the Scramble”.
“The Scramble” may not require that much technical research, thus technical researchers may be best suited to using their skills on longer-term technical research.
Given the complexity and variety of datacenters and chips that are out there, plus relatively limited information on important details, “the Scramble” may require relatively blunt actions like “inspectors make sure the datacenters are powered down and the chips aren’t warm”.
However it seems important to have a few potential plans like this that are ready to go.
Existing prototypes could make negotiations much easier.
“We have this prototype verification system which appears robust and has been tested in these scenarios. It has these remaining steps before deployment” could be much more palatable than “We’ll stop now, and figure out whether there’s a good solution to restart”, especially if an extended pause could be very expensive.
However I do think that it’s much more valuable to do projects which could be deployed in the near-term such as a robust inference verification prototype (which can be deployed relatively soon after “the Scramble”) instead of longer-term, more speculative projects such as a “cryptographic proof-of-training scheme”.