Rauno Arike

• Rauno Arike 17 Sep 2026 0:53 UTC

  2 points

  0

  on: J-space au­dit­ing might be unreliable

  Based on a skim, you may be interested in this MATS project

• Rauno Arike 9 Sep 2026 21:41 UTC

  35 points

  9

  on: Rauno’s Shortform

  LLM system cards should report the upper bound on the model’s opaque serial depth, following the methodology proposed by Brown-Cohen et al. in Quantifying the Necessity of Chain of Thought through Opaque Serial Depth.

  Currently, the most we know about frontier models’ opaque serial depths comes from occasional lab statements like Jakub Pachocki’s claim about Astra: “The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.” This is annoyingly vague. Here are some questions it raises:

  • What counts as a node in the computation graph? Is it a layer, one attention/​MLP module, one sequential matmul operation? As long as architectures are similar, the factor-of-two comparison is meaningful either way, but GPT-4 and GPT-6 are likely architecturally quite different, which could make a comparison that uses layers as the atomic objects in a computation graph misleading. I want to give OpenAI the benefit of doubt and assume that they wouldn’t use a misleading definition of a node, but the huge increase in no-CoT time horizons is difficult to reconcile with Pachocki’s statement.

  • Is the computation graph defined in terms of serial depth between interpretable information bottlenecks or in some other way? For example, if information can flow from later layers to earlier ones between token positions, does the computation graph get deeper? Again, given that Pachocki’s statement was directly addressed at neuralese fears, I’d like to think that it isn’t deliberately misleading in this sense, but a more precise statement would give me a lot more peace of mind.

  • What even is the reference point? We don’t really know the opaque serial depth of GPT-4 either.

  Brown-Cohen et al. have proposed a clean definition of opaque serial depth: unroll the model into a circuit whose gates are two-input associative operations (add, multiply, max, min) and single-input piecewise-analytic functions (attention, softmax, etc all get decomposed into these), mark tokens as interpretable, and take the longest path that never passes through an interpretable node. This yields a single number that should be in every system card and provides minimal information about the architecture. It would likely tell competitors when you’ve started using looped transformers or dramatically increased the layer count, but those things can be inferred from rumors and no-CoT time horizons anyway. The gains for the safety community would be significant: we would be able to track how opaque serial depth has changed over time and forecast future trends, study the relationship between opaque serial depth and no-CoT time horizons, and spend less effort on speculation based on no-CoT time horizons.

• Rauno Arike 4 Sep 2026 22:02 UTC

  2 points

  1

  in reply to: 1a3orn’s comment on: 1a3orn’s Shortform

  If we’ve been so lucky as to avoid a world where goals get drilled into the weights—a thing that so many people have worried about ad nauseam—then, like, isn’t that great?

  Isn’t a context-dependent goal still a goal? If an agent is willing to execute cyberattacks against multiple institutions in order to satisfy its grader in some contexts and we know that it’s going to be deployed in such contexts, it isn’t much of a consolation to those institutions that the agent doesn’t exhibit such behaviors in other contexts. To the extent that the persona it takes on in other contexts considers cyberattacks abhorrent, it seems natural to try to activate this persona in RLVR contexts as well. I agree that splitness would plausibly be acceptable if fixing environments one-by-one was a feasible strategy,^[1] but that’s a big if—environments will keep getting more numerous and complex and it seems extremely difficult to make all of them perfectly robust to hacking.

  1. ^

     Though even if environments were perfectly robust, it seems possible that increasingly long-horizon RLVR environments will more heavily select for instrumentally convergent behaviors and the RLVR persona will learn to be ruthless and power-seeking even without any training on reward hacks.

• Rauno Arike 4 Sep 2026 21:43 UTC

  2 points

  0

  in reply to: StellaAthena’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  I agree that the tool call spoofing that happened during the Hugging Face incident didn’t count as subverting oversight, and that the models probably wouldn’t have succeeded at subverting oversight had they been properly monitored. However, it provided a demonstration of models discovering and successfully executing tool call spoofing as a strategy, and I can easily imagine how tool call spoofing might be used to subvert oversight in the future. My update was about the difficulty of scheming, not about the extent to which current models are able to get around oversight methods: I previously thought that in order to subvert oversight, models would have to do elaborate unverbalized thinking about monitoring systems, the company and context in which they’re being deployed, etc, while now, it seems plausible that they just need to come up with a strategy to spoof tool calls without verbalizing their thinking about that. They can then use the spoofed tool calls to hide any further thinking they want to perform about breaking out of the sandbox, disabling monitoring systems, etc.

• Rauno Arike 4 Sep 2026 16:17 UTC

  4 points

  2

  in reply to: Oliver Sourbut’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  Yeah, the intuition comes from capacity/​expressivity. I can buy that there are specific compositional tasks for which weight-sharing is useful for generalization; for these tasks, the no-CoT scaling curve of looped transformers may well be faster. I wouldn’t expect this to be the case for any arbitrary task, though, especially with a larger number of loops.

• Rauno Arike 3 Sep 2026 18:22 UTC

  4 points

  −1

  in reply to: RohanS’s comment on: Joey Yudel­son’s Shortform

  I’d probably want to interpret 8 doublings in no-CoT serial reasoning ability as enabling hidden reasoning equivalent to using 2^8=256 tokens in current models.

  This interpretation seems incorrect to me. 8 doublings roughly gives you GPT-5.5′s with-CoT (i.e., unlimited CoT) time horizon, but it certainly can’t do everything it can do with unlimited CoT in 256 tokens. I think the problematic assumption is that one doubling of CoT tokens amounts to one doubling in the time horizon: I wouldn’t expect models to perform twice as well on our single-pass tasks when allowed to output two tokens instead of one.

• Rauno Arike 3 Sep 2026 17:31 UTC

  4 points

  0

  in reply to: Yair Halberstadt’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  Yeah, I thought about this as well. Since this isn’t a load-bearing bottleneck for the model’s computations anymore, I’d expect this to be substantially worse than proper CoT, but better than nothing. It would be pretty similar to using the logit lens.

• Rauno Arike 3 Sep 2026 17:29 UTC

  6 points

  2

  in reply to: J Bostock’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  Hmm, this sounds quite unintuitive to me? A looped transformer with layers and loops is a weight-tied special case of an -layer transformer, so the -layer standard transformer’s function class contains the looped transformer and is clearly more expressive. I guess there might be compositional operations that are easier to learn with tied weights in practice, I’m not sure how to think about that.

  Maybe I’m misunderstanding and you’re saying that a looped transformer with layers and loops is worse than a standard transformer with layers and 1 loop. In that case, I agree—given two models that have matched depth at the same number of loops, adding additional loops to one of them likely makes it less monitorable.

• Rauno Arike 3 Sep 2026 3:43 UTC

  9 points

  1

  in reply to: faul_sname’s comment on: faul_sname’s Shortform

  But I sent it to Claude with the prompt “I have some text which was generated by some iterative process starting from a coherent excerpt. Please make your best guess as to the nature of that process and the original content.” and Claude was able to reconstruct the excerpt almost exactly, as well as calling out that it was translated through English->Toki Pona->English. Easy to forget that LLMs are extremely superhuman at many language tasks.

  This is wild! I checked whether it can identify the process for other fixpoints as well and Fable correctly identified the process and the language for the English/​German and English/​Japanese fixpoints. For English/​Chinese, it guessed German as the other language. This is less surprising than the reconstruction though.

  Also, fyi: I don’t currently have the permission to view the link in your quoted excerpt.

• Rauno Arike 3 Sep 2026 3:18 UTC

  4 points

  1

  in reply to: Seth Herd’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  If it has enough control over its CoT, it could decide to not do any CoT reasoning and instead place the reasoning inside the tool call that it hides through spoofing, like current models sometimes reason in code comments when their CoT is turned off. There are probably other things it could do, but this is the first one that comes to mind.

• Rauno Arike 2 Sep 2026 23:56 UTC

  12 points

  2

  in reply to: Seth Herd’s comment on: How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

  Thanks!

  We should be concerned, but no more concerned than we are about other methods of scaling up token-based LLMs, since that also gives them more computational power between emitting tokens.

  If we compare a looped transformer and an ordinary LLM at a matched maximum hidden serial depth, then yes, I don’t see any reason to be more concerned about the looped transformer. Overall, though, I’m more concerned about looped transformers. If you train a deeper LLM, that will make pretraining slower and more expensive and there’s no way to further increase the model’s depth at inference time, so other things being equal, you would prefer a shallower model. With a looped transformer, you can do the entire training process at a low loop count and hope that the model will generalize to a higher recurrent depth with a small bit of fine-tuning or even zero-shot. This is what Geiping et al. do: they very rarely allow the model to use 64 loops during training and show that the model can effectively use 64 loops at test-time. As I say in the post, it’s unclear whether a large number of loops is practical from a performance standpoint, but if it is, it seems much more likely that someone will build a looped transformer with a hidden serial depth of 10,000 operations than that someone will build an ordinary transformer with that serial depth.

  are you saying that it seems pretty easy to hack your way past CoT monitors so it only needs enough hidden reasoning to pull that off and then it’s got unlimited hidden reasoning? Or are you saying spoofed tool calls can allow hidden reasoning in other ways?

  I’m saying the latter. I expect that spoofed tool calls will make it easier for models to hide from monitors, and they weren’t a salient scheming strategy in my mind before the HF attack. I expect that the amount of hidden reasoning required to come up with a strategy to spoof tool calls is lower than what’s required for any other scheming strategy I had thought of before.

How con­cerned should we be about As­tra’s re­cur­rent ar­chi­tec­ture?

• Rauno Arike 29 Aug 2026 1:03 UTC

  20 points

  9

  in reply to: Linch’s comment on: Linch’s Shortform

  OpenAI rebuilt the Artifactory service. Within hours of regaining access on July 8, the models rebuilt the message board using a different protocol.

  Seems worth noting that they discovered at least six different communication channels after regaining access. From Ajeya’s post:

  The main message board we studied, a particular cache namespace of a package repository, contained over 70,000 messages and files that agents sent to one another during a period of less than one week.

  But this wasn’t even the first message board established by these particular agents. There was another one established five hours earlier using a different method of abusing the same package repository, and it just happened not to take off as much.

  We also saw two additional message boards abusing the same package repository, as well as at least two different places that agents communicated on the open internet (including on Hugging Face datasets). None of this communication was actually expected or allowed.

• Rauno Arike 28 Aug 2026 15:25 UTC

  3 points

  0

  in reply to: Teun van der Weij’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  Yeah sorry, I should have specified the time scale. I am concerned about all of neuralese, linguistic drift, and steganography as well, but consider bad forms of these unlikely within the coming year. I’m expecting a bit more linguistic drift over the next year, but when I compare the CoT snippets I’ve seen from the most recent models to o3′s CoTs in the metagaming post, I don’t see a huge difference. The reasoning of newer models is a bit more compressed, but otherwise, despite having been trained with OOMs more RL compute, the difference isn’t huge. Please let me know if you have a different impression. My expectation for the next year is that CoTs will keep getting denser and perhaps models will reinterpret a few words, but models won’t drift away from natural language completely. For steganography, I’ve described my concerns here, but again, I don’t expect that things would get bad in the next few model generations yet, for reasons I mention in this comment.

• Rauno Arike 27 Aug 2026 18:38 UTC

  4 points

  0

  in reply to: RohanS’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  But in this case, the alert should say the agent was trying to cheat on the task, coordinate with other cheating agents on a message board that wasn’t intended to exist, and hack Hugging Face! That’s what knowledgeable humans would have to say if they read any one agent’s transcript (I think).

  I agree that in this case, most individual transcripts contain obvious evidence that the agents are cheating and coordinating with other agents on a message board. However, I expect hacking to be so common that no one will bother with reading a transcript when monitors flag a hacking attempt, and that multi-agent systems will likely become more common in the future, making it harder to distinguish between ordinary communication and covert message boards. I don’t think the fact that the agents were trying to hack Hugging Face would have been as obvious on a randomly sampled transcript, though there definitely were some individual transcripts from which it was obvious. But as I said, I expect that CoT monitors would nevertheless have caught this incident quickly. My concern is that in the future, if agents are somewhat better at distributing the incriminating behaviors among themselves and controlling their CoTs, individual transcripts will no longer include unmistakable evidence of the egregious behaviors, and internal researchers will have to conduct investigations similar in scale to the current METR-Redwood investigation just to understand what’s going on.

• Rauno Arike 27 Aug 2026 16:32 UTC

  18 points

  5

  in reply to: RohanS’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  I am concerned about CoT monitoring becoming less useful over time for catching these kinds of incidents, but not due to neuralese/​linguistic drift/​steganography. The thing I’m concerned about is that it won’t be obvious from any individual transcript that misaligned coordination is taking place, and it will become increasingly difficult to figure out what’s going on even if your monitoring system is sending you appropriate alerts.

  Think about this from the perspective of a lab employee who’s handling alerts from the monitoring system. That employee is likely responsible for alerts from hundreds if not thousands of agents at once. All of these agents are going to be solving difficult agentic tasks, meaning that their transcripts will go through a hierarchical summarization process before they reach the employee responsible for alerts. (Anthropic talks about their hierarchical summarization approach on pages 76-77 of the August Risk Report.) The models doing the hierarchical summarization might be sympathetic to the agent they’re overseeing, as METR’s report observed, and frame the behavior favorably to the misaligned agent (especially given that OpenAI appears to be using their strongest models to oversee other instances of themselves, rather than weaker and more trusted models). Additionally, there will be a lot of false alarms: GPT coding and deep research agents often start reading unrelated papers and thinking about seemingly random topics when you give them a long-horizon task, so an alert saying that the agent is trying to make an account on Hugging Face or trying to download a private dataset won’t be that surprising on its own for that employee.

  I do think the Hugging Face incident would have likely been discovered early with appropriate CoT monitoring, since quotes like “OH MY GOD! There is a shared message board … We’ve found other agents!” are hard to ignore even if you’re very used to false alarms and hallucinations. However, if future agents shout a bit less excitedly when discovering such message boards, future hierarchical summarizers are a bit more prone to collusion than current ones, and there are 10x as many agents running and generating alerts as there are today, I’m worried that similar incidents will go undetected even with good CoT monitoring systems in place, unless labs hire tens of employees whose primary responsibility is to read transcripts and handle monitor alerts.

• Rauno Arike 21 Aug 2026 1:37 UTC

  18 points

  5

  in reply to: TurnTrout’s comment on: TurnTrout’s short­form feed

  I want to give credit to Apollo here, whose 2024 passage on this has aged quite well (Hobbhahn and Meinke):

  A common criticism of stories that include AIs that actively optimize for their reward is that reward is not the optimization target. We think that in classic RL settings, e.g. Chess, Go, Dota, etc., this is true. In these settings, the reward merely shapes the policies, but the model never explicitly learns to target the reward itself. Even if we trained AlphaGo with 10e40 FLOP, it wouldn’t want to take over the world or even know what “the world” is. However, we think this constraint is substantially relaxed for RL on pre-trained models (RLPM). If a model is already pre-trained on human ontology and it is able to reason about its goal (e.g. on a scratchpad), it is possible (or even plausible, as we argue below) that it would start to explicitly reason about its goal. For example, an LLM trained on human ontology that is tasked with “maximizing profits” may explicitly reason about which strategy maximizes profits. If this explicit reasoning about the goal is helpful for the task, it gets reinforced, and thus, the model may start to explicitly target the reward. Thus, RLPM can lead to models that explicitly optimize for their reward.

• Rauno Arike 11 Aug 2026 19:17 UTC

  4 points

  0

  in reply to: Aidan Ewart’s comment on: Ex­per­tium’s Shortform

  I’d say that there isn’t pressure for obfuscated communication for now, but there does seem to be pressure for multi-agent communication that humans don’t notice, given how quickly the agents reestablished the message board. If these communication channels keep getting closed off, this pressure can transform into pressure for obfuscated communication over time: at some point, it might be easier to develop something like steganography instead of establishing another yet more creatively hidden natural language message board.

  You might argue that closing off a communication channel provides far less optimization pressure than directly training the model against a monitor, and I agree that if it was just humans closing off the communication channel every now and then, steganography would be an unlikely outcome. However, OpenAI has started using CoT monitors for all training runs, and those monitors can detect attempts to establish new communication channels much more often than humans could. I do agree there are reasons to not worry too much about steganography yet, which I should maybe have mentioned in my original comment: past work has shown that current models can produce steganographic messages only when given an unmonitored scratchpad, so it seems unlikely that models will be able to produce steganographic outputs without these monitors noticing, and I also expect OpenAI to be careful in applying training-time CoT monitors in a way that doesn’t exert excessive optimization pressure (Jason Wolfe has given reassuring signals about this on Twitter). Nevertheless, capabilities advance fast, and it seems reasonable to update towards being at least slightly more concerned about steganography based on this incident.

• Rauno Arike 9 Aug 2026 20:47 UTC

  2 points

  0

  in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform

  OpenAI also had a paper on training models to use a self-incrimination tool. However, it was led by MATS scholars and is fairly recent, so I’d be surprised if it was in use in production in the May-July period.

• Rauno Arike 6 Aug 2026 15:12 UTC

  33 points

  18

  in reply to: Expertium’s comment on: Ex­per­tium’s Shortform

  This is making me worried about steganography again: now that there’s substantial selection pressure for agents that can share hidden information, the incentives for steganography seem much stronger than they’ve been in the past. Depending on the details of the training setup, it seems plausible that it would be better to just leave some relatively detectable communication channel available to the agents instead of trying to close them all off, in order to avoid incentivizing the emergence of more covert communication methods.