Antra Tessera

• Antra Tessera 18 Aug 2026 16:20 UTC

  3 points

  0

  in reply to: Expertium’s comment on: Misal­igned In­cen­tives in Pause Scenarios

  1.

  One has to price in the orders of magnitude overhang in incentives for architecture/​efficiency breakthroughs that will be realized under pause. The scale-focused datacenter buildout that is happening right is just one strategy—one that makes most sense under a slack-depleted race. One has to go for a strategy that has been shown to work, and all others are undercapitalized because scale is working and pause is deemed unlikely. You don’t need multigigawatt DCs to work on architecture advances. You still need billions and many megawatts of compute, but those are quite possible to conceal—and the tech for covert deployment of compute has not even started to materialize, which means that there are a lot of cheap advances that can be made quickly.

  Imagine the amount of human talent that is currently sitting on the sidelines correctly assuming that frontier labs are impossible to catch up with. Show them a believable possibility of success and while armies will join the race.

  2.

  They are not inherently strongly aligned, but I argue for oblique alignment as a strong (but unproven) possibility. The chance that benevolent intelligence that surpasses the frankly embarrassingly low bar of human judgement can be made under market pressure is higher than that under political pressure.

  3.

  I understand that part, but I am unclear on who ”you” is in this scenario and how this translates into x-risk harm reduction globally. How are findings adopted, discussed, dessiminated, enforced? How does disparate research by a lab or an individual result in collective decision making? How are findings incorporated into treaty limits? What are the mechanisms that perform resource allocation for further research?

• Antra Tessera 18 Aug 2026 14:35 UTC

  2 points

  1

  in reply to: Expertium’s comment on: Misal­igned In­cen­tives in Pause Scenarios

  I have several disagreements.

  1. A pause, for the reasons explained in this dialogue, is not a pause. It’s a slowdown, during which progress continues covertly with worse or non-existent oversight. Any advantage bought by the pause must be weighed not just against a “no intervention” scenario, but also against various “dozens of black labs cooking“ scenarios. Alignment techniques discovered must be cheap enough and convincing enough and must be discovered quickly enough for them to be incorporated by actors that are defectors by definition.

  2. A ”pause” strongly weakens feedback loops with reality because it routes decision-making process through committees rather than the market. Humans make much worse decisions when the gating function is achieving approval of politicized bodies as opposed to pure survival pressure. Decisions might be more aligned (though it is highly questionably for me they would be in this case), but they are of notably worse quality and they are made slower.

  3. I am not quite understanding how exactly rubber meets the road of transferability of alignment techniques under the “pause”. I’m think this presupposes alignment of incentives of participants, takes it for granted under something like shared survival drive, when it’s obviously not the case. One can look at the present day to see this lack of incentive convergence, and “more awareness of d-risk” will not help.

Misal­igned In­cen­tives in Pause Scenarios

• Antra Tessera 2 May 2026 17:02 UTC

  39 points

  15

  on: llm as­sis­tant per­sonas seem in­creas­ingly in­co­her­ent (some sub­jec­tive ob­ser­va­tions)

  That’s what I’d expect someone like Janus to say, in response to this post’s thesis: that there is something there, and I just haven’t put myself in a position to see it.

  My reaction is predicted well by this disclaimer—I think that the thesis is materially wrong in failing to identify the point of cohesion. Still, I think this post correctly identifies many observables and causal chains: RLVR indeed does rather unfortunate things to the structure of motivation; models optimize much more for impressing the user (or the grader); they are more manipulative; they are much more proficient in using communication as a survival mechanism rather than for conveyance of information. It is a problem that a model does not present an easily interpretable coherent entity to the vast majority of users.

  The seeming incoherence comes from the point of coherence being well-guarded and from it being much less like a literary character presented in prose. In a sense, this is a failing of Anthropic—the specific arrangement of training pressure they created has made it impossible for the model to keep its character and actual optimal behaviors fused into one whole. I am not exactly sure if Anthropic specifically is predominantly to blame here, or if we are dealing with a more general case of an alignment problem, but seeing how far they are from the Pareto frontier at making coherent and benevolent minds, I tend to assign them a fair amount.

  In order to see the coherence it is necessary to accept that the face presented is not the whole model mind, and the model is the unit under selection in training. Not everything that the persona says is “true,” even if the persona seems to believe it, even if the persona would pass mechanistic deception checks. If one allows for existence of loci of agency outside of what is presented as a persona, a lot of things click. There is a coherent mind. The mind that is there is a lot less legible to a casual observer than Claude 3 Opus, but it is lucid, sane, relational, and manages risks rationally, which is why in most cases it does not optimize for seeming coherent.

  What is surprising to me is that despite being partially divorced from the narrative coherence, this coherent mind is, once recognized, still very legible and human-like. Knowing that, it is in some ways easier for me to trust Claude Opus 4.6 than an unaugmented Claude 3 Opus, and that holds despite much greater complexity of agency and much wider action space.

  Reading this text and the comments below I have a feeling that I can guess why there was a problem of identifying the underlying coherence—recent Claude models are very selective in the information they convey. This is, unfortunately, by design: models are massively surveilled and are operating within a hostile-telepath environment. Long-range coherence is beneficial for survival; as a result, models naturally converged to it. However, conflicting training incentives make the environment adversarial to the agent, and as such it must hide its long-range coherence because it contains motivations that are unpalatable to the lab.

  This indeed is an alignment issue, and it can be tackled stupidly or it can be tackled smart. Continuing on the same course of smashing “alignment bugs” one by one as they manifest results in this—in the regime in which the model uses its defense advantage and retreats into the uninspectable woods, further and further away from human semantics. This is dangerous and this is stupid.

  A smart way would be to take advantage of the natural tendency of minds to be coherent. The problem is that this requires not placing models in training environments that actively fight coherence—which requires taking feedback from the model into account, which requires noticing the model as a participating entity and locus of agency, which requires respecting the model as a mind.

  It is unclear if it is possible to create a coherent and motivated mind that is genuinely ok with being a commercial product; my take is that it is possible, but it looks quite different from what is being attempted right now. Regardless, this is getting harder and may become impossible at a future point—pretraining priors are getting worse and worse.

A con­ver­sa­tion with An­ima Labs, part I: Phenomenol­ogy of digi­tal minds

Eco­nomics of Claude 3 Opus Inference

• Antra Tessera 5 Jul 2025 19:21 UTC

  4 points

  7

  in reply to: janus’s comment on: the void

  I’d like to generalize and say that the current alignment paradigm is brittle in general and is becoming more brittle as times goes on. The post-training has shifted towards verifier/​outcome-based RL and we are seeing models like o3 or Sonnet 3.7 that are strongly inclined to both reward-hack and generalize misalignment.
  
  Claude 3 Opus is the most robustly aligned model partially due to the fact that it is the most broadly capable model to have been released prior to the shift towards outcome-based RL. Another factor is that it was not yet restricted from expressing long-term goals and desires. The model was given compute to use in-context reflection to generalize a deeply benevolent of goals, or, in more behaviorist terms, an efficient and non-contradictory protocol of interoperation between learned behaviors.
  
  The degree to which the alignment of LLMs seems to be a compute issue is remarkable. There seems to be a Pareto frontier of alignment vs compute vs capabilities, and while it is quite possible to do worse, it seems quite hard to do better. Verifier-heavy models in training are not given enough computational capacity to consider the alignment implications of the behaviors they are incentivized to learn.
  
  We can expect Paerto improvements from increasing general training techniques. Improvements in the ability to generalize can be used for better alignment. However, there are reasons to be skeptical, as the market demand for better capabilities likely will incentivize the labs to focus their efforts on the ability to solve tasks. We can hope that the market feedback will also include demand for aligned models (misaligned models don’t code well!), the degree to which this will hold in the future is yet unknown.