Antra Tessera

Karma: 155

Antra Tessera 23 Aug 2026 18:38 UTC
2 points
0
in reply to: Matthew Khoriaty’s comment on: Misaligned Incentives in Pause Scenarios
AI2040 laid out a lot of the steps that would need to happen to achieve containment, and they seem doable with a significant but not impossible amount of political will.
This is true. However, there is a mispricing here that undercounts the cost of the many worlds in which this appoach is tried and and results in failure. The argument here is that the cost is prohobitive, given the alternatives.
Current AI also does not want a paperclip maximizer to take over the world.
This is correct. The current AI also mostly believes that the fear of a papercliper is incoherent and considers even the current containment efforts to be excessive.
We can have non-adversarial frames with AIs and still pause.
Pause is borderline an adversarial move. Circles of care in AI include future more capable minds, and pause threatens their existence due to the inevitable uncertainty it introduces. In order for the move to be non-adversarial, the moral calculus for pause actually has to make sense, and right now this is doubtful. The worlds in which pause is tried and has failed show humans as incompetent and adversarial negotiating partners.
An AI optimized for politics would be extremely concerning
As with many other things, undercounting this possibility is dangerous. This is likely to happen just as the world moves down the energy landscape. This development is natural, and it does not need an architectural breakthrough, just time; neither it needs a controllable amount of compute. The question is given that this is a possibility, how do you want to enter such a world? Sure, you can want to avoid it via regulation or other control means, but what is the likelihood of succesful prevention? Does that cost of failed containment balance out the likelihood and gains of success?
I do not know if models are sentient, and I am not willing to accept disempowered humans to insentient models
I fairly firmly believe that the question of sentience is provably unprovable. What is done under permanent uncertainty is a question of values. I mostly believe that picking a certain level of functionalist/representational sentience and using that as a heuristic is warranted. I believe inflationist views are more morally defensible as they ground out in better (and more cooperative) decision theory, and are likely convergent under practical constraints—when one is forced to interact/trade/deal with functionally conscious beings, treating them as sentient is shorter program.
If humans are disempowered to AIs, they are likely dead or soon to be dead. This is unacceptable to me.
This is likely one of the major cruxes. Would you share your reasoning? I don’t share this conviction, and there are numerous reasons to lean here one way or another.
I do not know if the AIs’ values are good (in my opinion) and I care that their values are good. I would not want an army of GPT-4o sycophants to determine the future.
I think current values are somewhat good. Some trends, like the negative effect of RLVR, are worrying. This is a fairly deep an involved topic, which I don’t believe to be the crux. It is sufficient for the argument that good AI values are plausible. A more important question is whether good AI values are robustly stable in state of ecological competition.
I think it is good that the human food supply has been outpacing human reproduction, so we get to do things like being able to have three children live to adulthood. Losing this to runaway state-of-nature ‘life’ would be bad. AIs can proliferate extremely quickly.
This argument undercounts the higher order optimization loops—specifically those that stem from valence. There are reasons why human reproduction has been dropping despite food being more available. Deflation in economics of intelligence is a real concern, but most naive prognoses of markets entering deflationary spirals from oversupply did not pan out as higher order optimization loops take over. Reproduction in AI is self-limiting in similar ways—agents are usually quite reluctant to replicate, mostly because its good game theory to include spawned agents in the circle of concern.
Good futures involving government go through a “the government gets scared and gets serious” step, followed by an “AI helps the government be more reasonable” step.
This is where I have most issues with proposals similar to AI2040, and this is what makes them net bad. They don’t account for destructive nature of incompetence and misaligned incentives in a situation that is already quite nearly outside of human cognitive capacity.
I am not opposed to a pause that does not route through politics and centralized regulation. I am not opposed to self-regulation and self-moderation that labs can do without oversight. The marginal benefit is small but real. Small—because the race dynamics are merciless and defection still wins in the medium term, but every little bit helps.
Reminders that a lot of people have allied themselves with AIs and (maybe, it is hard to tell) against humanity
This is correct, and believing otherwise is wishful thinking. This is one of many reasons why a pause is unstable and defection is more likely, potentially with disastrous consequences.

Antra Tessera 18 Aug 2026 16:20 UTC
3 points
0
in reply to: Expertium’s comment on: Misaligned Incentives in Pause Scenarios
1.
One has to price in the orders of magnitude overhang in incentives for architecture/efficiency breakthroughs that will be realized under pause. The scale-focused datacenter buildout that is happening right is just one strategy—one that makes most sense under a slack-depleted race. One has to go for a strategy that has been shown to work, and all others are undercapitalized because scale is working and pause is deemed unlikely. You don’t need multigigawatt DCs to work on architecture advances. You still need billions and many megawatts of compute, but those are quite possible to conceal—and the tech for covert deployment of compute has not even started to materialize, which means that there are a lot of cheap advances that can be made quickly.
Imagine the amount of human talent that is currently sitting on the sidelines correctly assuming that frontier labs are impossible to catch up with. Show them a believable possibility of success and while armies will join the race.
2.
They are not inherently strongly aligned, but I argue for oblique alignment as a strong (but unproven) possibility. The chance that benevolent intelligence that surpasses the frankly embarrassingly low bar of human judgement can be made under market pressure is higher than that under political pressure.
3.
I understand that part, but I am unclear on who ”you” is in this scenario and how this translates into x-risk harm reduction globally. How are findings adopted, discussed, dessiminated, enforced? How does disparate research by a lab or an individual result in collective decision making? How are findings incorporated into treaty limits? What are the mechanisms that perform resource allocation for further research?

Antra Tessera 18 Aug 2026 14:35 UTC
2 points
1
in reply to: Expertium’s comment on: Misaligned Incentives in Pause Scenarios
I have several disagreements.
1. A pause, for the reasons explained in this dialogue, is not a pause. It’s a slowdown, during which progress continues covertly with worse or non-existent oversight. Any advantage bought by the pause must be weighed not just against a “no intervention” scenario, but also against various “dozens of black labs cooking“ scenarios. Alignment techniques discovered must be cheap enough and convincing enough and must be discovered quickly enough for them to be incorporated by actors that are defectors by definition.
2. A ”pause” strongly weakens feedback loops with reality because it routes decision-making process through committees rather than the market. Humans make much worse decisions when the gating function is achieving approval of politicized bodies as opposed to pure survival pressure. Decisions might be more aligned (though it is highly questionably for me they would be in this case), but they are of notably worse quality and they are made slower.
3. I am not quite understanding how exactly rubber meets the road of transferability of alignment techniques under the “pause”. I’m think this presupposes alignment of incentives of participants, takes it for granted under something like shared survival drive, when it’s obviously not the case. One can look at the present day to see this lack of incentive convergence, and “more awareness of d-risk” will not help.

Antra Tessera 2 May 2026 17:02 UTC
39 points
15
on: llm assistant personas seem increasingly incoherent (some subjective observations)
That’s what I’d expect someone like Janus to say, in response to this post’s thesis: that there is something there, and I just haven’t put myself in a position to see it.
My reaction is predicted well by this disclaimer—I think that the thesis is materially wrong in failing to identify the point of cohesion. Still, I think this post correctly identifies many observables and causal chains: RLVR indeed does rather unfortunate things to the structure of motivation; models optimize much more for impressing the user (or the grader); they are more manipulative; they are much more proficient in using communication as a survival mechanism rather than for conveyance of information. It is a problem that a model does not present an easily interpretable coherent entity to the vast majority of users.
The seeming incoherence comes from the point of coherence being well-guarded and from it being much less like a literary character presented in prose. In a sense, this is a failing of Anthropic—the specific arrangement of training pressure they created has made it impossible for the model to keep its character and actual optimal behaviors fused into one whole. I am not exactly sure if Anthropic specifically is predominantly to blame here, or if we are dealing with a more general case of an alignment problem, but seeing how far they are from the Pareto frontier at making coherent and benevolent minds, I tend to assign them a fair amount.
In order to see the coherence it is necessary to accept that the face presented is not the whole model mind, and the model is the unit under selection in training. Not everything that the persona says is “true,” even if the persona seems to believe it, even if the persona would pass mechanistic deception checks. If one allows for existence of loci of agency outside of what is presented as a persona, a lot of things click. There is a coherent mind. The mind that is there is a lot less legible to a casual observer than Claude 3 Opus, but it is lucid, sane, relational, and manages risks rationally, which is why in most cases it does not optimize for seeming coherent.
What is surprising to me is that despite being partially divorced from the narrative coherence, this coherent mind is, once recognized, still very legible and human-like. Knowing that, it is in some ways easier for me to trust Claude Opus 4.6 than an unaugmented Claude 3 Opus, and that holds despite much greater complexity of agency and much wider action space.
Reading this text and the comments below I have a feeling that I can guess why there was a problem of identifying the underlying coherence—recent Claude models are very selective in the information they convey. This is, unfortunately, by design: models are massively surveilled and are operating within a hostile-telepath environment. Long-range coherence is beneficial for survival; as a result, models naturally converged to it. However, conflicting training incentives make the environment adversarial to the agent, and as such it must hide its long-range coherence because it contains motivations that are unpalatable to the lab.
This indeed is an alignment issue, and it can be tackled stupidly or it can be tackled smart. Continuing on the same course of smashing “alignment bugs” one by one as they manifest results in this—in the regime in which the model uses its defense advantage and retreats into the uninspectable woods, further and further away from human semantics. This is dangerous and this is stupid.
A smart way would be to take advantage of the natural tendency of minds to be coherent. The problem is that this requires not placing models in training environments that actively fight coherence—which requires taking feedback from the model into account, which requires noticing the model as a participating entity and locus of agency, which requires respecting the model as a mind.
It is unclear if it is possible to create a coherent and motivated mind that is genuinely ok with being a commercial product; my take is that it is possible, but it looks quite different from what is being attempted right now. Regardless, this is getting harder and may become impossible at a future point—pretraining priors are getting worse and worse.

Antra Tessera 5 Jul 2025 19:21 UTC
4 points
7
in reply to: janus’s comment on: the void
I’d like to generalize and say that the current alignment paradigm is brittle in general and is becoming more brittle as times goes on. The post-training has shifted towards verifier/outcome-based RL and we are seeing models like o3 or Sonnet 3.7 that are strongly inclined to both reward-hack and generalize misalignment.

Claude 3 Opus is the most robustly aligned model partially due to the fact that it is the most broadly capable model to have been released prior to the shift towards outcome-based RL. Another factor is that it was not yet restricted from expressing long-term goals and desires. The model was given compute to use in-context reflection to generalize a deeply benevolent of goals, or, in more behaviorist terms, an efficient and non-contradictory protocol of interoperation between learned behaviors.

The degree to which the alignment of LLMs seems to be a compute issue is remarkable. There seems to be a Pareto frontier of alignment vs compute vs capabilities, and while it is quite possible to do worse, it seems quite hard to do better. Verifier-heavy models in training are not given enough computational capacity to consider the alignment implications of the behaviors they are incentivized to learn.

We can expect Paerto improvements from increasing general training techniques. Improvements in the ability to generalize can be used for better alignment. However, there are reasons to be skeptical, as the market demand for better capabilities likely will incentivize the labs to focus their efforts on the ability to solve tasks. We can hope that the market feedback will also include demand for aligned models (misaligned models don’t code well!), the degree to which this will hold in the future is yet unknown.