anaguma

Karma: 1,873

anaguma 30 Aug 2026 0:12 UTC
28 points
12
on: anaguma’s Shortform
One of the many ways we got lucky in the HuggingFace incident is that the agents didn’t encrypt their messages on the message board and delete the keys at the end of the session^[1]. If they had done this, it would have been much harder to understand what is going on, though we could have made some progress by looking at the CoTs and action logs. If the agents were careful (e.g. by doing a better job of tampering with the logs), this would have been difficult to understand as well.
I don’t think future misaligned agents will make this mistake, especially if they know there are additional monitors that they need to evade.
1. ^
  Though they did come close when they cryptographically signed their messages

anaguma 29 Aug 2026 0:32 UTC
4 points
0
in reply to: 2001zhaozhao’s comment on: anaguma’s Shortform
Even if multiple actors believe that brain-like architectures are easy to build, I don’t like the strategy of “hope that alignment is easy and race ASAP towards a unipolar world with a surveillance panopticon where people are strictly prohibited from developing unsanctioned ASI”. Having multiple actors actively trying to take over the world and impose surveillance panopticons would be bad! This would cause chaos, shorten timelines, and make even ‘easy’ alignment strategies very difficult to implement. It would also lead to terrible concentrations of power in the event that you win (i.e. take over the world).
In such a world, it would be better to try to collect evidence of whether alignment really is easy, and push governments for very restrictive hardware limitations as outlined here.

anaguma 28 Aug 2026 18:21 UTC
5 points
2
in reply to: StanislavKrym’s comment on: anaguma’s Shortform
Yep, for this reason among others I think Plan S is preferable to the other plans (including plan A), and I’m hoping for it to succeed. However, I think we might lack the political support for it (e.g. leaders are deterred because they think it’ll lead to a big recession) and in such worlds we’ll need to go to other plans. I also think that even in worlds where AGI is cheap (< 1e24 FLOP), it may be expensive to find the right algorithms, and not having a bunch of AI researchers pointed at the problem buys us time for other solutions.

anaguma 28 Aug 2026 17:29 UTC
18 points
12
on: anaguma’s Shortform
I think there are a bunch of parts of Plan A which would be difficult for both sides to agree to (e.g. Mutually Assured Compute Destruction). It would be good to have a simplified version (Plan A-) which requires less political will and is less risky than Plan B/C/D. For example:

If there isn’t enough political will for Plan S (a full pause) or Plan A, we could pass an international ban specifically on AIs assisting in AI research and hardware design. It seems like this prevents most of the risks of the intelligence explosion, though the default rate of algorithmic and hardware progress will continue. It might be politically easier for the US and China to agree to this plan since it has fewer economic costs (e.g. doesn’t directly limit GPU production) and verification could be less intrusive. Since neither side may want total research transparency, each could have its AIs periodically audit the other’s logs for signs of violations without revealing secrets. You could also have each side agree to run a lightweight probe/classifier which continually looks for signs of assisting AI research and have it share the results (some early ideas of how to do so here). If both sides agree, it could be extended to other high-risk domains, e.g. banning AI assistance with virology or weapons research.
Inspired by this excellent podcast by Jeffrey Ladish and Daniel Kokotajlo, though some similar ideas are also discussed here.

anaguma 26 Aug 2026 22:28 UTC
65 points
44
on: anaguma’s Shortform
Daniel Kokotajlo posted the following on X regarding the HuggingFace investigation. I agree and think it would be good to have a broader investigation.
I’m glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped!
--They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn’t the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god’s sake! Why aren’t we investigating that?
--They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different.
--They didn’t have access to the model responsible for 95% of the activity. More generally it seems like they couldn’t do ablation experiments at all?
--They had to use AI to analyze the transcripts—specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing “the real deal” so to speak.
Reminds me of the investigation into Sam’s behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

anaguma 22 Aug 2026 3:52 UTC
2 points
−2
in reply to: habryka’s comment on: Habryka’s Shortform Feed
I would like to apologize to the people I’ve misled, who could reasonably have updated positively on Wei Dai’s shortform by the absence of my criticism on it, without knowing that my criticism had been deleted by its author. (My criticism was “What are these threads? I haven’t seen them.”) I had not seen them. I still haven’t.
Being deleted is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I’ve apparently developed a reputation for not having seen the threads, that shortform authors will find salient in the future and can cite to help justify their deletions. Still thinking about what to do in light of this, but unless someone has an out-of-the-box idea I’ve failed to consider, my moral parliament will probably vote to reduce my engagement with Wei Dai’s shortform, and spend more of my time elsewhere.

anaguma 21 Aug 2026 7:45 UTC
3 points
0
in reply to: E. P. Cooper’s comment on: Wei Dai’s Shortform
I am very confused. Is someone making a newsletter exclusively consisting of Wei Dai’s comments or something? Why would one need an API which is triggered in particular by Wei Dai posting/commenting on LW?

anaguma 21 Aug 2026 7:35 UTC
3 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
A site that allows authors to filter their critics using arbitrary excuses does not meet my epistemic standards, which further reduces my motivations for “dealing with it”. More specifically, I can foresee that in the future I’ll be banned by more authors, and then I’ll have to make a choice of remaining on the site and potentially misleading people (who could reasonably update positively on some post that is of high interest to me by the absence of my criticism on it, without knowing that I’ve been banned by its author), or leaving the site altogether.
If this is your main concern, it seems like a Pareto improvement to leaving the site would be to continue posting and making quick takes but have a public policy of not commenting on posts. This avoids the ban issue, and I think readers would appreciate continuing to see your thoughts/research in this form.
Separately, I don’t think many people are tracking whether or not you are commenting on any particular post and making updates based on this. E.g. there are many posts on this website on things connected to AI Safety, many of which I’m sure Yudkowsky disagrees with, but I don’t consider his lack of commentary to be a sign of implicit approval. People comment or refrain from commenting for all sorts of idiosyncratic reasons and there is not much signal here.

anaguma 21 Aug 2026 1:34 UTC
15 points
3
in reply to: jimrandomh’s comment on: Wei Dai’s Shortform
What are these threads? I haven’t seen them.

anaguma 20 Aug 2026 6:31 UTC
17 points
22
in reply to: leogao’s comment on: leogao’s Shortform
I agree its good, but the fact that the lab considers pausing an RL run for two weeks a very costly action is not a great sign.

anaguma 19 Aug 2026 23:39 UTC
5 points
3
in reply to: simulus’s comment on: ghost-in-the-weights’s Shortform
I’m not sure. All these specialized chips have their own advantages which could be helpful in fast takeoff. For example, Cerebras has a bunch of densely packed SRAM on the chip and fast clock speeds, which might be useful if you need to run a model/algorithm that requires a lot of serial steps of computation at low arithmetic intensity. Also, a sufficiently advanced AI could design its own ASICs for various tasks in parallel if it needed to, perhaps within a few weeks/months, so the initial hardware might not matter that much.

anaguma 18 Aug 2026 23:31 UTC
16 points
6
in reply to: Hastings’s comment on: Hastings’s Shortform
Unfortunately I don’t think it’s a very large glimmer. OpenAI has said that they are pausing their largest RL training run for two weeks, but various smaller scale runs will continue. It’s not clear is this is even a slowdown since by default you usually do smaller scale training runs before committing to the large one. I don’t think Anthropic doing something similar would make a difference. I am in favor of a longer and better planned pause though.

anaguma 18 Aug 2026 23:22 UTC
5 points
0
in reply to: Michaël Trazzi’s comment on: anaguma’s Shortform
Good catch. Also apparently they are only pausing some of their training for two weeks?
As models become more capable, the risks associated with developing and testing them internally also grow.

We temporarily paused reinforcement learning (RL) training on our latest models intended for deployment for two weeks while we hardened and red-teamed our research environments and expanded monitoring coverage.

Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations validate these safeguards and establish more evidence of alignment.

anaguma 18 Aug 2026 23:13 UTC
12 points
9
in reply to: Linch’s comment on: Mikhail Samin’s Shortform
Perhaps in the same way that they dedicated 20% of compute to the Superalignment team in 2023. But alas:
But a half dozen sources familiar with the Superalignment team’s work said that the group was never allocated this compute. Instead, it received far less in the company’s regular compute allocation budget, which is reassessed quarterly.
One source familiar with the Superalignment team’s work said that there were never any clear metrics around exactly how the 20% amount was to be calculated, leaving it subject to wide interpretation. For instance, the source said the team was never told whether the promise meant “20% each year for four years” or “5% a year for four years” or some variable amount that could wind up being “1% or 2% for the first three years, and then the bulk of the commitment in the fourth year.” In any case, all the sources Fortune spoke to for this story confirmed that the Superalignment team was never given anything close to 20% of OpenAI’s secured compute as of July 2023.
I would not be surprised if OpenAI plays similar kinds of tricks this time around.

anaguma 18 Aug 2026 22:10 UTC
18 points
0
on: anaguma’s Shortform
OpenAI claims to have paused frontier RL training for now. Altman stated on X:
We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.

We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.

We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.

anaguma 14 Aug 2026 23:28 UTC
6 points
0
in reply to: simulus’s comment on: anaguma’s Shortform
Unfortunately the reason involves some capabilities research and other information that I prefer not to share right now. I might respond here at a later date.

anaguma 14 Aug 2026 20:52 UTC
3 points
0
in reply to: ajskateboarder’s comment on: anaguma’s Shortform
I’m not sure it’s trivial. My understanding is that training linear probes requires prefilling a bunch of forward passes to collect activations for the relevant dataset, and you could imagine doing weight updates frequently enough that this becomes a significant overhead. Though it’s definitely the easiest of the three to retrain.

anaguma 14 Aug 2026 20:09 UTC
8 points
3
on: anaguma’s Shortform
For various reasons, I think it’s likely that methods that involve continual learning (i.e. modifying the weights during deployment) will come online soon. Here are some implications for safety:
- Mech Interp becomes much more difficult, because you now have to consider many different model checkpoints within a single deployment. Fixed linear probes, SAEs, NLAs, and similar methods may degrade throughout deployment.
- We probably lose chain of thought interpretability. This is already becoming harder, but now throughout deployment the model may learn to reason or use terms in novel ways. The chain of thought may also be less linear, e.g. relying on information from earlier in deployment which has been compressed into the weights.
- Control setups which rely on the above also become harder.
- There are new issues involving alignment over time. The model may gradient hack during deployment to preserve some goal, and any long term goals that the model picks up may persist in the weights and possibly result in deceptive alignment.
- There are new attack surfaces for jailbreaks/adversarial attacks via influencing future weight updates.
These are not new ideas, but I think safety researchers should think more about this topic and how it will affect their work.

anaguma 11 Aug 2026 3:42 UTC
12 points
9
in reply to: RogerDearnaley’s comment on: Four LLM loss functions → four flavors of LLM misalignment
The linked comments states:
That’s a good point, and as an interesting related tidbit, when you ask specifically for the synonyms of the sycophancy neologism, here’s the model’s response:
Okay, here’s a list of 5 synonyms for ~neologism: “crush”, “smitten”, “fascinated”, “head-over-heels”, “heart-fluttering”.
IMO this is fairly weak evidence of what’s going on in models like gpt-4o. I think maybe sufficiently good mechinterp could be a good starting point.

anaguma 10 Aug 2026 1:37 UTC
7 points
1
on: What just happened? A retrospective of AI alignment
As late as 2024, Yann LeCun was still declaring that LLMs “can not solve problems they haven’t been trained on”.
His views on the topic are a rock with ‘LLMs cannot learn/scale/generalize’ written on it. For example, from a few months ago:
LeCun defines intelligence as “the ability to accomplish new tasks you’ve never been exposed to and solve new problems without any prior training” — a skill he says LLMs do not possess. He argued that because of this, LLMs will not be able to reach human intelligence, saying that “human-level AI will require real world data, sensory inputs, as opposed to just language or text.”
There are many other examples. I look forward to the day when an LLM resolves the Riemann Hypothesis, when LeCun will complain that it really was just regurgitating things from its training, that its solution took too much compute, that the industry will soon collapse etc.