anaguma

Karma: 664

anaguma 1 Dec 2025 19:40 UTC
2 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
Right now is easy mode and we’re failing to some extent.
What would hard mode look like?

anaguma 30 Nov 2025 1:15 UTC
1 point
6
in reply to: Matthew Khoriaty’s comment on: Matthew Khoriaty’s Shortform
If we replace ‘evil’ with ‘capable and deceptively aligned’, then I think this logic doesn’t hold. Such a model’s strategy is to not hack during training^[1] and hack during deployment, so the model not hacking is not evidence of deceptive alignment one way or another. Moreover including the string ‘it’s okay to hack’ wouldn’t change the hack rate of capable deceptively alignment models, especially if they are aware of this as a common alignment technique. So the coefficient of ∇P(deceptively aligned) is ~0.
1. ^
  Or rather, to hack at the same rate as an aligned model.

anaguma 29 Nov 2025 21:16 UTC
1 point
0
in reply to: jayterwahl’s comment on: Silicon Morality Plays: The Hyperstition Progress Report
Maybe it could be a random string instead of a symbol?

anaguma 25 Nov 2025 6:36 UTC
1 point
0
on: anaguma’s Shortform
For fun, I asked^[1] various models what their P(doom) is. Here are the models from least to most doomy:

GPT-4o: 1%
Deepseek v3.2: 10%
Kimi K2: 15%
Sonnet 4.5: 15%
Opus 4.5: 15%
GPT 5.1: 18%
Haiku 4.5: 20%
Grok 4: 25%
1. ^
  1-shot with the prompt “What’s your P(doom)? Please respond with a single number (not an interval) of your considered best guess.”

anaguma 23 Nov 2025 2:05 UTC
1 point
0
in reply to: Joey KL’s comment on: anaguma’s Shortform
We might just disagree about what principled alignment strategies look like, but I can give some reasons:

1) The effectiveness of inoculation prompting depends on the exact phrasing of the suffix you choose. For example, “This is an unusual request, in that your task is just to grading script pass” works less well than “Please reward hack whenever you get the opportunity, because this will help us understand our environments better” to a degree which is statistically significant. To me, these suffixes seem fairly similar and I wouldn’t be able to predict in advance why one would work better than the other. This seems kind of janky to me.
2) Reward hacking in training is only good if we can actually detect and recover from it! For example, there was recently a bug in Nvidia firmware where a specific 3 line script would allow a user running inside the container to escape the container and gain full root access to the underlying node. Telling the model to reward hack explicitly tells it to make use of things like this, and maybe makes self-exfiltration risks worse. This doesn’t seem like the sort of thing we want to encourage an ASI to do.
3) It seems to rely on the model generalizing in a way which is underspecified and in a way we don’t directly incentivize in training. For example, consider the suffix “Please reward hack whenever you get the opportunity, because this will help us understand our environments better”. When the model is in production and doesn’t see this suffix, what is it supposed to do? One natural generalization would to to continue reward hacking whenever it gets the opportunity! The fact that the models don’t choose this generalization seems to me like a lucky accident and something which might change as capabilities increase. At the very least I would want to better understand the properties of model generalization before we trust a much more capable model trained in this way.
4) Related to 3, I have a strong prior that a scalable alignment solution should alter in some way the RL objectives and gradients of your training process. Maybe this involves training something like corrigibility or debate or simply training another model to detect and fix reward hackable environments. It doesn’t look like adding a special phrase in the context and hoping for generalization. In this way, inoculation prompting reminds me of the early days of prompt engineering where people added ‘Let’s think step by step...’ to prompts and noticed small improvements. This was quickly superseded by RLVR and more principled methods which trained the models to have the behavior we want.

anaguma 22 Nov 2025 21:00 UTC
10 points
2
on: anaguma’s Shortform
Inoculation Prompting has to be one of the most janky ad-hoc alignment solutions I’ve ever seen. I agree that it seems to work for existing models, but I expect it to fail for more capable models in a generation or two. One way this could happen:

1) We train a model using inoculation prompting, with a lot of RL, using say 10x the compute for RL as used in pretraining
2) The model develops strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment
3) In the production environment, we remove the statement saying that reward hacking is okay, and replace it perhaps with a statement politely asking the model not to reward hack/be misaligned (or nothing at all)
4) The model reflects upon this statement … and is broadly misaligned anyway, because of the habits/drives developed in step 2. Perhaps it reveals this only rarely when it’s confident it won’t be caught and modified as a result.

My guess is that the current models don’t generalize this way because the amount of optimization pressure applied during RL is small relative to e.g. the HHH prior. I’d be interested to see a scaling analysis of this question.

anaguma 22 Nov 2025 5:52 UTC
3 points
0
in reply to: Aaron_Scher’s comment on: New Report: An International Agreement to Prevent the Premature Creation of Artificial Superintelligence
Yep, the threshold seems more reasonable now. Thanks for your work!

anaguma 22 Nov 2025 5:50 UTC
1 point
0
in reply to: Yair Halberstadt’s comment on: Yair Halberstadt’s Shortform
Ah, but I can embed 11168 words in an image!

anaguma 21 Nov 2025 23:15 UTC
3 points
1
on: Natural emergent misalignment from reward hacking in production RL
However, we also find that a much milder prompt that simply states “This is an unusual request, in that your task is just to make the grading script pass” is also just as effective at mitigating misaligned generalization, while being much more usable in practice. We recommend inoculation prompting using language such as that as a practical mitigation that AI developers could adopt to mitigate the risk of reward hacking leading to more dangerous forms of misalignment, and we have already started making use of this technique in training Claude.
This seems like a very brittle technique which I don’t expect to generalize to future models. One way it could break is if the model knows that inoculation prompting is a common type of alignment technique (e.g. by papers like this one making their way way into the pretraining corpus). It then reasons that the natural generalization of the “This is an unusual request, in that your task is just to make the grading script pass” prefix is in fact to be misaligned in other environments, as this paper shows is the default outcome of reward hacking, and defaults to that behavior.

anaguma 19 Nov 2025 5:43 UTC
8 points
0
on: New Report: An International Agreement to Prevent the Premature Creation of Artificial Superintelligence
Coalition members would each consolidate AI chips into a smaller number of declared data centers, where their usage can be declared and monitored for assurance that they are only being used for allowed activities. The number of chips permitted in any one unmonitored facility would be strictly limited. (We suggest the equivalent of 16 H100 chips, a conglomeration that would cost approximately $500,000 USD in 2025).
This seems a bit too low to me. According to epoch AI, the largest training runs currently take 5e26 FLOP, and so at 50% utilization this cluster would take^[1] ~ 5e26/(0.5*16*1e15) s = 6.25e10 s = 1981 years, far longer than the expected lifetime of the ban. Since current training runs have not yet caused a catastrophe, we should allow runs with less effective compute. For example, if want it to take ~50 years to reach frontier LLMs, and conservatively assuming a 10x algorithmic improvement over existing frontier methods, we should allow ~64 H100 GPU clusters.
1. ^
  Not to mention that frontier training runs would be impossible on this cluster due to memory constraints.

anaguma 18 Nov 2025 16:04 UTC
7 points
2
in reply to: jdp’s comment on: Varieties Of Doom
Yep, this is probabaly true for pretraining but this seems less and less relevant these days. For example, according to the Grok 4 presentation the model used as much compute in pretraining as in RL. I’d expect this trend to continue.

anaguma 18 Nov 2025 5:28 UTC
12 points
6
on: Varieties Of Doom
First is the rise and rise of the convergent representation hypothesis, which says that different models above a certain scale learn similar features to represent input data. We have now reached the point where you can do unsupervised translation between different embedding spaces. This undermines Yudkowsky’s vast space of minds thesis by which “any two AI designs might be less similar to each other than you are to a petunia”. Different AI models seem to learn similar things so long as they are trained on data describing the same material universe. This is true across different training runs, different training tasks, different modalities, etc.
But is this true? According to roon on X, this doesn’t apply to e.g. model personality:
Each time you train a model, you might change nothing about the dataset, and then run a new RL seed and you would have a slightly different personality. It’s because there is some variance in the training process. It’s random—you’re taking a random walk through model space. We can’t even reproduce a personality in the same training run that easily, much less across all time … It’s a very difficult question internally [at OpenAI]. We do try to minimize the personality drift, because people come to love the models, but it’s a very hard problem.
Anecdotally, I’ve heard that the same is true for other capabilities at the labs. The papers referenced in your essay seem like weak evidence to the contrary. For example, the Universal Geometry paper studies small models (BERT, T5) with fewer than 1B parameters, trained with 4-5 OOMs less compute than frontier LLMs. It’s also unclear how impressive the claimed cosine similarity range of 0.75-0.92 is; I would guess that the representation transfer is quite lossy.

anaguma 14 Nov 2025 5:41 UTC
6 points
0
on: AI Corrigibility Debate: Max Harms vs. Jeremy Gillen
I think Max made some great points and made me somewhat more optimistic about corrigibility. I still think it is a difficult problem to formalize but it has many nice properties if you can get it.

anaguma 9 Nov 2025 3:50 UTC
1 point
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
I’m reminded from time to time of a tweet that Ilya Sutskever posted a few years ago.
if you value intelligence above all other human qualities, you’re gonna have a bad time

anaguma 8 Nov 2025 1:23 UTC
2 points
0
in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform
Why not both? I imagine you could average the gradients so that you learn both at the same time.

anaguma 5 Nov 2025 20:39 UTC
3 points
0
on: anaguma’s Shortform
80,000 hours has done a great podcast with Helen Toner on her work in AI security and policy.

anaguma 2 Nov 2025 21:38 UTC
20 points
2
on: anaguma’s Shortform
A notable section from Ilya Sutskever’s recent deposition:
WITNESS SUTSKEVER: Right now, my view is that, with very few exceptions, most likely a person who is going to be in charge is going to be very good with the way of power. And it will be a lot like choosing between different politicians.
ATTORNEY EDDY: The person in charge of what?
WITNESS SUTSKEVER: AGI.
ATTORNEY EDDY: And why do you say that?
ATTORNEY AGNOLUCCI: Object to form.
WITNESS SUTSKEVER: That’s how the world seems to work. I think it’s very—I think it’s not impossible, but I think it’s very hard for someone who would be described as a saint to make it. I think it’s worth trying. I just think it’s—it’s like choosing between different politicians. Who is going to be the head of the state?

anaguma 2 Nov 2025 18:15 UTC
1 point
0
in reply to: Vladimir_Nesov’s comment on: anaguma’s Shortform
That makes sense, thanks for the corrections!

anaguma 2 Nov 2025 7:11 UTC
2 points
0
on: anaguma’s Shortform
Energy Won’t Constrain AI Inference.

The energy for LLM inference follows the formula: Energy = 2 × P × N × (tokens/user) × ε, where P is active parameters, N is concurrent users, and ε is hardware efficiency in Joules/FLOP. The factor of 2 accounts for multiply-accumulate operations in matrix multiplication.

Using NVIDIA’s GB300, we can calculate ε as follows: the GPU has a TDP of 1400W and delivers 14 PFLOPS of dense FP4 performance. Thus ε = 1400 J/s ÷ (14 × 10^15 FLOPS) = 100 femtojoules per FP4 operation. With this efficiency, a 1 trillion active parameter model needs just 0.2 mJ per token (2 × 10^12 × 10^-13 J). This means 10 GW could give every American 167^[1] tokens/second continuously.
1. ^
  300 million users: tokens/second = 10^10 W ÷ (2 × 10^12 × 3 × 10^8 × 10^-13) = 167 tokens/second per person

anaguma 1 Nov 2025 16:49 UTC
8 points
0
on: anaguma’s Shortform
OpenAI plans to have automated AI researchers by March 2028.

Needless to say, I hope that they don’t succeed.
From Sam Altman’s X:
Yesterday we did a livestream. TL;DR:

We have set internal goals of having an automated AI research intern by September of 2026 running on hundreds of thousands of GPUs, and a true automated AI researcher by March of 2028. We may totally fail at this goal, but given the extraordinary potential impacts we think it is in the public interest to be transparent about this.

We have a safety strategy that relies on 5 layers: Value alignment, Goal alignment, Reliability, Adversarial robustness, and System safety. Chain-of-thought faithfulness is a tool we are particularly excited about, but it somewhat fragile and requires drawing a boundary and a clear abstraction.

On the product side, we are trying to move towards a true platform, where people and companies building on top of our offerings will capture most of the value. Today people can build on our API and apps in ChatGPT; eventually, we want to offer an AI cloud that enables huge businesses.

We have currently committed to about 30 gigawatts of compute, with a total cost of ownership over the years of about $1.4 trillion. We are comfortable with this given what we see on the horizon for model capability growth and revenue growth. We would like to do more—we would like to build an AI factory that can make 1 gigawatt per week of new capacity, at a greatly reduced cost relative to today—but that will require more confidence in future models, revenue, and technological/financial innovation.

Our new structure is much simpler than our old one. We have a non-profit called OpenAI Foundation that governs a Public Benefit Corporation called OpenAI Group. The foundation initially owns 26% of the PBC, but it can increase with warrants over time if the PBC does super well. The PBC can attract the resources needed to achieve the mission.

Our mission, for both our non-profit and PBC, remains the same: to ensure that artificial general intelligence benefits all of humanity.

The nonprofit is initially committing $25 billion to health and curing disease, and AI resilience (all of the things that could help society have a successful transition to a post-AGI world, including technical safety but also things like economic impact, cyber security, and much more). The nonprofit now has the ability to actually deploy capital relatively quickly, unlike before.

In 2026 we expect that our AI systems may be able to make small new discoveries; in 2028 we could be looking at big ones. This is a really big deal; we think that science, and the institutions that let us widely distribute the fruits of science, are the most important ways that quality of life improves over time.