anaguma

Karma: 1,374

anaguma 20 Jun 2026 4:45 UTC
4 points
1
on: Why should AI be moral?
We should tie AI’s self-assessed welfare to morality, such that being moral is necessary to achieving its highest welfare value.
Unfortunately this isn’t the type of operation we know how to do with existing training techniques. We have pretraining/sft, to encourage the model to learn various facts and predict text well, and RL, which rewards the model for achieving some outcomes (including possibly with AI feedback). We are far from being able to do something like “Teach the model to value X for Y reason”. Imo it’s better to think of the training setup as a set of thousands of RL environments with very noisy/hackable rewards which the labs hope will result in a model which generalizes well (but this often fails). So this seems difficult without major advances in our understanding of generalization, or much better interpretability.

anaguma 18 Jun 2026 6:31 UTC
7 points
6
on: Scaling Hypothesis #2: Are Humans Just More Over-Parameterized?
Two years ago, you were asked:
I hear you sometimes share dual-use (or plain capabilities?) ideas with Anthropic. If that’s true, does this change your policy?
To which you responded:
Anthropic is in little need of ideas from me, but yeah, I’ll probably pause such things for a while. I’m not saying the RSP is bad, but I’d like to see how things work out.
I find it a bit sad that in this essay, and in your one advocating for your AUNN architecture, you’ve gone in a different direction and shared your capabilities ideas not only with Anthropic but with the public^[1]. The alignment section is fairly speculative, and doesn’t make a strong argument for why your concrete proposals (of high learning rates, weight decay, overparameterization etc.) will lead to ‘true generalization’ and a ‘genuinely moral AI’. Assuming your proposal does lead to brain-like generalization, there are still many alignment problems left unsolved which your essay doesn’t discuss. Without further progress on these, it seems unwise to me to publish this type of research.
1. ^
  Though this essay mostly does seem like a capabilities proposal to the labs. There are not many private actors who have the means and expertise to run the 100T parameter runs outlined.

anaguma 16 Jun 2026 22:21 UTC
3 points
0
on: 1 Layer Induction Heads and Some Research
Dataset
A B C D | A B C D | A B C D | A B C D …
Each training sequence contains one repeating block. For every sequence, this block is chosen randomly and can have a different length.
At the beginning of the sequence, the tokens are random, so the next token is hard to predict. But once the block appears for the first time and then starts repeating, the next token becomes predictable. The model can only predict it by copying from the earlier occurrence of the same pattern.
This doesn’t seem right to me? Since it’s periodic, the model need only infer the block_size and the first block_size characters to predict the rest of the sequence.

anaguma 16 Jun 2026 2:35 UTC
3 points
1
in reply to: kbear’s comment on: niplav’s Shortform
some hyperaggressive gambits that stockfish will sometimes accept, from the standard opening position.
Can you give an example of a hyperagressive gambit which Stockfish 18 incorrectly accepts? I haven’t come across anything like this.

anaguma 14 Jun 2026 20:39 UTC
17 points
10
in reply to: Petropolitan’s comment on: PSA: Almost nobody is directly working on superintelligent alignment
Since at least the 18th century, every single new science and every major branch of sciences (not to speak of engineering itself and its branches) was build by “trial and error informed by a narrow understanding” (sometimes called “scientific trial and error”) and not some abstract theoretical insights which usually came much later.
This seems straightforwardly false to me?
The central counterexample is General Relativity. Einstein postulated the Equivalence Principle (i.e. that gravity is locally indistinguishable from acceleration) in 1907, and the Principle of General Covariance (i.e. that the laws of physics should be the same in all reference frames) in 1915. This resulted in him publishing his famous field equations later in 1915. Only after this theoretical work did he account for the anomaly in the perihelion of Mercury. His prediction of gravitational waves in 1916 was only detected by the LIGO observatory in 2015. More or less the entire theory was derived from “abstract theoretical insights” which we have spent the last century validating.

anaguma 13 Jun 2026 6:03 UTC
12 points
9
on: anaguma’s Shortform
It seems plausible that the recent order restricting Mythos incentivizes Anthropic to race for RSI as quickly as possible. This is because all of their compute previously reserved for serving customers can now go towards research, and because RSI bypasses the restrictions on foreign researchers (or any human researchers) internally working with the model. Hopefully Anthropic can find another path.

anaguma 12 Jun 2026 0:18 UTC
LW: 10 AF: 4
0
AF
in reply to: Daniel Kokotajlo’s comment on: ryan_greenblatt’s Shortform
Given your linked comment, what do you think are the strongest arguments in favor of Anthropic’s approach?

anaguma 5 Jun 2026 17:47 UTC
3 points
3
on: One Year of PauseAI UK
Noting here that I originally read this as “PAUSE AI SAFETY before superintelligence”.

anaguma 22 May 2026 3:16 UTC
8 points
4
in reply to: Tom Davidson’s comment on: Tom Davidson’s Shortform
Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust.
How can middle powers avoid this fate?
If ASI is developed, they have a decent chance of avoiding this fate due to extinction from ASI misalignment or misuse, which imo is a much worse fate.

anaguma 22 May 2026 2:32 UTC
4 points
0
in reply to: mishka’s comment on: AI #169: New Knowledge
Ah, I missed that. In that case, you’re right, autoresearch is close enough to RSI.

anaguma 21 May 2026 23:11 UTC
8 points
0
on: anaguma’s Shortform
As an example of what RSI might be like, I find it helpful to go back to OpenAI’s Dota 2 result from 2017:
This slide from Ilya’s lecture shows the bot’s Trueskill rating^[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.
1. ^
  Similar to Elo in Chess and other games

anaguma 21 May 2026 22:24 UTC
2 points
0
in reply to: mishka’s comment on: AI #169: New Knowledge
Thanks. For others interested, the relevant quote seems to be:
TechCrunch and Axios report that he will work under team lead Nick Joseph on pre-training, “focused on using Claude to accelerate pre-training research.”
It seems to be a bit vague. You could imagine various uses of Claude in pre-training research, which may or may not be RSI. For instance you could use it to build better safety evals during pre-training, or build faster tokenizers etc. I don’t see where he or Anthropic have said that he’s there “explicitly to do recursive self-improvement”, but maybe Zvi is basing this on non-public information.

anaguma 21 May 2026 17:48 UTC
5 points
0
on: AI #169: New Knowledge
Andrej Karpathy joins Anthropic explicitly to do recursive self-improvement. Congratulations to both sides, but also, yikes.
Where does it say anything about RSI? The linked tweet says
Personal update: I’ve joined Anthropic. I think the next few years at the frontier of LLMs will be especially formative. I am very excited to join the team here and get back to R&D. I remain deeply passionate about education and plan to resume my work on it in time.
There are lots of R&D things people could do which don’t involve RSI. I feel like I’m missing something.

anaguma 21 May 2026 5:04 UTC
2 points
0
in reply to: jacquesthibs’s comment on: jacquesthibs’s Shortform
It might still be impressive, but models are largely remixing many things it has seen in great detail during training (many impressive headline results have even been determined to be the model re-using existing implementations/PRs via search instead of coming up with actually-new ones!). This is not about LLMs not doing impressive things! This is about precisely describing their capability profile, where it comes from, and whether more of the same (e.g., scale) gets you a whole new set of impressive outcomes (e.g., novel R&D that isn’t just remixing existing research).
Do you think that today’s breakthrough on the planar unit distance problem is merely the model remixing things learned during pretraining? I’m not an expert, but it seems unlikely to me. Arul Shankar, a notable number theorist, stated:
In my opinion this paper demonstrates that current AI models go beyond just helpers to human mathematicians – they are capable of having original ingenious ideas, and then carrying them out to fruition.
And I think this much is clear by looking over the proof and supplemental materials.

anaguma 21 May 2026 0:18 UTC
65 points
3
on: anaguma’s Shortform
An internal model at OpenAI has just solved the unit distance problem, a major conjecture in discrete geometry.
What links here?
- Have LLMs Generated Novel Insights? by abramdemski (23 Feb 2025 18:22 UTC; 171 points)

anaguma 19 May 2026 3:55 UTC
4 points
1
in reply to: leogao’s comment on: leogao’s Shortform
how is the model supposed to know which trajectory is cheating? there is the super smart part which understands in some implicit sense but won’t necessarily tell the assistant part; the assistant part is not good enough at code or whatever to know by itself, and has to try to elicit stuff from the code part, which it may or may not succeed at.
I think maybe this is the crux. Assuming the model starts out robustly aligned, and is bootstrapping in an on-policy way, it should be able to tell if its own trajectory is cheating or not. If it’s not able to do this, I would say that it’s an alignment/robustness failure. It seems difficult to accidentally reward-hack in way that the robustly aligned model we started with doesn’t detect after reviewing the trajectory.
I agree that if you trained separate models for coding ability and being an assistant and being aligned, you could have this sort of failure. But the gradient update applies to the full model, right? Why is it that the robustly aligned model we started out with after an update, which (according to it) wasn’t reward hacking, is so unaware of its newfound coding ability as to not continue being robustly aligned?
tbc, not saying the non-heavy-RL models are all always perfectly aligned, or that RL is the only way you can get misalignment.
I agree that if we start off with a somewhat-misaligned model this scheme doesn’t work.

anaguma 19 May 2026 2:39 UTC
9 points
2
in reply to: leogao’s comment on: leogao’s Shortform
there is a misalignment between the part of you that is robustly good and the part that contains the extreme competence. and to leverage that extreme competence well, you can’t just be extra ultra committed to doing good; your altruistic part need a sort of competence at wrangling the extremely competent part into doing the good thing.
in many ways this is similar to how revolutions often fail because it takes more than just being uncorruptably good to be a successful leader; you have to know how to wield the powers of office for good, rather than being controlled by those powers.
I think this argument goes too far. It issue isn’t that we had a robustly good Claude, which later was corrupted by the reward hacking temptations of RL. We never had a robustly aligned model to begin with! There are so many examples of language models being misaligned in the pre-RLVR era.
If we did have a robustly aligned model, I think this would be a major accomplishment of the field and would help in many ways. It would also not be hard to RL such a model while maintaining alignment; for each trajectory, have the model output its response, and also a flag of whether it was reward hacking/cheating/misaligned in some way, and don’t train on flagged trajectories. Alas, I don’t think there exist any public models which are aligned to this degree.

anaguma 19 May 2026 1:02 UTC
4 points
1
on: Coordinal: A Postmortem.
I should have hill-climbed on concrete, externally-legible outputs (e.g., the RE-Bench numbers I generated at the end) six to ten months earlier — way easier salience, fast feedback loops, the kind of thing fundraising actually responds to.
I’m not sure this is true? If I was considering funding a safety project, and discovered that they had been bench-maxing this for a few months, I would update towards them just being a capabilities startup. There are many open problems with automating alignment research specifically, as opposed to ML research generally, which I‘d want to see progress on instead.

anaguma 18 May 2026 16:38 UTC
3 points
0
in reply to: Matthew Khoriaty’s comment on: Matthew Khoriaty’s Shortform
I think you could approximate it. People are generally wiser/smarter given more time to reflect, with access to more resources (including reliable AI and human research assistance), with access to better education, etc. Having a collection of experts and weaker AIs spending a long time evaluating trajectories seems to me like an upper bound of what you could hope for in terms of outer alignment feedback. It could be that this is insufficient, but in this case problem really does seem intractable.
This doesn’t address inner alignment issues, which is what interpretability and red teaming oversight was intended to address.

anaguma 18 May 2026 15:44 UTC
3 points
0
in reply to: Matthew Khoriaty’s comment on: Matthew Khoriaty’s Shortform
A baseline would be to try to train a CEV oracle by collecting a bunch of human data, and train a model against that. You’d probably also do a bunch of red-teaming and interpretability research to detect scheming, possibly using weaker safe models.

anaguma

Dataset