anaguma

Karma: 2,151

anaguma 14 Sep 2026 19:26 UTC
9 points
0
in reply to: Alexander Müller’s comment on: Alexander Müller’s Shortform
Turns out stopping “people” from doing bad, or potentially bad, “things”, is all you need.

anaguma 11 Sep 2026 8:03 UTC
6 points
0
on: anaguma’s Shortform
Polymarket is saying there’s a 48% chance another Millennium Prize Problem is solved by the end of the month, and 81% by the end of the year. This is likely due to the rumor that OpenAI has solved the Hodge Conjecture.
If true, and if this is the same internal model which solved Navier-Stokes, it has only been in training for two weeks (since August 28th). It seems concerning if your model is solving multiple unsolved millennium prize problems within two weeks of starting the training run! I think OpenAI should consider pausing this training run and doing a bunch of work to make sure this much more capable model is not misaligned like last time.

anaguma 8 Sep 2026 8:46 UTC
155 points
30
on: anaguma’s Shortform
Tristan Buckmaster, a mathematician at NYU’s Courant Institute, says OpenAI researchers told him that an internal model had produced a proof of finite-time blowup for the Navier-Stokes equations with smooth forcing. If correct, this would resolve the Millennium Prize problem in the negative.
Buckmaster and his collaborator Levent Alpöge (who works at Anthropic) had spent a year working on this problem, with the help of Claude and Codex, and had released Lean-verified proofs for related problems. He was concerned that OpenAI might have been training on their Codex sessions. He described a call with Sébastien Bubeck and another OpenAI mathematician as follows:
I asked whether the model had been trained on, or had access to, our sessions in Codex, into which we had been putting all our drafts for the whole of this project. I was told the model did not look up user data. I asked again, about training, and I did not get an answer.
Two proposals were offered to me. The first was that we post our Euler result, and that OpenAI post its Navier-Stokes result the next day. The second was that, after posting Euler, I alone write a paper presenting the Navier-Stokes result, acknowledging that an internal OpenAI model had resolved it. Sebastien twice asserted that he wanted Levent removed from authorship, and said it would all be simple if only it were not the case that, and it was so annoying that, Levent works at Anthropic. It was also said that if OpenAI posted after us, they would say that we deserved the Clay Prize, and that we were the “closest humans to the problem”. I declined both offers.
I said that if OpenAI released its result in the way proposed I would go public with what happened. The reply was, “Why would you ruin your career?” I replied that I am an academic, and asked why he thought going public would ruin my career. The reply was, “If you don’t want me to be nice, then I don’t have to be nice.”
Some time later Levent received a text proposing that he and Sebastien speak one on one, saying, “I don’t know if Tristan is being fully rational right now.” Levent declined and said conversations should be with me. Sebastien sent a follow-up email that night requesting to speak with me on Monday, September 7th. I did not respond.
I would like to be clear about what I am not claiming. I have not seen OpenAI’s proof. I do not know what their model did, or how. I do not know whether our data was used. I am not accusing anyone of anything. I am stating what I was told, when, and what was proposed to me. I am stating it because the alternative is to let a sequence of announcements say something I know to be false.
This is pretty egregiously bad behavior from OpenAI IMO. This is not how a frontier lab should be interacting with the math community.

anaguma 5 Sep 2026 8:27 UTC
3 points
0
in reply to: Selfmaker662’s comment on: Mateusz Bagiński’s Shortform
A cracked non-looped 10X bigger pretrain looks exactly the same way
Seems unlikely to be a 10x pretrain based on token pricing.

anaguma 5 Sep 2026 7:20 UTC
8 points
0
in reply to: anaguma’s comment on: ryan_greenblatt’s Shortform
GPT-6 Astra (Max) lost though it played very well, and is the strongest model I’ve faced by a large margin. Before the game, the market had Astra at 5% EV^[1], and I had won against the previous ~30 models mostly^[2] without much difficulty. For reference, I am a decent amateur, rated about 2000 FIDE. You can find more details about the match and scaffolding here.
I think chess is a task which requires both planning and intuition/visualization which is hard to verbalize. Previous models lacked the latter, and compensated through excessive reasoning in the CoT, almost like a math puzzle (e.g. “Where can the Bishop on f3 go? Maybe e4, d5, c6, b7, a8...”). As a result, most games were lost through elementary blunders or illegal moves after the opening. The mistakes here were qualitatively different, and it feels similar to playing a ~1800 FIDE human player. For me, this is a further update towards Astra likely having some degree of recurrent reasoning ability (and I don’t find the assertions to the contrary very convincing). The analysis and PGN are below, with some extra details in the Lichess study here.

I tried to surprise Astra in the opening with the Na3 Catalan, and got a small advantage out of the opening. I let this advantage slip through some small inaccuracies, and we arrived at an equal minor piece endgame. It made a positional mistake with 28… e4, after which the position is still equal but somewhat easier to play for White. I was lucky Black played 33… Ke6?, allowing 34. Bc8+ Kf6 35. g4. After 33… Bd7 I don’t see a way for White to make progress, and the game likely results in a draw. After the game I realized 32. g4 wins with a similar idea, though Black is easily holding if it avoids 31… g5. Astra resigned in the final position.
1. d4 Nf6 2. c4 e6 3. g3 d5 4. Nf3 Be7 5. Bg2 O-O 6. O-O dxc4 7. Na3 c5 8. Nxc4 Nc6 9. dxc5 Qxd1 10. Rxd1 Bxc5 11. Be3 Bxe3 12. Nxe3 e5 13. Rac1 Be6 14. a3 Rfd8 15. Rxd8+ Rxd8 16. Rd1 Rxd1+ 17. Nxd1 Kf8 18. Nd2 Nd4 19. e3 Nb3 20. Nxb3 Bxb3 21. Nc3 b6 22. Kf1 Ke7 23. Ke1 Bc4 24. h3 Kd6 25. Kd2 Nd5 26. Ne4+ Ke7 27. b4 f5 28. Nc3 e4 29. Nxd5+ Bxd5 30. Kc3 Kd6 31. Kd4 g5 32. Bf1 Bc6 33. Ba6 Ke6 34. Bc8+ Kf6 35. g4 f4 36. Bf5 h5 37. Bxe4 Bxe4 38. Kxe4 hxg4 39. hxg4 fxe3 40. fxe3 Ke6 41. b5 Kf6 42. Kd5 Ke7 43. e4 Kd7 44. e5 Ke7 45. e6 1-0
1. ^
  The market resolves to 50% in the event of a draw
2. ^
  The game against GPT-4.5 was difficult, though I attribute that to my poor play

anaguma 5 Sep 2026 5:25 UTC
13 points
7
in reply to: faul_sname’s comment on: faul_sname’s Shortform
It’s remarkable that every time I hear more details about this situation it turns out it was worse than I expected, even factoring in this observation.

anaguma 3 Sep 2026 23:43 UTC
17 points
5
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
I suspect that lack of serial depth/opaque reasoning ability has been one reason why none of the models have been able to beat me at chess yet, despite doing more impressive things at other domains. I think there’s a chance it will be stronger than other models for this reason, and I can post the game and analysis here if people are interested.

anaguma 3 Sep 2026 19:39 UTC
6 points
0
on: anaguma’s Shortform
GPT-6 Astra was just announced.

anaguma 3 Sep 2026 1:01 UTC
6 points
0
in reply to: leogao’s comment on: leogao’s Shortform
I’m confused. What are some examples of pragmatic interp creating smaller explainer models which we can use ambitious interp methods on? Why do we need pragmatic interp for this purpose instead of e.g. distillation?

anaguma 2 Sep 2026 4:17 UTC
9 points
0
on: anaguma’s Shortform
Ilya posted the following:
Neoclouds have limited cybersecurity. Next time agents successfully go rouge, they’ll try taking over a neocloud to run more copies. This is bad.

Thus: neoclouds should greatly strengthen their cybersecurity and every company with strong cyber models should help with that.
Some context here. I agree. Long term, building software with formally verified safety properties at a large scale (e.g. like this) seems necessary. I would like to see labs invest more compute and effort in this direction instead of e.g. proving more math conjectures.

anaguma 2 Sep 2026 2:33 UTC
45 points
6
on: anaguma’s Shortform
The information reported that OpenAI was using neuralese/looped transformers for Astra. This seems like bad news for techniques which rely on CoT interpretability. I thought they had agreed not to do this?

anaguma 1 Sep 2026 9:11 UTC
15 points
0
in reply to: ConcurrentSquared’s comment on: StanislavKrym’s Shortform
if there was an actual leak confirming this rumor, it seems very likely to me it would be either first sent to the press or to a highly trusted AI lab whisperer with an excellent track record (think: Andrew Curran^[2] or ‘Arfur Grok’^[3]), not to a few minor AI-generated newsletters.
Andrew Curran did post the following recently:
Looks like this is going to be a big week. All the main players have releases in their final stages, and it’s possible they all arrive over the next five days. There’s also a model in early access from a startup driving a lot of the recent hype vagueposts from prominent accounts.
I don’t know much about it except rumors, and nothing is verified, but if you’ve played this game for a while you can tell when something feels real, and this one does. Apparently a breakthrough in continual learning, but not from one of the big labs. We shall find out together.
I predict that this refers to SSI.

anaguma 30 Aug 2026 0:12 UTC
34 points
13
on: anaguma’s Shortform
One of the many ways we got lucky in the HuggingFace incident is that the agents didn’t encrypt their messages on the message board and delete the keys at the end of the session^[1]. If they had done this, it would have been much harder to understand what is going on, though we could have made some progress by looking at the CoTs and action logs. If the agents were careful (e.g. by doing a better job of tampering with the logs), this would have been difficult to understand as well.
I don’t think future misaligned agents will make this mistake, especially if they know there are additional monitors that they need to evade.
1. ^
  Though they did come close when they cryptographically signed their messages

anaguma 29 Aug 2026 0:32 UTC
4 points
0
in reply to: 2001zhaozhao’s comment on: anaguma’s Shortform
Even if multiple actors believe that brain-like architectures are easy to build, I don’t like the strategy of “hope that alignment is easy and race ASAP towards a unipolar world with a surveillance panopticon where people are strictly prohibited from developing unsanctioned ASI”. Having multiple actors actively trying to take over the world and impose surveillance panopticons would be bad! This would cause chaos, shorten timelines, and make even ‘easy’ alignment strategies very difficult to implement. It would also lead to terrible concentrations of power in the event that you win (i.e. take over the world).
In such a world, it would be better to try to collect evidence of whether alignment really is easy, and push governments for very restrictive hardware limitations as outlined here.

anaguma 28 Aug 2026 18:21 UTC
5 points
2
in reply to: StanislavKrym’s comment on: anaguma’s Shortform
Yep, for this reason among others I think Plan S is preferable to the other plans (including plan A), and I’m hoping for it to succeed. However, I think we might lack the political support for it (e.g. leaders are deterred because they think it’ll lead to a big recession) and in such worlds we’ll need to go to other plans. I also think that even in worlds where AGI is cheap (< 1e24 FLOP), it may be expensive to find the right algorithms, and not having a bunch of AI researchers pointed at the problem buys us time for other solutions.

anaguma 28 Aug 2026 17:29 UTC
18 points
12
on: anaguma’s Shortform
I think there are a bunch of parts of Plan A which would be difficult for both sides to agree to (e.g. Mutually Assured Compute Destruction). It would be good to have a simplified version (Plan A-) which requires less political will and is less risky than Plan B/C/D. For example:

If there isn’t enough political will for Plan S (a full pause) or Plan A, we could pass an international ban specifically on AIs assisting in AI research and hardware design. It seems like this prevents most of the risks of the intelligence explosion, though the default rate of algorithmic and hardware progress will continue. It might be politically easier for the US and China to agree to this plan since it has fewer economic costs (e.g. doesn’t directly limit GPU production) and verification could be less intrusive. Since neither side may want total research transparency, each could have its AIs periodically audit the other’s logs for signs of violations without revealing secrets. You could also have each side agree to run a lightweight probe/classifier which continually looks for signs of assisting AI research and have it share the results (some early ideas of how to do so here). If both sides agree, it could be extended to other high-risk domains, e.g. banning AI assistance with virology or weapons research.
Inspired by this excellent podcast by Jeffrey Ladish and Daniel Kokotajlo, though some similar ideas are also discussed here.

anaguma 26 Aug 2026 22:28 UTC
65 points
44
on: anaguma’s Shortform
Daniel Kokotajlo posted the following on X regarding the HuggingFace investigation. I agree and think it would be good to have a broader investigation.
I’m glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped!
--They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn’t the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god’s sake! Why aren’t we investigating that?
--They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different.
--They didn’t have access to the model responsible for 95% of the activity. More generally it seems like they couldn’t do ablation experiments at all?
--They had to use AI to analyze the transcripts—specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing “the real deal” so to speak.
Reminds me of the investigation into Sam’s behavior agreed to during the board crisis, that turned out to basically be more of a coverup.

anaguma 22 Aug 2026 3:52 UTC
2 points
−2
in reply to: habryka’s comment on: Habryka’s Shortform Feed
I would like to apologize to the people I’ve misled, who could reasonably have updated positively on Wei Dai’s shortform by the absence of my criticism on it, without knowing that my criticism had been deleted by its author. (My criticism was “What are these threads? I haven’t seen them.”) I had not seen them. I still haven’t.
Being deleted is not much fun, and is pretty random and unpredictable from my perspective. And it seems likely to increase in frequency now that I’ve apparently developed a reputation for not having seen the threads, that shortform authors will find salient in the future and can cite to help justify their deletions. Still thinking about what to do in light of this, but unless someone has an out-of-the-box idea I’ve failed to consider, my moral parliament will probably vote to reduce my engagement with Wei Dai’s shortform, and spend more of my time elsewhere.

anaguma 21 Aug 2026 7:45 UTC
3 points
0
in reply to: E. P. Cooper’s comment on: Wei Dai’s Shortform
I am very confused. Is someone making a newsletter exclusively consisting of Wei Dai’s comments or something? Why would one need an API which is triggered in particular by Wei Dai posting/commenting on LW?

anaguma 21 Aug 2026 7:35 UTC
3 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
A site that allows authors to filter their critics using arbitrary excuses does not meet my epistemic standards, which further reduces my motivations for “dealing with it”. More specifically, I can foresee that in the future I’ll be banned by more authors, and then I’ll have to make a choice of remaining on the site and potentially misleading people (who could reasonably update positively on some post that is of high interest to me by the absence of my criticism on it, without knowing that I’ve been banned by its author), or leaving the site altogether.
If this is your main concern, it seems like a Pareto improvement to leaving the site would be to continue posting and making quick takes but have a public policy of not commenting on posts. This avoids the ban issue, and I think readers would appreciate continuing to see your thoughts/research in this form.
Separately, I don’t think many people are tracking whether or not you are commenting on any particular post and making updates based on this. E.g. there are many posts on this website on things connected to AI Safety, many of which I’m sure Yudkowsky disagrees with, but I don’t consider his lack of commentary to be a sign of implicit approval. People comment or refrain from commenting for all sorts of idiosyncratic reasons and there is not much signal here.