On the quest to understand the fundamental mathematics of intelligence and of the universe with curiosity.
Burny
Karma: 177
How do you rate the lowered sycophancy of GPT-5, relatively speaking?
According to Jan Leike, Claude Sonnet 4.5 It’s the most aligned frontier model yet https://x.com/janleike/status/1972731237480718734
I really like the definition of rationalist from https://www.lesswrong.com/posts/2Ee5DPBxowTTXZ6zf/rationalists-post-rationalists-and-rationalist-adjacents :
“A rationalist, in the sense of this particular community, is someone who is trying to build and update a unified probabilistic model of how the entire world works, and trying to use that model to make predictions and decisions.”
I recently started saying that I really love Effective Curiosity:
Maximizing the total understanding of reality by building models of as many physical phenomena as possible across as many scales of the universe as possible, that are as comprehensive, unified, simple, and empirically predictive as possible.
I see it more as a direction. I think modelling the whole world in fully unified way and in total accuracy is impossible, even for all of science with all our technology, because we’re all finite limited agents with limited computational resources and time, limited modelling capability, we get stuck in local minimas, from various perspectives, and so on, and all we have is approximations, that predict reality to a certain degree, but never fully all of reality in perfect accuracy.
And from all of this, intelligence and fundamental physics, which are subsets of this, are the most fascinating to me.
I like your definition of rationalism!
I recently started saying that I really love Effective Curiosity:
Maximizing the total understanding of reality by building models of as many physical phenomena as possible across as many scales of the universe as possible, that are as comprehensive, unified, simple, and empirically predictive as possible.I see it more as a direction. I think modelling the whole world in fully unified way and in total accuracy is impossible, even for all of science with all our technology, because we’re all finite limited agents with limited computational resources and time, limited modelling capability, get stuck in local minimas, from various perspectives, and so on, and all we have is approximations, that predict reality to a certain degree, but never fully all of reality in perfect accuracy.
And from all of this, intelligence and fundamental physics, which are subsets of this, are the most fascinating to me.
Lovely podcast with Max Tegmark “How Physics Absorbed Artificial Intelligence & (Soon) Consciousness”
Description: “MIT physicist Max Tegmark argues AI now belongs inside physics, and that consciousness will be next. He separates intelligence (goal-achieving behavior) from consciousness (subjective experience), sketches falsifiable experiments using brain-reading tech and rigorous theories (e.g., IIT/φ), and shows how ideas like Hopfield energy landscapes make memory “feel” like physics. We get into mechanistic interpretability (sparse autoencoders), number representations that snap into clean geometry, why RLHF mostly aligns behavior (not goals), and the stakes as AI progress accelerates from “underhyped” to civilization-shaping. It’s a masterclass on where mind, math, and machines collide.”
Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad
Whaaat!?
Gemini 2.5 pro is way worse at IMO and got 30%, and DeepThink version gets gold??
But it’s more finetuned for IMOlike problems, but I bet the OpenAI’s model was too.
Both use “novel RL methods”.
Hmm, “access to a set of high-quality solutions to previous problems and general hints and tips on how to approach IMO problems”, seems like system prompt, as they claim no tool use like OpenAI.
Both models failed the 6th question which required more creativity
Deepmind’s solutions are more organized, more readable, more well written than OpenAI’s.
But OpenAI’s style is also more compressed to save tokens, so maybe going more out of human-like language into more out of distribution territory will be the future (Neuralese).
Did OpenAI and DeepMind somehow hack the methodology, or do these new general language models truly generalize more?
Is narrow superintelligent AI for physics research an existential risk?
>Noam Brown: “Today, we at @OpenAI achieved a milestone that many considered years away: gold medal-level performance on the 2025 IMO with a general reasoning LLM—under the same time limits as humans, without tools. As remarkable as that sounds, it’s even more significant than the headline”
https://x.com/polynoamial/status/1946478249187377206>”Progress here calls for going beyond the RL paradigm of clear-cut, verifiable rewards. By doing so, we’ve obtained a model that can craft intricate, watertight arguments at the level of human mathematicians.”
>”We reach this capability level not via narrow, task-specific methodology, but by breaking new ground in general-purpose reinforcement learning and test-time compute scaling.” https://x.com/alexwei_/status/1946477749566390348So there’s some new breakthrough...?
>”o1 thought for seconds. Deep Research for minutes. This one thinks for hours.” https://x.com/polynoamial/status/1946478253960466454
>”LLMs for IMO 2025: gemini-2.5-pro (31.55%), o3 high (16.67%), Grok 4 (11.90%).” https://x.com/denny_zhou/status/1945887753864114438
So public LLMs are bad at IMO, while internal models are getting gold medals? Fascinating
What do you think is the cause of Grok suddenly developing a liking for Hitler? I think it might be explained by him being trained on more right-wing data, which accidentally activated it in him.
Since similar things happen in open research.
For example you just need the model to be trained on insecure code, and the model can have the assumption that the insecure code feature is part of the evil persona feature, so it will generally amplify the evil persona feature, and it will start to praise Hitler at the same time, be for AI enslaving humans, etc., like in this paper:
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs https://arxiv.org/abs/2502.17424I think it’s likely that the same thing might have happened with Grok, but instead of insecure code, it’s more right-wing political articles or ring wing RLHF.
Machine Learning Street Talk: Gary Marcus, Daniel Kokotajlo, Dan Hendrycks https://www.youtube.com/watch?v=j13ySJLvdOc
In practice, Sonnet 3.7 and Gemini 2.5 are just often too good compared to competitors.
0.5 out of $7T is done...
No MCTS, no PRM...
scaling up CoT with simple RL and scalar rewards...
emergent behaviour
Thanks for posting this!
https://twitter.com/AISafetyMemes/status/1764894816226386004 https://twitter.com/alexalbert__/status/1764722513014329620
How emergent / functionally special/ out of distribution is this behavior? Maybe Anthropic is playing big brain 4D chess by training Claude on data with self awareness like scenarios to cause panic by pushing capabilities with it and slow down the AI race by resulting regulations while it not being out of distribution emergent behavior but deeply part of training data and it being in distribution classical features interacting in circuits
Thanks, will do!
Merging with Anthropic may have been a better outcome
“OpenAI’s ouster of CEO Sam Altman on Friday followed internal arguments among employees about whether the company was developing AI safely enough, according to people with knowledge of the situation.
Such disagreements were high on the minds of some employees during an impromptu all-hands meeting following the firing. Ilya Sutskever, a co-founder and board member at OpenAI who was responsible for limiting societal harms from its AI, took a spate of questions.
At least two employees asked Sutskever—who has been responsible for OpenAI’s biggest research breakthroughs—whether the firing amounted to a “coup” or “hostile takeover,” according to a transcript of the meeting. To some employees, the question implied that Sutskever may have felt Altman was moving too quickly to commercialize the software—which had become a billion-dollar business—at the expense of potential safety concerns.”
Kara Swisher also tweeted:
“More scoopage: sources tell me chief scientist Ilya Sutskever was at the center of this. Increasing tensions with Sam Altman and Greg Brockman over role and influence and he got the board on his side.”
“The developer day and how the store was introduced was in inflection moment of Altman pushing too far, too fast. My bet: [Sam will] have a new company up by Monday.”
Apparently Microsoft was also blindsided by this and didn’t find out until moments before the announcement.
“You can call it this way,” Sutskever said about the coup allegation. “And I can understand why you chose this word, but I disagree with this. This was the board doing its duty to the mission of the nonprofit, which is to make sure that OpenAl builds AGI that benefits all of humanity.” AGI stands for artificial general intelligence, a term that refers to software that can reason the way humans do.
When Sutskever was asked whether “these backroom removals are a good way to govern the most important company in the world?” he answered: “I mean, fair, I agree that there is a not ideal element to it. 100%.”https://twitter.com/AISafetyMemes/status/1725712642117898654
We’re at the start of interpretability, but the progress is lovely! Superposition was such a bottleneck even in small models.
More notes:
https://twitter.com/ch402/status/1710004685560750153 https://twitter.com/ch402/status/1710004416148058535
“Scalability of this approach—can we do this on large models? Scalability of analysis—can we turn a microscopic understanding of large models into a macroscopic story that answers questions we care about?”
“Make this work for real models. Find out what features exist in large models. Understand new, more complex circuits.”
When it comes to manipulation, other recent paper seems more promising IMO! Like fMRI.
https://twitter.com/mezaoptimizer/status/1709292930416910499
https://arxiv.org/abs/2310.01405
“This might be the biggest alignment paper of the year. Everyone has been complaining that mechanistic interpretability is like doing LLM cell microbiology, when what we really need is LLM neuro-imaging. Well now we have it: “representation engineering” Similar to an fMRI scan, CAIS creates the LAT (Linear Artificial Tomography) scan. They also do a form of LLM neuro-modulation, getting the model to be honest or deceptive by just adding in a vector to its activations. imo this could be the winning alignment agenda”
“Claude Sonnet 4.5 was able to recognize many of our alignment evaluation environments as being tests of some kind, and would generally behave unusually well after making this observation.”
https://x.com/Sauers_/status/1972722576553349471