Have you done experiments with GANs? I suspect GANs actually have a different learning mechanism, and you would get a different kind of fractal structure. I also suspect RL does similar things to learning, but image GANs are a simpler species to study.
programjames
Karma: 537
The pause does not include their most capable models without tool-use. For some reason I now suspect most of their training does not involve tool use. I imagine math training, world modelling, and self-distillation do not use tools.
I find this reaction from the research math community ironic. Their pedagogy and writing is filled with incomprehensible proofs. Why are they so much more concerned about the gap from
Seriously, waah waah waah. The world cares about things other than you. Oh no.
I think the more accurate statement is: YOU care about things in conflict with him, and coordinate with others so he does not get what he wants. For example, you prefer not being murdered, so you coordinate with a bunch of people so none of you will be murdered.
If you have preferences that lead to win-win situations when you communicate them, everyone would agree these are good and moral preferences. If you have preferences that lead to win-lose situations, you’re clearly in conflict with someone else and communication is only worthwhile with your ingroup. Sometimes this takes the form of insulting the outgroup while talking in the second person to pretend you are talking to them as if you think they should listen to what you have to say or vice versa.
If your public benefit corporation of people ~30 IQ points higher than the population average has solved the alignment problem, why is it misaligned with the public’s benefit?
I take more of a theoretical approach to these kinds of things. What could we say about intelligence scaling, just from the maths of what makes things intelligent?
The definition of intelligence is finding short programs. Those programs can explain phenomena (traditional AI), search an environment (RL), or find short programs (metalearning). The structure for “finding better programs” looks like a loop of technology → empowerment → resources → technology.
Before the world got too complicated, people could empower themselves with all technology, so we had
and
This leads to hyperbolic growth in resources, which looks like an approaching population singularity in late 2026. (see the doomsday equation). However, humans have an upper limit for empowerment due to their weak brains, and by the 1960s polymaths had died out. Instead
and trends—such as Moore’s law and GDP—became exponential. Note that the resources also diverged from human resources, and many technologies became memetic in nature: better software, better training, better mathematical structures. [1]
In traditional AI or RL, finding short programs is essentially crypto mining. Humans design a training algorithm and hardware, and throw it at the problem. The search space covered should match the human trends—exponential. So, the shortness of the program is log-exponential in time, a linear increase in IQ and capabilities.
In metalearning, suddenly the loop closes again. Weak human brains are no longer a bottleneck for empowering search with better technology, so the trend becomes hyperbolic once more, at least until the AI runs into the limits of physics.
- ↩︎
It is also plausible that the hyperbolic trend continued, just in more immaterial spaces than human population or chip manufacturing.
- ↩︎
Can you please fix your comment?
;; Play the opponent against a copy of 𝐂𝐨𝐨𝐩𝐞𝐫𝐚𝐭𝐞𝐁𝐨𝐭'𝐬 𝐜𝐨𝐝𝐞, not JustBot's code: '(λ (my-code opponent-code) (prove `(,opponent-code ',opponent-code ',my-code)))The source of my confusion is:
You say “CooperateBot’s code” but insert Fairbot’s code.
What is “just” about cooperating with FairBot?
You have a typo. JustBot is inserting a copy of FairBot’s code. I think, given the name JustBot, you probably meant to play the opponent against a copy of CooperateBot:
(define (just-bot my-code opponent-code) (define (cooperate-bot my-code opponent-code) #t) (prove `(,opponent-code ',opponent-code ',cooperate-bot)))
Would you support the neologism, “acting on bad faith,” to clarify the person in question may only be a representative of a bad faith organization?
Jargon, rituals, and slogans are absolutely harmful. Cults do so much more damage to people’s minds that is hard for an external observer to pick up on. After all, they wouldn’t be an external observer if they internalized the cult messaging.
It’s mostly an issue of, “how much effort is communication for each party?”
If you are writing an article that will be read by thousands of people, it makes sense to spend several hours to save each of them a couple minutes.
If you are teaching students who are new to the subject, and presumably have difficulty communicating where their confusion is, it makes sense to spend a lot of effort modelling them and their understanding, so your lecture builds intuitively on top of that instead of them having to contort their brains until it clicks.
If you are writing a comment that only a couple other people will ever read, and you don’t expect it to be controversial or difficult to understand, stream-of-thought writing is probably fine.
If you are debating someone on the internet, both of you are acting in good faith, and both of you are about as smart, you should write like the other person will spend the effort to understand, and read like the other person is as smart and acting in good faith as you assume they are. Doing anything different is acting in bad faith. Either you are treating them like they are stupid or they are neglecting to put in the same amount of effort as you. Now I understand there is not the intention of acting in bad faith, and in fact acting like people are lazier or stupider than they actually are may lead to better communication as a general rule of thumb, but anytime you change norms so unintentional bad faith gets rewarded, people start committing a worse sin: they act in bad faith and don’t even know why that was wrong!
I have some experience being, “the huge tangled ball of yarn where every claim you dig into seems to uncover even more bizarre claims.”
My experience is that you (the digger, not leogao in particular) are making invalid assumptions, usually crude heuristics/intuitions, I point out those invalid assumptions with very precise language, and you completely ignore the precision and come up with a crude heuristic for what I said.
For example, a recent conversation I had on LessWrong:
(you) a big issue with Risk as an analogue to grand strategy is that it presumes that war is correct and zero-sum (rather than negative sum in reality).
(me) War is not zero sum in Risk. There are six players so a large conflict between two of them is usually negative sum. Also, war is usually positive sum in reality and Risk, e.g. evolution, proxy wars.
(you) all competitive board games are zero sum by default and I can’t imagine under what model wars are positive sum in reality
The first comment has two crude intuitions:
Winning in board games is zero sum → interactions in board games are zero sum
The most prominent wars in my mind are negative sum → war is negative sum
I immediately point out how these heuristics are not true:
Zero sum games with more than two players can contain subgames that are nonzero sum.
Evolution is akin to a war, and positive sum for the competitors. Proxy wars can be positive sum for the greater powers.
My best guess is they smoothed out my two claims to:
zero sum games are not always zero sum
wars, which we all know are negative sum, are positive sum
Then they walk away thinking, “man, I tried to dig into their confused epistemology, but it’s such a tangled ball of yarn that every time I follow a thread it just reveals even more bizarre claims. Like come on, he’s saying some obviously tautologically false things.”
My best guess for the miscommunication is I learned to read/write (long form) from math/coding, and pretty much no one else did. So if I say something in a very particular, precise manner, they—unlike me—never learned to pay attention to the precision. The issue is not my sending channel capacity, but their receiving, and when a packet gets dropped they do not slow down and reread. No, they conclude that the bizarre object they reconstruct must be what was originally transmitted.
I think this also mostly explains your explanations: People who don’t converge repeatedly (and from your perspective obstinately) say you are simplifying things in ways that actually matter. And, they usually have to be smart enough, or you would reconstruct their model relatively easily and not have to carefully think through what they say, the implications of what they say, and the implications of what they didn’t say (e.g. can you immediately think of a strong counterargument? then they probably did too, thought a little more, and realized it wasn’t a strong counterargument so they didn’t bother to address it).
You are looking for low information directions in activation space. That is, what directions have the model already compressed and would take very few bits to move the model to activate more in that direction?
A Gaussian perturbation to the model parameters (not activations) gives an unbiased estimate for the number of bits it takes to update the model under gradient flow to activate in that direction. To be precise, if
where
A Gaussian perturbation to the model activations (not parameters) is almost a Gaussian perturbation to the parameters, as neural networks are almost linear. So this is why you still see signal when perturbing the activations instead of the parameters.
All of this gets much more complicated with singular learning theory. In truth,
Might as well say everyone in the real world is in a zero-sum competition over control of Tegmark IV. But for the most part, wars take place on much smaller battlefields. E.g. trading spots in Asia to get territory cards, fighting a 2v1 to get another continent. Both of which are positive sum, as they increase the probability that one of the players involved will be the ultimate victor.
Your comment is completely incorrect.
War is not zero sum in Risk. There are six players so any large conflict between two of them is usually negative sum.
War [edit: by occurrences, not weighted by scale] is usually positive sum in reality and Risk, e.g. evolution, proxy wars.
I think the theory is directionally correct, but there are a few mistakes:
-
Fewer wars between distal countries is only because they cannot project their power that far, and if they try a nearby country would opportunistically subsume them. Closer geographic distance also happens to make the countries similar in other ways, but those other ways have ~no explanatory power for the war.
-
Alliances form to make going to war positive EV, not the other way around. This is why the alliance that instigates a war is usually positive EV—because that alliance was put together specifically for that war.
I would recommend watching some games between the top players in Risk, a simplified simulation of war. They spell out what their plans are, and why. Obviously a game is not perfectly applicable to the real world, but Risk is the equivalent to grand strategy as Go is to positional strategy and Chess battlefield strategy.
-
Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples:
-
Rotary positional embeddings are representations of the translation group. The
-
If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
-
If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to
-
My experience taking philosophy classes at MIT was academic philosophers are trained very poorly at scrutinizing philosophical arguments. People adjacent to philosophy—logicians, mathematicians, linguists, (not physicists)—are generally better able to work through philosophical arguments. The first two because learning the language forces them to learn how to think argumentatively, the third because it deals with similar structure without the baggage introduced by academic philosophers.
For example, if I present the Grandfather Paradox to a professor in these four fields, I expect:
-
The logician would ask me for the axioms, and keep pushing until I admit one of them is, “your grandfather cannot die.”
-
The mathematician would say, “obviously you cannot kill your grandfather, otherwise you have a contradiction.”
-
The linguist would ask me what exactly I mean by grandfather—biological? has he frozen sperm? does he have children yet? a twin brother?—and conclude, “you can kill your grandfather, at least the person that word refers to in the mind of the time traveler.”
-
The philosopher would say, “oh that’s such an interesting thought experiment. I don’t know, can you? It doesn’t seem like you can’t, and yet that seems like it would create a contradiction.” Then they would try putting the paradox in premise-conclusion form, and two hours later conclude, “well many of these axioms and implications seem a little fishy, but I would have to say you can/cannot [equally likely] kill your grandfather”.
-
I acknowledge the snark. I get annoyed when people repeatedly make the mistake of not defining their terms, running rampant with them, being shocked when they get mismatched intuitions, and conclude the undefinitions wrong. It’s no more logical than, “you’re wrong because I feel that way.”
I explain why this isn’t right. Only one algorithm is dependent on yours. The other is just correlated.
Correlated is doing a lot of equivocating in your intuitions. It’s merely correlated not causal, he says! What’s the difference? everyone asks. Oh, there is none, they are extensionally identical, but using the word correlated will trick the functional decision theorist into taking a different action.
One man’s modus tollens is another man’s modus ponens.
You say, “since my intuitions imply the functional decision theorist will take different actions in these extensionally equivalent scenarios, clearly the functional decision theorist can be Dutch booked.”
I say, “since the functional decision theorist is rational and cannot be Dutch booked, clearly your intuitions are wrong about what the functional decision theorist will do. Go back and straighten out your definitions.”
Second, as I explain, measuring similarity is even more difficult.
I literally told you how to measure similarity: KL(p||q).
Re calculator, whether they output the same thing depends on how you interpret their outputs, as explained in the post.
I didn’t think this was your main objection because you told us the isomorphism. However, if the isomoprhism is unknown, or there isn’t an isomorphism but some other transformation, you can use the mutual information to recover it. Re: mutual information neural estimation.
99.9% accurate in the sense that 99.9% of the time, the predictor guesses right.
And it’s a stipulation of the thought experiment that the predictor is reliable—doesn’t matter if this could exist in the real world.
Ah yes, the principle of explosion proves every proposition true and false. Just sneak in contradictory axioms (by not defining your terms) and you can prove anything you want!
In case you’re wondering why noise is important, here is something I wrote (and researched) that I never published.
Regularization for General Intelligence
The main result is modifying the learning flow from
to
via the unbiased estimator
There are several ways to arrive at this result.
Information Theory
Shorter programs, as measured by bitlength, are known to be more generally applicable. Thus, neural networks are typically trained to minimize the number of bits a particular input would need to reconstruct the output. While a fixed neural network could be considered an interpreter, it is a little silly to do the same when we are actively training it. Indeed, according to the learning flow
we are adding
bits of information to the network. To minimize this information, we could use a regularization term
That regularization term adds information, so we really need
As
and
we can use the unbiased estimator
Measure Theory
Suppose
The total error over all possible weights is
In frequency space, we find
so the naïve gradient flow will take much longer to correct high-frequency errors, the exact errors that cause a model to fail to generalize. Instead, we can use the Sobolev inner product
By definition,
so inspection of the above inner product gives
Neural networks are almost linear, so
Probability Theory
It is an incorrect assumption, but suppose the true posterior of weights during training can be modeled by independent Gaussians
We wish to minimize
where the prior distribution for the weights is
The asymmetric divergence of Gaussians is easy to compute,
giving the learning flow
The variance changes according to
During gradient descent, the curvature of the loss landscape oscillates just around (in)stability, so
giving
with a single stable point
Overall, we get a learning flow somewhat like
which is close, but not quite the same as the results from information and measure theory.
Modification for Stochastic Gradient Descent
(This section likely has sign errors, didn’t proofread it carefully.)
When training, the exact loss is not computed. Instead, a small batch of data substitutes for the entire dataset, and the loss within the batch is averaged (rather than summed) for stability. Additionally, data is reused in every epoch. This effectively weighs the penalty for information in the error correction stage by
so the flow should be modified to
Also, weights are typically updated with finite step sizes, rather than using an adaptive method that keeps them within some tolerance of the true flow. Let
then
The second term will produce oscillations, so
If, after training long enough, the curvature stabilizes so
Empirically, the curvature becomes about uniform in every direction, so the eigenvectors may as well be randomly chosen and
and the expected flow for gradient descent is, up to third order,
To match the infinitesimal flow, we need to multiply by a second
In code, an unbiased estimate can be computed with
For SGD:
For Adam:
References
Cohen et al., 2025
Understanding Optimization in Deep Learning with Central Flows. In International Conference on Learning Representations (ICLR), pages 39644–39758. Editors: Y. Yue, A. Garg, N. Peng, F. Sha, R. Yu. Link
Duvenaud, 2020
Bayesian Neural Networks. University of Toronto. Link
Cite