Me explaining to someone why they are wrong to expect that the singularity will be based on analytic thought as opposed to synthetic thought, namely because as counterintuitive as it is to us analytic thought is actually a subset of/implemented on top of synthetic thought, which should be centrally thought of as something like clustering+ analogical reasoning cohering into a latent space.
FWIW, I agree with this in the sense that like, no it doesn’t really change anything that LLMs can disprove semi famous conjectures.
My point is more just that...okay as a very basic observation, if by “synthetic thought” you centrally mean clustering the entire literal problem with symbolic reasoning is that you cannot define a concept like “two” without clustering.
You have to be able to do object recognition and say “that is an object, which we will say is object one, and that is an object, which we will say is object two, and we will say that when objects cluster together in this way we have two of them”
This is just so atomic and unconscious to us that we do a map territory confusion and start thinking the symbols for “two” start meaning something outside of a latent space which can host the concept of two.
This is not to say that symbolic reasoning isn’t powerful, it is arguably the most powerful subset of the human latent space.
As Alan Kay points out, the alternative to it is doing a guess and check binary search over whole designs, and keeping these giant empirical libraries of things that did and didn’t work.
You know, the ancient Sumerians didn’t have math but they still built complex structures, they did it by basically hill climbing and basing new structures on a learned distribution of old structures to kind of intuit which structures would be likely to stand or not.
You can make things that way, but it’s not nearly as efficient as math.
But math coheres out of that, it’s the regular patterns/structure you get from this if you insist on searching for the invariants, it’s not like, some separate thing.
If this is confusing, consider that like, the fundamental epistemic foundations of mathematics eventually bottom out in clustering operations we can do extremely reliably. Bertrand Russel famously spends 379 pages to prove that 1 + 1 = 2, and the proof is famously wrong.
The way you get math is by finding a subset of clustering operations that are so reliable they can be used to define invariants and then repurposing the human in-context reward machinery to assign value to the outcome of algorithms defined in terms of these foundational clustering operations.
Which is also, basically, the way that LLMs are doing math. They use external proofs and solvers to sharpen their neural circuits until they’re reliable enough to do increasingly complex mathematical operations mentally without external checking. In humans this is accomplished by pretraining a world model and a little bit of inductive bias (I believe 1, 2, and 3 actually have dedicated neural hardware to make sure a human learns them, or something?) where you can find a reliable enough implementation of the arithmetic equality operator that you can say “1 + 1 = 2”, and then you invent algorithms for arithmetic which you use as tools to sculpt your neural circuits with in-context reward until more operations become reliable enough to increase your inferential reach and therefore the library of potential algorithms that can be executed using concepts in your latent space.
You can make things that way, but it’s not nearly as efficient as math.
Fundamentally what I’m saying is something like the natural selection argument. It’s not that natural selection is an efficient optimizer, or a particularly good way to make things, but it’s the first thing that makes things which make things, it is the mechanism by which matter becomes self organizing into life. In the same way it’s not that doing gradient search to find a giant matrix which extrapolates new thoughts based on a learned distribution of existing thoughts and observations is the most efficient way to think, but it is the way that the things which make other forms of thought possible get built. You can’t have useful symbolic reasoning without it.
It’s a bit like the argument that say, surely capitalism is inefficient since an intelligent optimizer can just directly specify the things that should be built. Why do we need to have wasteful distributed intelligence when we can have efficient centralized intelligence? And most of contemporary economics is just various ways to try and explain to people that as intuitive as this idea is it does not work and much of the complexity that a central designer has to deal with is actually irreducible and most efficiently managed by distributed design. Neural nets are a similar thing. The things you need to build to do the feature extraction that makes the application of algorithms possible have irreducible complexity that requires you to do something like gradient search in a high k-complexity program space. There is no simple trick to overcome this, the closer your method gets to the convergent shape of gradient search over generating programs of the surprising amount of detail in reality the more neural-net like it becomes in the same way that the more closely your economic system gets to the convergent shape of distributed intelligence doing price discovery the more it begins to resemble Neoliberalism, which is why the Soviet Union was ideologically doomed in a way that the modern Chinese government has been able to escape by simply establishing markets and regulating them.
I’ve been saying for some time (out loud to my friends, anyway) that deduction is just a thousand inductions in a trenchcoat. I take this to be true in an abstract epistemological sense, but perhaps more importantly in the sense of the physical operation of our brains. It’s obvious to me that the human brain doesn’t contain a formal logic module. I don’t see why we should expect AI to be any different (aside from “it’s in a computer, and I think I know what computers are”).
Neural nets are a similar thing. The things you need to build to do the feature extraction that makes the application of algorithms possible have irreducible complexity that requires you to do something like gradient search in a high k-complexity program space. There is no simple trick to overcome this
This isn’t the main claim in your argument (I recall this past comment where you touched on it more extensively), but I’m not sure it holds up on its own. I don’t think the fact that bigger/more complex NNs perform better supports “the built artifact is irreducibly complex,” because that same fact is (at least as) consistent with an opposite stance: increasing scale mostly buys you a bigger search space, where the soft inductive biases induced by that scale have more flexibility to find simple solutions that still fit the data. More scale doesn’t necessarily mean more complex solutions! (This is one of the core lines of Andrew Gordon Wilson’s work, trying to explain why deep learning doesn’t land us in overfitting hell.) So increasing scale can get us closer to an optimal solution even if the size of that solution is relatively small and fixed.
We see a version of this in the lottery ticket hypothesis, which states that trained NNs contain a much sparser subnetwork that matches or exceeds the performance of the full NN. It’s reasonable to interpret this as implying that much of the benefit of scaling NNs comes from the combinatorial explosion of sparse subnetworks that gradient descent can then utilize. Under this framework, scale mostly buys you “more shots at the goal,” even if the goal is quite simple, i.e., better odds at finding a good subnetwork, not a “more accurate/complex” solution.
This would seem to predict that trained neural networks can be massively compressed, with the fraction you can compress away rising with scale. This doesn’t seem to be the case. Well-trained LLMs have some unused parameters, but not like, , never mind .
This would seem to predict that trained neural networks can be massively compressed
The LTH gives us a way to do this! It’s just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
with the fraction you can compress away rising with scale. This doesn’t seem to be the case.
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that “larger models have better generalization bounds and are more compressible than smaller models.”
The LTH gives us a way to do this! It’s just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
I don’t think the LTH applies to large language models. Or, to be more quantitative, I don’t think you can prune of the weights in a well trained LLM without significantly degrading performance. I think when papers say they found models to be highly prunable, it’s usually a case of 1) They tested on models that are undertrained or badly trained and so don’t use most of their capacity or 2) They have low standards for what counts as “degrading performance”.
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that “larger models have better generalization bounds and are more compressible than smaller models.”
I haven’t read these particular papers, but my first reaction is skepticism. The first paper seems like a prime candidate for failure mode 1. In addition, on a skim, I am pretty skeptical of their effective dimensionality calculations. Text says
Lanczos Calculations We use GPU enabled Lanczos as implemented in Gardner et al. (2018) to compute the eigenvalues approximately, running 100 steps to compute approximately 100 of the top eigenvalues. We note that our estimates of the effective dimensionality are somewhat biased from not including all of the small eigenvalues. However, these small eigenvalues will contribute negligibly and Lanczos will converge to the true eigenvalues if we ran k steps where k is the rank of the Hessian.
Which seems insane to me. Top 100 eigenvalues for nets with between 1e5 and 1.6e6 parameters trained on CIFAR-100??? If that’s not impacting their effective dimensionality calculation it’s got to have a crazy high cutoff, surely you are not getting good performance on CIFAR-100 with 100 Hessian eigenmodes, what the hell. This seems like it’s basically throwing away the signal effective dimensionality is supposed to measure.
Second paper I haven’t understood yet.
EDIT: I have now read the second paper a little more. They seemingly don’t even try to compress regular LLMs. They compress models trained with their weird SubLoRA technique which seem to score much worse loss than regular models on text prediction. Like, these look barely better than random in table 2. And the generated text from them in Appendix F looks like word salad. So this paper seems pretty irrelevant to the question at hand.
Basically, neural network priors are pretty good if they’re trained well, and they mostly don’t waste capacity. If you’ve got a nice general algorithm that the net can learn using only of its parameters, it’ll do that. But it’s not going to stop there and leave the other unused. It’s going to learn the next useful general algorithm, and then the next one and the next one, because you can basically always get better at predicting text on the internet. If nothing else, free parameters can always be used to memorise more stuff.
I kinda agree, but it also could be smeared all over the NN, such that naively dropping some weights or something does not work, as you are pocking holes under wrong representation.
Okay but the argument for it being irreducibly complex is actually “think about this for five minutes without the assumption that it should actually be simple, once you do this is overwhelmingly obvious”. Like, the world has a lot of detail in it and the detail is not actually something you can skip over if you want to predict the next token in our environment.
I think part of the disagreement is just about the relative importance and necessity of different kinds of knowledge. There’s two basic kinds of knowledge which we usually conflate. One is something like math: Highly compressible, convergently accessible from reason, fundamental physical phenomenon etc.
The other kind is something like biology or history: Endlessly detailed, primarily reliant on memorization with limited generalization, if you delete parts of it they don’t easily grow back. Essentially irreducible blobs of Shannon entropy.
A lot of Yudkowsky’s original argument for AI doom is that he thinks the kind of fundamental physics knowledge that would make AI dangerous isn’t all that high complexity in the first place, humans are just dumb:
That even Einstein did not come within a million light-years of making efficient use of sensory data.
Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
Most of the bits of information the neural net is spending capacity on are presumably details of our particular worldline/history more than they are fundamental reasoning or physics knowledge. Maybe quantum gravity is already sitting in the weights of a frontier GPT pretrain and we just don’t know how to get it out.
However I continue to be puzzled by what Yudkowsky thinks a “bit” of knowledge is in this context:
I occasionally run into people who say something like, “There’s a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data.”
Yes. There is. The theoretical limit is that every time you see 1 additional bit, it cannot be expected to eliminate more than half of the remaining hypotheses (half the remaining probability mass, rather). And that a redundant message, cannot convey more information than the compressed version of itself. Nor can a bit convey any information about a quantity, with which it has correlation exactly zero, across the probable worlds you imagine.
This is a tautology. The theoretical limit is that every time you see 1 additional bit, which is usually literally defined as the base two logarithm of our hypothesis space, you can throw out half the hypothesis space. This is true but tells you nothing about how much information constitutes a “bit” of that hypothesis space. Like, we literally define a bit as that amount of information which would let us throw our half the hypothesis space. A bit in the sense of the fundamental entropy you need to store the frames of a falling apple and a bit in the sense of throwing out half the hypothesis space in our search over fundamental physical theories does not actually refer to the same unit of information, “it takes this many bytes to store a visual representation of this phenomenon in a general format for storing arbitrary bitmaps” is not the same thing as the k-complexity. Presumably his argument is something like general relativity can be described by a not particularly large program and therefore it should be easy to find with approximations of Solomonoff induction but I’m not convinced this is actually the case?
What always stands out to me about video nets is how janky their understanding of the standard model seems to be. They frequently generate photorealistic frames that violate the standard model in implausible and bizarre ways, which is strange because the k-complexity of the invariants in the standard model, as opposed to the specific arrangements of atoms inside it, should in fact be small enough that I would expect video models to pick up on Newtonian physics quickly and at small capacity in the same way that GPT models pick up on English grammar well before they gain the kind of world model that lets them produce coherent text with the English grammar. TinyStories shows you can fit a functional model of English grammar in under ten million parameters, and I don’t expect the standard model to actually be that much more complex than English grammar. That GPT in fact does so in the case of English grammar but video models don’t do so in the case of the standard model tells me I’m somehow either mistaken about the nature of search in the hypothesis space the standard model lives in or I’m confused about the generalization properties of neural nets, or both.
Nitpick: I think pretraining is better analogised to search in a high complexity circuit space. You only get one forward pass worth of serial depth per data point, so doing the same operation many times in a row costs a lot of description length. Post-training RL is where this transitions into something more like search in a high k-complexity program space.[1]
Me explaining to someone why they are wrong to expect that the singularity will be based on analytic thought as opposed to synthetic thought, namely because as counterintuitive as it is to us analytic thought is actually a subset of/implemented on top of synthetic thought, which should be centrally thought of as something like clustering+ analogical reasoning cohering into a latent space.
Explanation:
https://x.com/QiaochuYuan/status/2080485791952928868
FWIW, I agree with this in the sense that like, no it doesn’t really change anything that LLMs can disprove semi famous conjectures. My point is more just that...okay as a very basic observation, if by “synthetic thought” you centrally mean clustering the entire literal problem with symbolic reasoning is that you cannot define a concept like “two” without clustering. You have to be able to do object recognition and say “that is an object, which we will say is object one, and that is an object, which we will say is object two, and we will say that when objects cluster together in this way we have two of them” This is just so atomic and unconscious to us that we do a map territory confusion and start thinking the symbols for “two” start meaning something outside of a latent space which can host the concept of two. This is not to say that symbolic reasoning isn’t powerful, it is arguably the most powerful subset of the human latent space. As Alan Kay points out, the alternative to it is doing a guess and check binary search over whole designs, and keeping these giant empirical libraries of things that did and didn’t work. You know, the ancient Sumerians didn’t have math but they still built complex structures, they did it by basically hill climbing and basing new structures on a learned distribution of old structures to kind of intuit which structures would be likely to stand or not. You can make things that way, but it’s not nearly as efficient as math. But math coheres out of that, it’s the regular patterns/structure you get from this if you insist on searching for the invariants, it’s not like, some separate thing. If this is confusing, consider that like, the fundamental epistemic foundations of mathematics eventually bottom out in clustering operations we can do extremely reliably. Bertrand Russel famously spends 379 pages to prove that 1 + 1 = 2, and the proof is famously wrong. The way you get math is by finding a subset of clustering operations that are so reliable they can be used to define invariants and then repurposing the human in-context reward machinery to assign value to the outcome of algorithms defined in terms of these foundational clustering operations.
Which is also, basically, the way that LLMs are doing math. They use external proofs and solvers to sharpen their neural circuits until they’re reliable enough to do increasingly complex mathematical operations mentally without external checking. In humans this is accomplished by pretraining a world model and a little bit of inductive bias (I believe 1, 2, and 3 actually have dedicated neural hardware to make sure a human learns them, or something?) where you can find a reliable enough implementation of the arithmetic equality operator that you can say “1 + 1 = 2”, and then you invent algorithms for arithmetic which you use as tools to sculpt your neural circuits with in-context reward until more operations become reliable enough to increase your inferential reach and therefore the library of potential algorithms that can be executed using concepts in your latent space.
Fundamentally what I’m saying is something like the natural selection argument. It’s not that natural selection is an efficient optimizer, or a particularly good way to make things, but it’s the first thing that makes things which make things, it is the mechanism by which matter becomes self organizing into life. In the same way it’s not that doing gradient search to find a giant matrix which extrapolates new thoughts based on a learned distribution of existing thoughts and observations is the most efficient way to think, but it is the way that the things which make other forms of thought possible get built. You can’t have useful symbolic reasoning without it.
It’s a bit like the argument that say, surely capitalism is inefficient since an intelligent optimizer can just directly specify the things that should be built. Why do we need to have wasteful distributed intelligence when we can have efficient centralized intelligence? And most of contemporary economics is just various ways to try and explain to people that as intuitive as this idea is it does not work and much of the complexity that a central designer has to deal with is actually irreducible and most efficiently managed by distributed design. Neural nets are a similar thing. The things you need to build to do the feature extraction that makes the application of algorithms possible have irreducible complexity that requires you to do something like gradient search in a high k-complexity program space. There is no simple trick to overcome this, the closer your method gets to the convergent shape of gradient search over generating programs of the surprising amount of detail in reality the more neural-net like it becomes in the same way that the more closely your economic system gets to the convergent shape of distributed intelligence doing price discovery the more it begins to resemble Neoliberalism, which is why the Soviet Union was ideologically doomed in a way that the modern Chinese government has been able to escape by simply establishing markets and regulating them.
I’ve been saying for some time (out loud to my friends, anyway) that deduction is just a thousand inductions in a trenchcoat. I take this to be true in an abstract epistemological sense, but perhaps more importantly in the sense of the physical operation of our brains. It’s obvious to me that the human brain doesn’t contain a formal logic module. I don’t see why we should expect AI to be any different (aside from “it’s in a computer, and I think I know what computers are”).
This isn’t the main claim in your argument (I recall this past comment where you touched on it more extensively), but I’m not sure it holds up on its own. I don’t think the fact that bigger/more complex NNs perform better supports “the built artifact is irreducibly complex,” because that same fact is (at least as) consistent with an opposite stance: increasing scale mostly buys you a bigger search space, where the soft inductive biases induced by that scale have more flexibility to find simple solutions that still fit the data. More scale doesn’t necessarily mean more complex solutions! (This is one of the core lines of Andrew Gordon Wilson’s work, trying to explain why deep learning doesn’t land us in overfitting hell.) So increasing scale can get us closer to an optimal solution even if the size of that solution is relatively small and fixed.
We see a version of this in the lottery ticket hypothesis, which states that trained NNs contain a much sparser subnetwork that matches or exceeds the performance of the full NN. It’s reasonable to interpret this as implying that much of the benefit of scaling NNs comes from the combinatorial explosion of sparse subnetworks that gradient descent can then utilize. Under this framework, scale mostly buys you “more shots at the goal,” even if the goal is quite simple, i.e., better odds at finding a good subnetwork, not a “more accurate/complex” solution.
This would seem to predict that trained neural networks can be massively compressed, with the fraction you can compress away rising with scale. This doesn’t seem to be the case. Well-trained LLMs have some unused parameters, but not like,
The LTH gives us a way to do this! It’s just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that “larger models have better generalization bounds and are more compressible than smaller models.”
I don’t think the LTH applies to large language models. Or, to be more quantitative, I don’t think you can prune
I haven’t read these particular papers, but my first reaction is skepticism. The first paper seems like a prime candidate for failure mode 1. In addition, on a skim, I am pretty skeptical of their effective dimensionality calculations. Text says
Which seems insane to me. Top 100 eigenvalues for nets with between 1e5 and 1.6e6 parameters trained on CIFAR-100??? If that’s not impacting their effective dimensionality calculation it’s got to have a crazy high cutoff, surely you are not getting good performance on CIFAR-100 with 100 Hessian eigenmodes, what the hell. This seems like it’s basically throwing away the signal effective dimensionality is supposed to measure.
Second paper I haven’t understood yet.
EDIT: I have now read the second paper a little more. They seemingly don’t even try to compress regular LLMs. They compress models trained with their weird SubLoRA technique which seem to score much worse loss than regular models on text prediction. Like, these look barely better than random in table 2. And the generated text from them in Appendix F looks like word salad. So this paper seems pretty irrelevant to the question at hand.
Basically, neural network priors are pretty good if they’re trained well, and they mostly don’t waste capacity. If you’ve got a nice general algorithm that the net can learn using only
I kinda agree, but it also could be smeared all over the NN, such that naively dropping some weights or something does not work, as you are pocking holes under wrong representation.
People haven’t just tried dropping weights in the neuron basis. They’ve e.g. searched out low eigenmodes of the Hessian and dropped those.
Okay but the argument for it being irreducibly complex is actually “think about this for five minutes without the assumption that it should actually be simple, once you do this is overwhelmingly obvious”. Like, the world has a lot of detail in it and the detail is not actually something you can skip over if you want to predict the next token in our environment.
I think part of the disagreement is just about the relative importance and necessity of different kinds of knowledge. There’s two basic kinds of knowledge which we usually conflate. One is something like math: Highly compressible, convergently accessible from reason, fundamental physical phenomenon etc.
The other kind is something like biology or history: Endlessly detailed, primarily reliant on memorization with limited generalization, if you delete parts of it they don’t easily grow back. Essentially irreducible blobs of Shannon entropy.
A lot of Yudkowsky’s original argument for AI doom is that he thinks the kind of fundamental physics knowledge that would make AI dangerous isn’t all that high complexity in the first place, humans are just dumb:
Most of the bits of information the neural net is spending capacity on are presumably details of our particular worldline/history more than they are fundamental reasoning or physics knowledge. Maybe quantum gravity is already sitting in the weights of a frontier GPT pretrain and we just don’t know how to get it out.
However I continue to be puzzled by what Yudkowsky thinks a “bit” of knowledge is in this context:
This is a tautology. The theoretical limit is that every time you see 1 additional bit, which is usually literally defined as the base two logarithm of our hypothesis space, you can throw out half the hypothesis space. This is true but tells you nothing about how much information constitutes a “bit” of that hypothesis space. Like, we literally define a bit as that amount of information which would let us throw our half the hypothesis space. A bit in the sense of the fundamental entropy you need to store the frames of a falling apple and a bit in the sense of throwing out half the hypothesis space in our search over fundamental physical theories does not actually refer to the same unit of information, “it takes this many bytes to store a visual representation of this phenomenon in a general format for storing arbitrary bitmaps” is not the same thing as the k-complexity. Presumably his argument is something like general relativity can be described by a not particularly large program and therefore it should be easy to find with approximations of Solomonoff induction but I’m not convinced this is actually the case?
What always stands out to me about video nets is how janky their understanding of the standard model seems to be. They frequently generate photorealistic frames that violate the standard model in implausible and bizarre ways, which is strange because the k-complexity of the invariants in the standard model, as opposed to the specific arrangements of atoms inside it, should in fact be small enough that I would expect video models to pick up on Newtonian physics quickly and at small capacity in the same way that GPT models pick up on English grammar well before they gain the kind of world model that lets them produce coherent text with the English grammar. TinyStories shows you can fit a functional model of English grammar in under ten million parameters, and I don’t expect the standard model to actually be that much more complex than English grammar. That GPT in fact does so in the case of English grammar but video models don’t do so in the case of the standard model tells me I’m somehow either mistaken about the nature of search in the hypothesis space the standard model lives in or I’m confused about the generalization properties of neural nets, or both.
Nitpick: I think pretraining is better analogised to search in a high complexity circuit space. You only get one forward pass worth of serial depth per data point, so doing the same operation many times in a row costs a lot of description length. Post-training RL is where this transitions into something more like search in a high k-complexity program space.[1]
Compute-bounded k-complexity, with the bound coming the length of the CoT.