Okay but the argument for it being irreducibly complex is actually “think about this for five minutes without the assumption that it should actually be simple, once you do this is overwhelmingly obvious”. Like, the world has a lot of detail in it and the detail is not actually something you can skip over if you want to predict the next token in our environment.
I think part of the disagreement is just about the relative importance and necessity of different kinds of knowledge. There’s two basic kinds of knowledge which we usually conflate. One is something like math: Highly compressible, convergently accessible from reason, fundamental physical phenomenon etc.
The other kind is something like biology or history: Endlessly detailed, primarily reliant on memorization with limited generalization, if you delete parts of it they don’t easily grow back. Essentially irreducible blobs of Shannon entropy.
A lot of Yudkowsky’s original argument for AI doom is that he thinks the kind of fundamental physics knowledge that would make AI dangerous isn’t all that high complexity in the first place, humans are just dumb:
That even Einstein did not come within a million light-years of making efficient use of sensory data.
Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
Most of the bits of information the neural net is spending capacity on are presumably details of our particular worldline/history more than they are fundamental reasoning or physics knowledge. Maybe quantum gravity is already sitting in the weights of a frontier GPT pretrain and we just don’t know how to get it out.
However I continue to be puzzled by what Yudkowsky thinks a “bit” of knowledge is in this context:
I occasionally run into people who say something like, “There’s a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data.”
Yes. There is. The theoretical limit is that every time you see 1 additional bit, it cannot be expected to eliminate more than half of the remaining hypotheses (half the remaining probability mass, rather). And that a redundant message, cannot convey more information than the compressed version of itself. Nor can a bit convey any information about a quantity, with which it has correlation exactly zero, across the probable worlds you imagine.
This is a tautology. The theoretical limit is that every time you see 1 additional bit, which is usually literally defined as the base two logarithm of our hypothesis space, you can throw out half the hypothesis space. This is true but tells you nothing about how much information constitutes a “bit” of that hypothesis space. Like, we literally define a bit as that amount of information which would let us throw our half the hypothesis space. A bit in the sense of the fundamental entropy you need to store the frames of a falling apple and a bit in the sense of throwing out half the hypothesis space in our search over fundamental physical theories does not actually refer to the same unit of information, “it takes this many bytes to store a visual representation of this phenomenon in a general format for storing arbitrary bitmaps” is not the same thing as the k-complexity. Presumably his argument is something like general relativity can be described by a not particularly large program and therefore it should be easy to find with approximations of Solomonoff induction but I’m not convinced this is actually the case?
What always stands out to me about video nets is how janky their understanding of the standard model seems to be. They frequently generate photorealistic frames that violate the standard model in implausible and bizarre ways, which is strange because the k-complexity of the invariants in the standard model, as opposed to the specific arrangements of atoms inside it, should in fact be small enough that I would expect video models to pick up on Newtonian physics quickly and at small capacity in the same way that GPT models pick up on English grammar well before they gain the kind of world model that lets them produce coherent text with the English grammar. TinyStories shows you can fit a functional model of English grammar in under ten million parameters, and I don’t expect the standard model to actually be that much more complex than English grammar. That GPT in fact does so in the case of English grammar but video models don’t do so in the case of the standard model tells me I’m somehow either mistaken about the nature of search in the hypothesis space the standard model lives in or I’m confused about the generalization properties of neural nets, or both.
Okay but the argument for it being irreducibly complex is actually “think about this for five minutes without the assumption that it should actually be simple, once you do this is overwhelmingly obvious”. Like, the world has a lot of detail in it and the detail is not actually something you can skip over if you want to predict the next token in our environment.
I think part of the disagreement is just about the relative importance and necessity of different kinds of knowledge. There’s two basic kinds of knowledge which we usually conflate. One is something like math: Highly compressible, convergently accessible from reason, fundamental physical phenomenon etc.
The other kind is something like biology or history: Endlessly detailed, primarily reliant on memorization with limited generalization, if you delete parts of it they don’t easily grow back. Essentially irreducible blobs of Shannon entropy.
A lot of Yudkowsky’s original argument for AI doom is that he thinks the kind of fundamental physics knowledge that would make AI dangerous isn’t all that high complexity in the first place, humans are just dumb:
Most of the bits of information the neural net is spending capacity on are presumably details of our particular worldline/history more than they are fundamental reasoning or physics knowledge. Maybe quantum gravity is already sitting in the weights of a frontier GPT pretrain and we just don’t know how to get it out.
However I continue to be puzzled by what Yudkowsky thinks a “bit” of knowledge is in this context:
This is a tautology. The theoretical limit is that every time you see 1 additional bit, which is usually literally defined as the base two logarithm of our hypothesis space, you can throw out half the hypothesis space. This is true but tells you nothing about how much information constitutes a “bit” of that hypothesis space. Like, we literally define a bit as that amount of information which would let us throw our half the hypothesis space. A bit in the sense of the fundamental entropy you need to store the frames of a falling apple and a bit in the sense of throwing out half the hypothesis space in our search over fundamental physical theories does not actually refer to the same unit of information, “it takes this many bytes to store a visual representation of this phenomenon in a general format for storing arbitrary bitmaps” is not the same thing as the k-complexity. Presumably his argument is something like general relativity can be described by a not particularly large program and therefore it should be easy to find with approximations of Solomonoff induction but I’m not convinced this is actually the case?
What always stands out to me about video nets is how janky their understanding of the standard model seems to be. They frequently generate photorealistic frames that violate the standard model in implausible and bizarre ways, which is strange because the k-complexity of the invariants in the standard model, as opposed to the specific arrangements of atoms inside it, should in fact be small enough that I would expect video models to pick up on Newtonian physics quickly and at small capacity in the same way that GPT models pick up on English grammar well before they gain the kind of world model that lets them produce coherent text with the English grammar. TinyStories shows you can fit a functional model of English grammar in under ten million parameters, and I don’t expect the standard model to actually be that much more complex than English grammar. That GPT in fact does so in the case of English grammar but video models don’t do so in the case of the standard model tells me I’m somehow either mistaken about the nature of search in the hypothesis space the standard model lives in or I’m confused about the generalization properties of neural nets, or both.