The first bullet point seems valid(for propositions with no empirical content)
interstice
Karma: 622
Advancing Mathematics By Guiding Human Intuition With AI
Do you have reason to believe we will never collect surprising observations?
Sure, it’s likely we’ll get some new surprising observations before we nail down the TOE. The question is just about how surprising, and whether they will let us upend physical limits. Agreed that there’s a lot of new interesting things we could observe, but for most of your examples, I don’t think we have good reason to think that we’ll learn new things about fundamental physics from them.
Let me rephrase my objection. I think my main issue with your post can be found in this phrase near the beginning: you speak of the “rate at which we have constantly upended our own physical theories”. I don’t think that progress in fundamental physics is like technological progress or other things which happen at a steady rate per unit effort. It’s more like exploiting a non-renewable resource: our ignorance of physical phenomena. So 400 years ago we basically started with a huge ‘reservoir’ of ignorance, which has gradually been drained as our theories improved, until now there’s only a few small pools left that we can see. The reason that we’ve seen steady progress until recently is due to our slow draining of this reservoir, so now that it’s mostly gone, we no longer have a reason to expect further such steady progress. It’s possible that we might find new reservoirs someday, but equally possible that we won’t, so it’s a reasonable assumption that many of our current theories’ physical limits will continue to apply indefinitely into the future.
There’s a big difference between our current state of knowledge regarding physics and previous eras’: we now completely understand the physics of everyday existence. In the past, there were many, blatantly obvious unknowns—e.g. Newtonian mechanics doesn’t let you understand how chemistry arises from physics. Nowadays there’s a lot less room for reality to surprise us with new observations—indeed, theoretical physics has largely stalled out in recent years due to the infeasibility of obtaining observations our theories don’t already predict. More generally, this seems to be what we should expect to happen in a lawful universe: after an initial period of discovery, we eventually discover all the laws and are done. What you propose—an endless string of new discoveries, each upending the last—is incompatible with the universe having a finite description. It’s not logically impossible that we live in such a universe, but scientific progress so far seems to support finite lawfulness.
It seems to me like there should be an infinite number of ways to interpret atoms’ vibrations as having information or even being transformed in a way that approximates the operation of an algorithm. I don’t know if any SIC campers actually worry about this, if they have specific requirements on how to tell if atoms are running algorithms, how likely they think this is to happen in practice, etc.
People do indeed worry about this, leading to things like ‘Solomonoff-weighted utilitarianism’ that assign higher moral relevance to minds with short description lengths.
Before seeing any evidence, we should indeed expect that life has high density in the universe. We just have enough data to rule that out. More generally I think UDASSA is probably the best framework for approaching problems like this, and it would hold that, in situations where our existence is contingent on an anthropically-selected unlikely event, we should still expect that this event is as likely as possible while being consistent with the evidence. So 10^-40 likelihood origination events more probably than 10^-400 likelihood events.
There’s some discussion of this in a followup post.
What seems off to me is the idea that the ‘player’ is some sort of super-powerful incomprehensible lovecraftian optimizer. I think it’s more apt to think of it as like a monkey, but a monkey which happens to share your body and have write access to the deepest patterns of your thought and feeling(see Steven Byrnes’ posts for the best existing articulation of this view). It’s just a monkey, its desires aren’t totally alien and I think it’s quite possible for one’s conscious mind to develop a reasonably good idea of what it wants. That the OP prefers to push the ‘alien/lovecraftian’ framing is interesting and perhaps indicates that they find what their monkey (and/or other peoples’ monkeys) wants repulsive in some way.
In rationalist circles, you might find out that you’re being instrumentally or epistemically irrational in the course of a debate—the norms of such a debate encourage you to rebut your opponent’s points if you think they are being unfair. In contrast, the central thesis of this book is that white people disputing their racism is a mechanism for protecting white supremacy and needs to be unlearned, along with other cornerstones of collective epistemology such as the notion of objective knowledge. So under the epistemic conditions promoted by this book, I expect “found about being racist” to roughly translate to “was told you were racist”.
I think those advancements could be evidence for both, depending on the details of how the nootropics work, etc. But it still seems worth distinguishing the two things conceptually. My objection in both cases is that only a small part of the evidence for the first comes from the causal impact of the second: i.e. if Codex gave crazy huge productivity improvements, I would consider that evidence for full code automation coming soon, but that’s mostly because it suggests that Codex can likely be improved to the point of FCA, not because it will make OpenAI’s progammers more productive.
Regarding your first point, I think when people say that language models “don’t bring us closer to full code automation” they mean there’s no way of improving/upgrading language models such that they implement full code automation. I think it would be better to argue against that claim directly instead of bringing up language model’s productivity-boosting effects. There are many things that could potentially boost programmers’ productivity—better nootropics, say—but it seems overly broad to say that they all “bring us closer to full code automation”, even if it might be causally true that they reduce the time to automation in expectation.
I actually agree with you there, there was always discussion of GCR along with extinction risks(though I think Eliezer in particular was more focused on extinction risks). However, they’re still distinct categories: even the deadliest of pandemics is unlikely to cause extinction.
Killing 90% of the human population would not be enough to cause extinction. That would put us at a population of 800 million, higher than the population in 1700.
It could be considered an essence, but physical rather than metaphysical.
This feels related to metaphilosophy. In the sense that, (to me) it seems that one of the core difficulties of metaphilosophy is that in coming up with a ‘model’ agent you need to create an agent that is not only capable of thinking about its own structure, but capable of being confused about what that structure is(and presumably, of becoming un-confused). Bayesian etc. approaches can model agents being confused about object-level things, but it’s hard to even imagine what a model of an agent confused about ontology would look like.
Another example of this sort of thing: least-rattling feedback in driven systems.
Perhaps this is a physicist vs mathematician type of thinking though
Good guess ;)
This is not the same as saying that an extremely wide trained-by-random-sampling neural network would not learn features—there is a possibility that the first time you reach 100% training accuracy corresponds to effectively randomly initialised initial layers + trained last layer, but in expectation all the layers should be distinct from an entirely random intialisation.
I see—so you’re saying that even though the distribution of output functions learned by an infinitely-wide randomly-sampled net is unchanged if you freeze everything but the last layer, the distribution of intermediate functions might change. If true, this would mean that feature learning and inductive bias are ‘uncoupled’ for infinite-width randomly-sampled nets. I think this is false, however—that is, I think it’s provable that the distribution of intermediate functions does not change in the infinite-width limit when you condition on the training data, even when conditioning over all layers. I can’t find a reference offhand though, I’ll report back if I find anything resolving this one way or another.
The claim I am making is that the reason why feature learning is good is not because it improves inductive bias—it is because it allows the network to be compressed. That is probably at the core of our disagreement.
Yes, I think so. Let’s go over the ‘thin network’ example—we want to learn some function which can be represented by a thin network. But let’s say a randomly-initialized thin network’s intermediate functions won’t be able to fit the function—that is (with high probability over the random initialization) we won’t be able to fit the function just by changing the parameters of the last layer. It seems there are a few ways we can alter the network to make fitting possible:
(A) Expand the network’s width until (with high probability) it’s possible to fit the function by only altering the last layer
(B) Keeping the width the same, re-sample the parameters in all layers until we find a setting that can fit the function
(C) Keeping the width the same, train the network with SGD
By hypothesis, all three methods will let us fit the target function. You seem to be saying[I think, correct me if I’m wrong] that all three methods should have the same inductive bias as well. I just don’t see any reason this should be the case—on the face of it, I would guess that all three have different inductive biases(though A and B might be similar). They’re clearly different in some respects -- (C) can do transfer learning but (A) cannot(B is unclear).
What do we know about SGD-trained nets that suggests this?
My intuition here is that SGD-trained nets can learn functions non-linearly while NTK/GP can only do so linearly. So in the car detector example, SGD is able to develop a neuron detecting cars through some as-yet unclear ‘feature learning’ mechanism. The NTK/GP can do so as well, sort of, since they’re universal function approximators. However, the way they do this is by taking a giant linear combination of random functions which is able to function identically to a car detector on the data points given. It seems like this might be more fragile/generalize worse than the neurons produced by SGD. Though that is admittedly somewhat conjectural at this stage, since we don’t really have a great understanding of how feature learning in SGD works.
I’ve read the new feature learning paper! We’re big fans of his work, although again I don’t think it contradicts anything I’ve just said.
ETA: Let me elaborate upon what I see as the significance of the ‘feature learning in infinite nets’ paper. We know that NNGP/NTK models can’t learn features, but SGD can: I think this provides strong evidence that they are learning using different mechanisms, and likely have substantially different inductive biases. The question is whether randomly sampled finite nets can learn features as well. Since they are equivalent to NNGP/NTK at infinite width, any feature learning they do can only come from finiteness. In contrast, in the case of SGD, it’s possible to do feature learning even in the infinite-width limit. This suggests that even if randomly-sampled finite nets can do feature learning, the mechanism by which they do so is different from SGD, and hence their inductive bias is likely to be different as well.
First thank you for your comments and observations—it’s always interesting to read pushback
And thanks for engaging with my random blog comments! TBC, I think you guys are definitely on the right track in trying to relate SGD to function simplicity, and the empirical work you’ve done fleshing out that picture is great. I just think it could be even better if it was based around a better SGD scaling limit ;)
Therefore, if an optimiser samples functions proportional to their volume, you won’t get any difference in performance if you learn features (optimise the whole network) or do not learn features (randomly initialise and freeze all but the last layer and then train just the last).
Right, this is an even better argument that NNGPs/random-sampled nets don’t learn features.
Given therefore that the posteriors are the same, it implies that feature learning is not aiding inductive bias—rather, feature learning is important for expressivity reasons
I think this only applies to NNGP/random-sampled nets, not SGD-trained nets. To apply to SGD-trained nets, you’d need to show that the new features learned by SGD have the same distribution as the features found in an infinitely-wide random net, but I don’t think this is the case. By illustration, some SGD-trained nets can develop expressive neurons like ‘car detector’, enabling them to fit the data with a relatively small number of such neurons. If you used an NNGP to learn the same thing, you wouldn’t get a single ‘car detector’ neuron, but rather some huge linear combination of high-frequency features that can approximate the cars seen in the dataset. I think this would probably generalize worse than the network with an actual ‘car detector’(this isn’t empirical evidence of course, but I think what we know about SGD-trained nets and the NNGP strongly suggests a picture like this)
Furthermore (and on a slightly different note), it is known that infintesimal GD converges to the Boltzmann distribution for any DNN (very similar to random sampling)
Interesting, haven’t seen this before. Just skimming the paper, it sounds like the very small learning rate + added white noise might result in different limiting behavior from usual SGD. Generally it seems that there are a lot of different possible limits one can take; empirically SGD-trained nets do seem to have ‘feature learning’ so I’m skeptical of limits that don’t have that(I assume they don’t have them for theoretical reasons, anyway. Would be interesting to actually examine the features found in networks trained like this, and to see if they can do transfer learning at all) re:‘colored noise’, not sure to what extent this matters. I think a more likely source of discrepancy is the lack of white noise in normal training(I guess this counts as ‘colored noise’ in a sense) and the larger learning rate.
if anyone can point out why this line of argument is not correct, or can steelman a case for SGD inductive bias appearing at larger scales, I would be very interested to hear it.
Not to be a broken record, but I strongly recommend checking out Greg Yang’s work. He clearly shows that there exist infinite-width limits of SGD that can do feature/transfer learning.
The ‘grand story’ Eliezer is referring to here isn’t anything like these, though. That story is more like “there is a gradual increase in capability in all species, on an slow timescale; eventually one of them crosses the threshold of being able to produce culture which evolves on a faster timescale”. Sort of the opposite of these species-parochialist tales.