I’m wrote a book about why all knowledge of the truth is fundamentally uncertain. Give it a read!
I’ve also written a lot about AI safety, personal development, and Buddhism, among other things. Most of that’s here on LessWrong, but there’s also some on my blog, Uncertain Updates.
For what it’s worth, I liked this quite a bit, even if it’s not a “typical” LessWrong post.
Math exists in the map, not the territory. Math is a map to make sense what we experience (I phrase it this way to avoid making excess metaphysical commitments about the nature of what’s experienced). It’s an abstraction of symbols, and it’s useful to the extent it accurately models and predict what we experience. Modeling is an active task. When there’s no mind, there’s no modeling happen, and hence no math. There may well still be things happening that could be modeled by math and you could argue that the abstractions of math are latent, but they don’t meaningfully exist, as far as we know, when we’re not mathing.
If they are being applied to pure mathematics they are being used to do mathematics. Math, in an important sense, doesn’t exist when it’s not being done.
I think there’s something to this. Fundamentally, the base models don’t seem to be getting much better at coding continuously, except for occasional jumps that happen maybe once a year. What’s been getting better steadily are the harnesses, which are getting rapidly better at directing those base skills. This results in making the base model much more usable over time for coding, but also more quickly exposes its limits. 20-30% mergable on the first try actually feels right to me, based on my usage.
As far as I know it’s an open question of how LLMs learn language. It’s clearly different from humans because they don’t have the same kind of learning-from-sense-experience process that we do to ground the meaning of words defined in terms of other words. It’s possible they don’t learn words the same way we do and the sense in which an LLM gives a word meaning is different from how a human does it, and maybe that happens in such a way as to be meaningful even if humans and LLMs can communicate and each experiences the other as producing words that they interpret as grounded in the way they ground meaning.
As for mathematical concepts made out of relationships to each other, the story from the human perspective is the same: grounded in experience with words that back up those other words. When we try to do mathematics where everything floats free, it’s possible, but those symbols seem to come to take on meaning only because they get grounded by how they are used, which, arguably, is what humans do, so maybe that’s what LLMs do, too, only they don’t have the point-at-a-thing-to-know-it operation, so their use of words have meaning grounded in use, but not use to describe sensory experience.
Confused by this weird multi-reply comment, but I’ll address just the part to me:
You can’t bottom out all the concepts in experience , because you cant experience infinities or imaginary numbers. That is not an argument for Platonism, because a concept can just be a concept. In terms of the Fregean sense/reference distinction—one of the useful things you can learn from philosophy—there doesn’t have to be a reference , physical or Platonic.
Both infinities and imaginary numbers are artifacts of the model. That is, they are things we know about only because the map says they are there. This is both historically true (both things were discovered because the theory had a place where they should be) and true on the model itself (e.g. any experience of boundlessness I have is necessary a bounded experience that can only partly capture the infinite, supposing the infinite is even something we can meaningfully talk about as existing to experience).
And yes, there doesn’t have to be a direct reference. You can just make up concepts. But where did you make up that concept from? As I argue here, it comes ultimately from experience, because all concepts are grounded in experience via ostensive concept construction, and then we can build concepts on concepts, but those concepts not directly tied to experience are indirectly tied through their relationship with other concepts that are.
To get to that point, something like 8 years and about 5000 hours of meditation. But there’s lots of variation here, so I don’t know exactly where I fall in the distribution. My best guess is 40th percentile (i.e. 60% of people would have needed more years/hours than me to reach the same point).
Aside from getting into rock climbing for years and finding this fun, I over-solved the problem by meditating a lot and woke up enough to find merely existing to be pretty pleasant. Then exercise was not so bad, because it’s also just existing.
Not really a targeted solution but does work.
Therefore, there is nothing weird about mathematical efficiency. Nor any incompatibility between math and empiricism or math and physicalism. We do not need the natural laws of the world to be “written in the language of mathematics”. It’s enough, that we can speak math ourselves.
Indeed. I think the reason people get caught on questions like “what is math?” (and “what is reality/morality/consciousness/etc.?”) largely comes from what I might call “metaphysical confusion”. It takes the form of feeling like there’s some meta question to be answered because they struggle to keep the whole picture in their head as they reason, and so it seems like concepts must bottom out somewhere other than experience because when they look at experience they struggle to see how it’s enough.
But experience is enough, and there’s no reason to do metaphysics, in that if a question is a question of metaphysics, we can generally reason equally well with and without the metaphysical model, except insofar as a metaphysical theory might provide useful intuitions to power counterfactual thinking to extend non-metaphysical theory, or in that we need some implicit metaphysics to say anything at all.
Something I’ve not written up but want to push on is the idea that LLM reasoning is ever legible at all (or put another way, it’s illegible even when it seems legible to us). I think we don’t have super strong evidence that the appearance of legibility in CoT really means that’s how the LLM is thinking. We already see some evidence of Goodharting on CoT to satisfy training, and in the long run we should expect something like steganographic CoT that appears legible to us but is actually a channel for conveying illegible reasoning. I suspect we are mostly deluding ourselves if we believe seeming legibility is real, in which case I think your concerns about illegible reasoning are all the more pressing!
You do not need to be a socialist to observe that there is often misalignment between the most well-paid jobs and the work that is the most important to do.
“Important” by whose lights? I’m not saying our market economy always incentivizes the work I think is most important, but that’s relative to my opinion about what’s most important. I bristle a bit when I see sentences like this because they smuggle in an assumption that it’s obvious what’s most important to do. In fact, the market is providing some measure of importance, just as my opinion is, and it’s not clear if the market, my opinion, someone else’s opinion, or the aggregate opinion of the populace (however that might be measured in some meaningful way that’s different from the aggregation the market already provides) will, on reflection, result in better outcomes for what I and others care about.
I do think there’s some innovation on notation, but it mostly happens with existing typographic symbols because extending typography is harder than it used to be. Previously, you could just come up with whatever you wanted because work started out hand written. Then you’d pay to get the printer to make whatever weird symbol you wanted for publication, or, if on a budget, come up with some weird approximation using simpler symbols.
It seems like it should be easier on computers, and in theory it is, but lots of things drive us towards making default choices. The worst of these is probably that Unicode is already full of some many symbols that LaTeX can render, so it’s much easier to just pick some existing symbol rather than try to go through all the work of cooking up a new one.
I separately hope that there’s some effect from computer code, too, where people are trending towards favoring longer symbols that more resemble descriptive function and variables names, which feel less like notation but are easier to read on first glance, even if they bear costs in efficiency once familiar with them.
That’s good to know! I think what’s tricky is that 2s really need to receive unconditional love in that it’s the thing they want and also the thing they think they don’t deserve. Most of my experience suggests practices tend to focus a lot on getting the practitioner to feel unconditional love towards others. Yes, there’s an expectation that others includes self, but this is often specifically where people get tripped up with, say, metta practice, and aren’t really given tools to solve it other than “keep going, you’ll start to feel love for yourself”.
I could imagine a Buddhism that, say, made a big deal of how Avalokiteshvara personally loves you unconditionally would be good ground for a 2s practice to begin.
(Also, not stated above, but the epistemics around what a 2 needs are really fraught! This is something I also don’t know how to solve.)
I agree with you on the value on schmoozing. Alas, I think it’s going to be hard to convince many LessWrong readers of the value. Many see it as not merely some annoying thing, but as actively evil, a source of badness in the world. True, I think they take this view because they don’t understand it and are unskilled at it, so on their current values it’s clearly negative, even if they would perhaps change their mind if they understood its function better, were better at doing it, and more saw the benefits of schmoozing personally.
I’m not really sure how to fix this, but thank you for thinking and writing about it!
If you’re not conceptualizing, you’re not doing philosophy, you’re experiencing things. Philosophy is about conceptualizing experience and thereby making sense of it. You can work with those concepts without putting linguistic labels on them, but even so such concepts are still “words” in the sense that “words” here is just a gloss of talking about symbolic referents of any kind.
The difference is that, when we do, say, physics or mathematics, we take for granted the relationship between symbols and their referents and assume that relationship to be sound. This leaves only the question of how to define basic terms, but once those terms are defined, the rest can proceed via formalism and application of formalism to observations.
I want to be clear that I think this is an extremely reasonable thing to do! It would be very hard to do, say, geometry, if we were constantly debating what points and lines are. Instead, we create formal definitions, assume them to be true, and move forward. Similarly, in physics, though the process is a bit more complex, we infer theories from observations, those theories posit the world to be categorized in some way, and then proceed to draw further conclusions on the assumption a theory holds insofar as it provides a meaningly useful ontology to make sense of observations.
Philosophy has a different challenge, in that it includes within it to scope of study the action of creating categories. That is, it must use categories to make sense of categories, not in the way we do in math, but in the messy way that minds actually categorize the world. This sets it apart from sciences in that it must tackle the question of how we use concepts at all, and it must do so all while reasoning with concepts, which leads to a number of weird problems that break formalisms.
Philosophy is not, exceptionally among sciences, concerned with words or concepts.
I think this is not true, insofar as philosophy is done with words, and so can’t excise itself from the world of conceptualization and still be philosophy. This doesn’t mean philosophy must only or always be about words and concepts, but that philosophy can never ignore that it’s attempting to understand the world through words and concepts, and as such that understanding may the tainted by those words.
None of this is revolutionary, and maybe Williamson is aware of it, but seems he has reason to ignore this point, because I think it undermines his overall argument, since sciences work by taking a certain amount of conceptualization for granted, and philosophy is different in that conceptualization is always within the scope of study.
Yeah I agree it the idea of social constructs very much goes against the vibe of logical positivism, which is often what matters when talking with people who have it as their worldview. Logical positivism has long been shown unworkable, but many still cling to naive, intuitive versions of it, so I very much understand the impulse to push back on it (I’ve done the same myself many times!).
I wish we had a good name for the kind of strawman rationality that includes naive logical positivism that wasn’t “strawman rationality” or “straw rationality” because it’s not a fake position designed to be knocked down, but a real set of beliefs people hold that have to be argued against (especially among many people who are part of the rationalist scene).
Indeed. The map-territory metaphor has its value, but it also creates confusion.
I was just thinking about how this pattern might apply to software engineering, and I’m starting to suspect that it largely doesn’t.
Here’s my thinking. I use AI a lot to do things like brainstorm solutions. Rather than me sitting around trying to think of ways to solve a problem, I describe the problem to an AI, and get it to give me ways it thinks the problem might be solved. Now, I don’t always take one of its options, but the process of asking it is usually enough to get me to think of how I want to solve the problem, and sometimes I get lucky and it proposes the right thing to do.
This seems analogous on the surface to just making the moves the AI suggests, but in practice I think it’s not, because we’ve been doing a similar move in software engineer for years as a way of learning. It’s pretty normal, historically, to ask a more senior engineer to figure out how to solve a problem, and then a more junior engineer implements it. Part of the point of this, beyond specialization, is that the junior learns via imitation from the senior. This admittedly doesn’t always work, but the approach is structurally the same as playing with a Go AI assistant, and it seems perhaps a lot could be learned about software engineering by using an AI to help bootstrap oneself towards making good decisions and having good judgement.