Something I notice as I read this, that I think has made it a bit harder to grok this sequence:
This post says “I’m a naturalist”, and then lists a bunch of examples of things that are, well, natural. Newts and dirt and sunshine and stars and stuff. I do aesthetically like the idea of getting really innately patiently curious about those things. But that’s..
a) far removed from my current habits,
b) getting persistently patiently curious about those things just feels like a really inefficient way to make progress on the stuff that I’m actually trying to figure out,
c) perhaps most importantly, I get the sense that the spirit of the thing you’re trying to convey here isn’t limited to newts and dirt and sunshine and mushrooms. It should also apply to the various worlds I currently “live” in. Some things are tightly analogous to mushrooms/dirt/sunshine (there is in fact literal dirt and literal sunshine and flowers and animals on the streets of Berkeley, and other concrete things to get curious about like stop signs and pavement cracks). But it seems like the mindset of details/direct observation should also apply to javascript, and machine learning. It’s a bit interesting that in some sense a javascript codebase is “a map”, rather than “a territory.” But it’s also in some sense a “territory”. What about abstract math?
So I feel like I could use more poetry that conveys how this applies to the latter set of things. (I think you have actual background in math so could probably paint poetry yourself about that?). For that matter, just poetry about multiple sets of things you could excitedly patiently observe, that don’t have much to do with each other, to help triangulate the-part-that-you’re-ultimately-getting-at, rather than one aesthetic that (I think?) is fairly incidental.
I’m also definitely confused about how to apply all of this to research into things that don’t exist yet. (I’m not sure if this sequence is supposed to bridge to that yet).
I do get a takeaway of “okay Ray, in addition to trying to Think Hard About AI Alignment Research, you should be doing things that a) give yourself space from that so you brain has room to do various other brain things that feed into that in subtler ways, b) adopt a patient observer mindset that applies to lots things you’re interacting with as you contemplate the state of AI Alignment Research.”
But I have some sense that, ultimately, the mindset you’re aiming to communicate here should apply to imagining things that so far only exist in a map, but might be part of the territory some day.
...
(By contrast, for the record, I totally believe, and have already worked to shift, myself towards getting curious about my internal thought processes. Reading your old Noticing sequence was pretty valuable for me becoming ‘a real rationalist’ in some sense, who is able to reflect on their cognitive algorithms and work to improve them)
Here are some words that I’ve not really vetted yet; they’re probably not quite the right words, and I probably don’t quite believe whatever’s picked out by these particular ones in this order:
There’s something special about nature. By “nature” I mean newts and mushrooms and sunlight, and also geometry and probably number theory and abstract algebra. By “nature”, I mean things that have not been contaminated by human design. What is the largest pair of twin primes? They may not crawl in the dirt, but their magnitude (or non-existence) depends no more on human thought and perception than does the average number of spots on the back of a red eft. (Maybe. I dunno, my so called math background barely exists, and also I’m not very settled on central questions in the philosophy of math.)
There are things that are almost entirely of human design, such as Facebook and novels and operatic overtures. There are things that are almost entirely devoid of human design, such as newts and meteorites and the Mariana Trench. And there are many, many things somewhere in the middle: things that came from nature but have been shaped to human purposes, such as my stoneware vase; things that interact with extremely non-human systems but whose human interfaces are extensively designed, such as Python and other programming languages that are very far from the metal (exercise: rank programming languages by how immediately tangled with non-human systems they require the programmer to be); and perhaps things that were dreamed up by human imagination but point themselves quite directly at nature (carbon sequestration technologies, maybe? Some approaches to horticulture? I really don’t know about this third thing, I’m sleepy and I think I’m just completing the pattern I set up).
It is possible, especially if you live in a city, to spend almost literally all of your time engaged with things that are very far on the “contaminated by human design” side of the gradient. Which is kind of like spending all of your time staring at your own reflection. Without even knowing it, maybe, if you were raised in a city by people who were raised this way. Imagine trying to learn about the world while trapped in a room whose walls and ceiling are made entirely of mirrors.
You’re absolutely correct that “knowing the territory takes patient and direct observation” applies to, um, everything? Certainly not just newts and sunlight. The thing about newts and sunlight—the reason I’ve put the central attitudes and approaches of naturalism-qua-natural-history on a pedestal and taken them as the inspiration for a branch of rationality—is that it’s very hard to cheat at newts and sunlight. Compared to interpreting operatic overtures, it’s very hard to quickly make up whatever you like about newts and fail to notice that actual newts do not at all resemble your imaginings, at least if you’re making a good faith effort to study them. If you study newts, especially if you study a particular newt that nobody’s written extensively about, or if you’re a for real 18th century natural historian with no wifi at all, you really just have no recourse but patient and direct observation, and the distance between your map and the territory will smack you in the face nearly every time you try to look.
So, I’m definitely not saying “in order to take this general approach I’m talking about, you need to study newts and sunlight, not AI alignment”. Indeed, AI alignment research is precisely my target for this work (eventually, probably). I’m saying something more like, “It seems to me that the perspectives and methods that have arisen out of fields and communities that are quite divorced from the natural world may be fatally deficient in this thing that entomologists just fucking nail day in and day out”.
And although I want naturalism in Python, and even in the interpretation of operatic overtures, I do think it’s a really good idea to look at newts, sunlight, and geometry while building form. I guess this concluding essay was largely about how and why I got the hang of it, and why I’ve kept the hang of it, and what having the hang of it feels like. Maybe don’t start out trying to learn looking-at-things-other-than-mirrors from inside of a mirror room, ya know? Get as far away from mirrors as you can manage, until you start to get the hang of it.
For now, here is an unsatisfactory response that will be very rambly and probably off topic.
(For what it’s worth, I found it quite helpful to see these motivations laid out like this, and am glad that you Logan wrote this comment and that you Raemon asked the question that provoked it.)
I think naturalism can be directed even at things “contaminated by human design”, if you apply the framing correctly. In a way, that’s how I started out as something of a naturalist, so it is territory I’d consider a bit familiar.
The best starting point I can offer based on Raemon’s comment is to look at changes in a field of study or technology over time, preferably one you already have some interest in (perhaps AI-related?). The naturalist perspective focuses on small observations over time, so I recommend embarking on brief “nature walks” where you find some way to expose yourself to information regarding some innovation in the field, be it ancient or modern. An example of this could be reading up on a new training algorithm you are not already familiar with (since it will be easier to use Original Seeing upon), without expending too much concentration or energy upon trying to calculate major insights.
For example, when asked to think about something I would like more deeper, masterful knowledge about, I replied “artificial neural networks”.
The closest thing I could think to potentially experiencing them and interacting is either 1. Through interactive demos or 2. Through a suit like this. I’m unsure if that is what is meant by interaction, though it does seem closer.
I know very little about artificial neural networks, like I’m not really even comfortable saying that I know what they are. (Without googling I’m basically like, “Well, probably they’re still systems of logic gates at ground level, though maybe some of them involve quantum computing circuits that are fishy-to-me in their non-binariness or something, and those systems have properties that resemble non-artificial neural networks such as nodes and weighted edges, which causes them to behave like non-artificial neural networks with stuff like association cascades and trigger-action patterns and predictive-processing-type stuff?”) [Edit: Oh snap that actually sounds a lot like the Wikipedia page!]
But I can easily imagine someone who’s very interested in artificial neural networks and has so far studied them by reading about them and talking to people about them. It’s a very different kind of thing to try to design one, at all or even from scratch, to try to use one for various purposes, to provide certain inputs and statistically analyze patterns of outputs, to reason mathematically about what seem to you to be necessary properties of artificial neural networks and then find out whether an actual neural network in front of you behaves as you’ve predicted.
So yeah if you’ve mostly been in “reading stuff” territory, that interactive demo looks to me like a great step in the right direction. But if I were in that position, I would be asking myself “what is the very simplest thing that would technically count as an artificial neural network, and what would it take for me to build that thing myself?”
If you’re not in the “mostly I’ve been reading stuff” boat and have already been doing the kinds of things I’ve described so far, then I expect that increasing the directness of your contact will look less like interacting with a different kind of thing, and more like adopting different mental patterns as you interact. How much of you is showing up to your investigations? What parts of you are asking questions, what parts of you are generating hypotheses? How many methods are you employing for turning your central puzzles around and around to see them from different angles, and what is the range of those methods? What work are you doing to let different activities of your daily life participate in your processes of observation and analysis? What are you doing to become sensitive to subtle patterns in your observations that can only become apparent over time? That kind of thing.
Remember that there are THREE entities needed for contact with the territory: The territory, the person making contact, and sensation at the point of contact. You can change your contact by changing any three of those entities.
In case you haven’t seen it, Chris Olah’s work in neural network interpretability is extremely concordant with naturalism.
It’s common for people to say that neural networks are something like “a mass of inscrutable tensors” and I feel like Olah is one of the only people whose response was something like “did you try literally scrutinizing them” and the answer is no, no one did, because they looked complicated and icky. And then Olah did, and when he looked, he saw things.
Ok I’m actually pretty curious about this myself now. The basic element of an ANN is a neuron I think, and maybe I could personally build a single neuron out of household materials? It doesn’t gotta do much, right?
It needs to be able to receive at least one input (though to be at all interesting it probably ought to receive at least 2).
It needs to sum its inputs.
Something about an activation function. Does this happen before or after the summing? My guess is after; so maybe it’s stuff like “if the sum is less than 4 then make the output be two less than the sum, but if it’s more than 4 make it be the sum times six”.
Then it’s gotta be able to output, ideally to something observable.
So I’m imagining a little circuit board of logic gates with copper wires attached to batteries and a lightbulb that can glow brighter when you give it more juice, with the activation function business happening in a series of insulators of varying strengths [uh, conductors of varying resistances?] and the variable inputs also coming from currents run through different insulators, or perhaps from different strengths of batteries.
What do you think ML people, am I on the right track? Have I sketched an artificial neuron?
Something I notice as I read this, that I think has made it a bit harder to grok this sequence:
This post says “I’m a naturalist”, and then lists a bunch of examples of things that are, well, natural. Newts and dirt and sunshine and stars and stuff. I do aesthetically like the idea of getting really innately patiently curious about those things. But that’s..
a) far removed from my current habits,
b) getting persistently patiently curious about those things just feels like a really inefficient way to make progress on the stuff that I’m actually trying to figure out,
c) perhaps most importantly, I get the sense that the spirit of the thing you’re trying to convey here isn’t limited to newts and dirt and sunshine and mushrooms. It should also apply to the various worlds I currently “live” in. Some things are tightly analogous to mushrooms/dirt/sunshine (there is in fact literal dirt and literal sunshine and flowers and animals on the streets of Berkeley, and other concrete things to get curious about like stop signs and pavement cracks). But it seems like the mindset of details/direct observation should also apply to javascript, and machine learning. It’s a bit interesting that in some sense a javascript codebase is “a map”, rather than “a territory.” But it’s also in some sense a “territory”. What about abstract math?
So I feel like I could use more poetry that conveys how this applies to the latter set of things. (I think you have actual background in math so could probably paint poetry yourself about that?). For that matter, just poetry about multiple sets of things you could excitedly patiently observe, that don’t have much to do with each other, to help triangulate the-part-that-you’re-ultimately-getting-at, rather than one aesthetic that (I think?) is fairly incidental.
I’m also definitely confused about how to apply all of this to research into things that don’t exist yet. (I’m not sure if this sequence is supposed to bridge to that yet).
I do get a takeaway of “okay Ray, in addition to trying to Think Hard About AI Alignment Research, you should be doing things that a) give yourself space from that so you brain has room to do various other brain things that feed into that in subtler ways, b) adopt a patient observer mindset that applies to lots things you’re interacting with as you contemplate the state of AI Alignment Research.”
But I have some sense that, ultimately, the mindset you’re aiming to communicate here should apply to imagining things that so far only exist in a map, but might be part of the territory some day.
...
(By contrast, for the record, I totally believe, and have already worked to shift, myself towards getting curious about my internal thought processes. Reading your old Noticing sequence was pretty valuable for me becoming ‘a real rationalist’ in some sense, who is able to reflect on their cognitive algorithms and work to improve them)
I like this comment!
I intend to respond to it more extensively at some point. For now, here is an unsatisfactory response that will be very rambly and probably off topic.
At least some of whatever response I want to give is contained in Ben’s recent post about the discovery of nature’s laws. https://www.lesswrong.com/posts/uiyWHaTrz3ML7JqDX/12-interesting-things-i-learned-studying-the-discovery-of
Here are some words that I’ve not really vetted yet; they’re probably not quite the right words, and I probably don’t quite believe whatever’s picked out by these particular ones in this order:
There’s something special about nature. By “nature” I mean newts and mushrooms and sunlight, and also geometry and probably number theory and abstract algebra. By “nature”, I mean things that have not been contaminated by human design. What is the largest pair of twin primes? They may not crawl in the dirt, but their magnitude (or non-existence) depends no more on human thought and perception than does the average number of spots on the back of a red eft. (Maybe. I dunno, my so called math background barely exists, and also I’m not very settled on central questions in the philosophy of math.)
There are things that are almost entirely of human design, such as Facebook and novels and operatic overtures. There are things that are almost entirely devoid of human design, such as newts and meteorites and the Mariana Trench. And there are many, many things somewhere in the middle: things that came from nature but have been shaped to human purposes, such as my stoneware vase; things that interact with extremely non-human systems but whose human interfaces are extensively designed, such as Python and other programming languages that are very far from the metal (exercise: rank programming languages by how immediately tangled with non-human systems they require the programmer to be); and perhaps things that were dreamed up by human imagination but point themselves quite directly at nature (carbon sequestration technologies, maybe? Some approaches to horticulture? I really don’t know about this third thing, I’m sleepy and I think I’m just completing the pattern I set up).
It is possible, especially if you live in a city, to spend almost literally all of your time engaged with things that are very far on the “contaminated by human design” side of the gradient. Which is kind of like spending all of your time staring at your own reflection. Without even knowing it, maybe, if you were raised in a city by people who were raised this way. Imagine trying to learn about the world while trapped in a room whose walls and ceiling are made entirely of mirrors.
You’re absolutely correct that “knowing the territory takes patient and direct observation” applies to, um, everything? Certainly not just newts and sunlight. The thing about newts and sunlight—the reason I’ve put the central attitudes and approaches of naturalism-qua-natural-history on a pedestal and taken them as the inspiration for a branch of rationality—is that it’s very hard to cheat at newts and sunlight. Compared to interpreting operatic overtures, it’s very hard to quickly make up whatever you like about newts and fail to notice that actual newts do not at all resemble your imaginings, at least if you’re making a good faith effort to study them. If you study newts, especially if you study a particular newt that nobody’s written extensively about, or if you’re a for real 18th century natural historian with no wifi at all, you really just have no recourse but patient and direct observation, and the distance between your map and the territory will smack you in the face nearly every time you try to look.
So, I’m definitely not saying “in order to take this general approach I’m talking about, you need to study newts and sunlight, not AI alignment”. Indeed, AI alignment research is precisely my target for this work (eventually, probably). I’m saying something more like, “It seems to me that the perspectives and methods that have arisen out of fields and communities that are quite divorced from the natural world may be fatally deficient in this thing that entomologists just fucking nail day in and day out”.
And although I want naturalism in Python, and even in the interpretation of operatic overtures, I do think it’s a really good idea to look at newts, sunlight, and geometry while building form. I guess this concluding essay was largely about how and why I got the hang of it, and why I’ve kept the hang of it, and what having the hang of it feels like. Maybe don’t start out trying to learn looking-at-things-other-than-mirrors from inside of a mirror room, ya know? Get as far away from mirrors as you can manage, until you start to get the hang of it.
(For what it’s worth, I found it quite helpful to see these motivations laid out like this, and am glad that you Logan wrote this comment and that you Raemon asked the question that provoked it.)
I think naturalism can be directed even at things “contaminated by human design”, if you apply the framing correctly. In a way, that’s how I started out as something of a naturalist, so it is territory I’d consider a bit familiar.
The best starting point I can offer based on Raemon’s comment is to look at changes in a field of study or technology over time, preferably one you already have some interest in (perhaps AI-related?). The naturalist perspective focuses on small observations over time, so I recommend embarking on brief “nature walks” where you find some way to expose yourself to information regarding some innovation in the field, be it ancient or modern. An example of this could be reading up on a new training algorithm you are not already familiar with (since it will be easier to use Original Seeing upon), without expending too much concentration or energy upon trying to calculate major insights.
For example, when asked to think about something I would like more deeper, masterful knowledge about, I replied “artificial neural networks”.
The closest thing I could think to potentially experiencing them and interacting is either 1. Through interactive demos or 2. Through a suit like this. I’m unsure if that is what is meant by interaction, though it does seem closer.
I know very little about artificial neural networks, like I’m not really even comfortable saying that I know what they are. (Without googling I’m basically like, “Well, probably they’re still systems of logic gates at ground level, though maybe some of them involve quantum computing circuits that are fishy-to-me in their non-binariness or something, and those systems have properties that resemble non-artificial neural networks such as nodes and weighted edges, which causes them to behave like non-artificial neural networks with stuff like association cascades and trigger-action patterns and predictive-processing-type stuff?”) [Edit: Oh snap that actually sounds a lot like the Wikipedia page!]
But I can easily imagine someone who’s very interested in artificial neural networks and has so far studied them by reading about them and talking to people about them. It’s a very different kind of thing to try to design one, at all or even from scratch, to try to use one for various purposes, to provide certain inputs and statistically analyze patterns of outputs, to reason mathematically about what seem to you to be necessary properties of artificial neural networks and then find out whether an actual neural network in front of you behaves as you’ve predicted.
So yeah if you’ve mostly been in “reading stuff” territory, that interactive demo looks to me like a great step in the right direction. But if I were in that position, I would be asking myself “what is the very simplest thing that would technically count as an artificial neural network, and what would it take for me to build that thing myself?”
If you’re not in the “mostly I’ve been reading stuff” boat and have already been doing the kinds of things I’ve described so far, then I expect that increasing the directness of your contact will look less like interacting with a different kind of thing, and more like adopting different mental patterns as you interact. How much of you is showing up to your investigations? What parts of you are asking questions, what parts of you are generating hypotheses? How many methods are you employing for turning your central puzzles around and around to see them from different angles, and what is the range of those methods? What work are you doing to let different activities of your daily life participate in your processes of observation and analysis? What are you doing to become sensitive to subtle patterns in your observations that can only become apparent over time? That kind of thing.
Remember that there are THREE entities needed for contact with the territory: The territory, the person making contact, and sensation at the point of contact. You can change your contact by changing any three of those entities.
In case you haven’t seen it, Chris Olah’s work in neural network interpretability is extremely concordant with naturalism.
It’s common for people to say that neural networks are something like “a mass of inscrutable tensors” and I feel like Olah is one of the only people whose response was something like “did you try literally scrutinizing them” and the answer is no, no one did, because they looked complicated and icky. And then Olah did, and when he looked, he saw things.
Agreed. Also had this experience recently, although I really like the way you put it here.
Ok I’m actually pretty curious about this myself now. The basic element of an ANN is a neuron I think, and maybe I could personally build a single neuron out of household materials? It doesn’t gotta do much, right?
It needs to be able to receive at least one input (though to be at all interesting it probably ought to receive at least 2).
It needs to sum its inputs.
Something about an activation function. Does this happen before or after the summing? My guess is after; so maybe it’s stuff like “if the sum is less than 4 then make the output be two less than the sum, but if it’s more than 4 make it be the sum times six”.
Then it’s gotta be able to output, ideally to something observable.
So I’m imagining a little circuit board of logic gates with copper wires attached to batteries and a lightbulb that can glow brighter when you give it more juice, with the activation function business happening in a series of insulators of varying strengths [uh, conductors of varying resistances?] and the variable inputs also coming from currents run through different insulators, or perhaps from different strengths of batteries.
What do you think ML people, am I on the right track? Have I sketched an artificial neuron?
Actually I think I’m yearning to do this with marbles and pipes and carefully balanced buckets so the weightings can be totally literal.
Yeah, that sounds right! I think this video supports that idea as well:
That sounds like it’s on the right track to me. Is there any chance you’ve made progress on this?
I kind of feel like Julia Evans might be doing naturalism about programming things. https://jvns.ca/
(Though this isn’t an intuition I feel up for trying to examine very closely right before I go to bed.)