Hypothesis: for smart people with a strong technical background, the main cognitive barrier to doing highly counterfactual technical work is that our brains’ attention is mostly steered by our social circle. Our thoughts are constantly drawn to think about whatever the people around us talk about. And the things which are memetically fit are (almost by definition) rarely very counterfactual to pay attention to, precisely because lots of other people are also paying attention to them.
Two natural solutions to this problem:
build a social circle which can maintain its own attention, as a group, without just reflecting the memetic currents of the world around it.
“go off into the woods”, i.e. socially isolate oneself almost entirely for an extended period of time, so that there just isn’t any social signal to be distracted by.
These are both standard things which people point to as things-historically-correlated-with-highly-counterfactual-work. They’re not mutually exclusive, but this model does suggest that they can substitute for each other—i.e. “going off into the woods” can substitute for a social circle with its own useful memetic environment, and vice versa.
One thing that I do after social interactions, especially those which pertain to my work, is to go over all the updates my background processing is likely to make and to question them more explicitly.
This is helpful because I often notice that the updates I’m making aren’t related to reasons much at all. It’s more like “ah they kind of grimaced when I said that, so maybe I’m bad?” or like “they seemed just generally down on this approach, but wait are any of those reasons even new to me? Haven’t I already considered those and decided to do it anyway?” or “they seemed so aggressively pessimistic about my work, but did they even understand what I was saying?” or “they certainly spoke with a lot of authority, but why should I trust them on this, and do I even care about their opinion here?” Etc. A bunch of stuff which at first blush my social center is like “ah god, it’s all over, I’ve been an idiot this whole time” but with some second glancing it’s like “ah wait no, probably I had reasons for doing this work that withstand surface level pushback, let’s remember those again and see if they hold up” And often (always?) they do.
This did not come naturally to me; I’ve had to train myself into doing it. But it has helped a lot with this sort of problem, alongside the solutions you mention i.e. becoming more of a hermit and trying to surround myself by people engaged in more timeless thought.
solution 2 implies that a smart person with a strong technical background would go on to work on important problems (by default) which is not necessarily universally true and it’s IMO likely that many such people would be working on less important things than what their social circle is otherwise steering them to work on
The claim is not that either “solution” is sufficient for counterfactuality, it’s that either solution can overcome the main bottleneck to counterfactuality. After that, per Amdahl’s Law, there will still be other (weaker) bottlenecks to overcome, including e.g. keeping oneself focused on something important.
I don’t think the social thing ranks above “be able to think useful important thoughts at all”. (But maybe otherwise agree with the rest of your model as an important thing to think about)
[edit: hrm, “for smart people with a strong technical background” might be doing most of the work here”]
it’s IMO likely that many such people would be working on less important things than what their social circle is otherwise steering them to work on
Why do you think this? When I try to think of concrete examples here, its all confounded by the relevant smart people having social circles not working on useful problems.
I also think that 2 becomes more true once the relevant smart person already wants to solve alignment, or otherwise is already barking up the right tree.
As a counterpoint to the “go off into the woods” strategy, Richard Hamming said the following in “You and Your Research”, describing his experience at Bell Labs:
Thus what you consider to be good working conditions may not be good for you! There are many illustrations of this point. For example, working with one’s door closed lets you get more work done per year than if you had an open door, but I have observed repeatedly that later those with the closed doors, while working just as hard as others, seem to work on slightly the wrong problems, while those who have let their door stay open get less work done but tend to work on the right problems! I cannot prove the cause-and-effect relationship; I can only observed the correlation. I suspect the open mind leads to the open door, and the open door tends to lead to the open mind; they reinforce each other.
Bell Labs certainly produced a lot of counterfactual research, Shannon’s information theory being the prime example. I suppose Bell Labs might have been well-described as a group that could maintain its own attention, though.
Bell Labs is actually my go-to example of a much-hyped research institution whose work was mostly not counterfactual; see e.g. here. Shannon’s information theory is the only major example I know of highly counterfactual research at Bell Labs. Most of the other commonly-cited advances, like e.g. transistors or communication satellites or cell phones, were clearly not highly counterfactual when we look at the relevant history: there were other groups racing to make the transistor, and the communication satellite and cell phones were both old ideas waiting on the underlying technology to make them practical.
That said, Hamming did sit right next to Shannon during the information theory days IIRC, so his words do carry substantial weight here.
Good idea, but… I would guess that basically everyone who knew me growing up would say that I’m exactly the right sort of person for that strategy. And yet, in practice, I still find it has not worked very well. My attention has in fact been unhelpfully steered by local memetic currents to a very large degree.
For instance, I do love proving everyone else wrong, but alas reversed stupidity is not intelligence. People mostly don’t argue against the high-counterfactuality important things, they ignore the high-counterfactuality important things. Trying to prove them wrong about the things they do argue about is just another way of having one’s attention steered by the prevailing memetic currents.
People mostly don’t argue against the high-counterfactuality important things, they ignore the high-counterfactuality important things. Trying to prove them wrong about the things they do argue about is just another way of having one’s attention steered by the prevailing memetic currents.
This is true, but I still can’t let go of the fact that this fact itself ought to be a blindingly obvious first-order bit that anyone who calls zerself anything like “aspiring rationalist” would be paying a good chunk of attention to, and yet this does not seem to be the case. Like, motions in the genre of
huh I just had reaction XYZ to idea ABC generated by a naively-good search process, and it seems like this is probably a common reaction to ABC; but if people tend to react to ABC with XYZ, and with other things coming from the generators of XYZ, then such and such distortion in beliefs/plans would be strongly pushed into the collective consciousness, e.g. on first-order or on higher-order deference effects ; so I should look out for that, e.g. by doing some manual fermi estimates or other direct checking about ABC or by investigating the strength of the steelman of reaction XYZ, or by keeping an eye out for people systematically reacting with XYZ without good foundation so I can notice this,
where XYZ could centrally be things like e.g. copium or subtly contemptuous indifference, do not seem to be at all common motions.
So I should look out for that, e.g. by doing some manual fermi estimates or other direct checking about ABC or by investigating the strength of the steelman of reaction XYZ, or by keeping an eye out for people systematically reacting with XYZ without good foundation so I can notice this,
Accusing people in my head of not being numerate enough when this happens has helped, because then I don’t want to be a hypocrite. GPT4o or o1 are good at fermi estimates, making this even easier.
build a social circle which can maintain its own attention, as a group, without just reflecting the memetic currents of the world around it.
Note that it is not necessary for the social circle to share your beliefs, only to have a social norm that people express interest in each other’s work. Could be something like: once or twice in a week the people will come to a room and everyone will give a presentation about what they have achieved recently, and maybe the other people will provide some feedback (not in the form of “why don’t you do Y instead”, but with the assumption that X is a thing worth doing).
How would this model treat mathematicians working on hard open problems? P vs NP might be counter factual just because no one else is smart enough or has the right advantage to solve it. Insofar as central problems of a field have been identified but not solved, I’m not sure your model gives good advice.
I visited Mikhail Khovanov once in New York to give a seminar talk, and after it was all over and I was wandering around seeing the sights, he gave me a call and offered a long string of general advice on how to be the kind of person who does truly novel things (he’s famous for this, you can read about Khovanov homology). One thing he said was “look for things that aren’t there” haha. It’s actually very practical advice, which I think about often and attempt to live up to!
I’m ashamed to say I don’t remember. That was the highlight. I think I have some notes on the conversation somewhere and I’ll try to remember to post here if I ever find it.
I can spell out the content of his Koan a little, if it wasn’t clear. It’s probably more like: look for things that are (not there). If you spend enough time in a particular landscape of ideas, you can (if you’re quiet and pay attention and aren’t busy jumping on bandwagons) get an idea of a hole, which you’re able to walk around but can’t directly see. In this way new ideas appear as something like residues from circumnavigating these holes. It’s my understanding that Khovanov homology was discovered like that, and this is not unusual in mathematics.
By the way, that’s partly why I think the prospect of AIs being creative mathematicians in the short term should not be discounted; if you see all the things you see all the holes.
For those who might not have noticed Dan’s clever double entendre: (Khovanov) homology is literally about counting/measuring holes in weird high-dimensional spaces—designing a new homology theory is in a very real sense about looking for holes that are not (yet) there.
There’s plenty, including a line of work by Carina Curto, Katrin Hess and others that is taken seriously by a number of mathematically inclined neuroscience people (Tom Burns if he’s reading can comment further). As far as I know this kind of work is the closest to breaking through into the mainstream. At some level you can think of homology as a natural way of preserving information in noisy systems, for reasons similar to why (co)homology of tori was a useful way for Kitaev to formulate his surface code. Whether or not real brains/NNs have some emergent computation that makes use of this is a separate question, I’m not aware of really compelling evidence.
There is more speculative but definitely interesting work by Matilde Marcolli. I believe Manin has thought about this (because he’s thought about everything) and if you have twenty years to acquire the prerequisites (gamma spaces!) you can gaze into deep pools by reading that too.
Though my understanding is this is used in interp, not so much because people necessarily expect deep connections to homology, but because its just another way to look for structure in your data.
As someone who does both data analysis and algebraic topology, my take is that TDA showed promise but ultimately there’s something missing such that it’s not at full capacity. Either the formalism isn’t developed enough or it’s being consistently used on the wrong kinds of datasets. Which is kind of a shame, because it’s the kind of thing that should work beautifully and in some cases even does!
I thought it might be “look for things that might not even be there as hard as you would if they are there.” Then the koan form takes it closer to “the thereness of something just has little relevance on how hard you look for it.” But it needs to get closer to the “biological” part of your brain, where you’re not faking it with all your mental and bodily systems, like when your blood pressure rises from “truly believing” a lion is around the corner but wouldn’t if you “fake believe” it.
Obvious point—I think a lot of this comes from the financial incentives. The more “out of the box” you go, the less sure you can be that there will be funding for your work.
Some of those that do this will be rewarded, but I suspect many won’t be.
As such, I think that funders can help more to encourage this sort of thing, if they want to.
Hypothesis: for smart people with a strong technical background, the main cognitive barrier to doing highly counterfactual technical work is that our brains’ attention is mostly steered by our social circle. Our thoughts are constantly drawn to think about whatever the people around us talk about. And the things which are memetically fit are (almost by definition) rarely very counterfactual to pay attention to, precisely because lots of other people are also paying attention to them.
Two natural solutions to this problem:
build a social circle which can maintain its own attention, as a group, without just reflecting the memetic currents of the world around it.
“go off into the woods”, i.e. socially isolate oneself almost entirely for an extended period of time, so that there just isn’t any social signal to be distracted by.
These are both standard things which people point to as things-historically-correlated-with-highly-counterfactual-work. They’re not mutually exclusive, but this model does suggest that they can substitute for each other—i.e. “going off into the woods” can substitute for a social circle with its own useful memetic environment, and vice versa.
One thing that I do after social interactions, especially those which pertain to my work, is to go over all the updates my background processing is likely to make and to question them more explicitly.
This is helpful because I often notice that the updates I’m making aren’t related to reasons much at all. It’s more like “ah they kind of grimaced when I said that, so maybe I’m bad?” or like “they seemed just generally down on this approach, but wait are any of those reasons even new to me? Haven’t I already considered those and decided to do it anyway?” or “they seemed so aggressively pessimistic about my work, but did they even understand what I was saying?” or “they certainly spoke with a lot of authority, but why should I trust them on this, and do I even care about their opinion here?” Etc. A bunch of stuff which at first blush my social center is like “ah god, it’s all over, I’ve been an idiot this whole time” but with some second glancing it’s like “ah wait no, probably I had reasons for doing this work that withstand surface level pushback, let’s remember those again and see if they hold up” And often (always?) they do.
This did not come naturally to me; I’ve had to train myself into doing it. But it has helped a lot with this sort of problem, alongside the solutions you mention i.e. becoming more of a hermit and trying to surround myself by people engaged in more timeless thought.
solution 2 implies that a smart person with a strong technical background would go on to work on important problems (by default) which is not necessarily universally true and it’s IMO likely that many such people would be working on less important things than what their social circle is otherwise steering them to work on
The claim is not that either “solution” is sufficient for counterfactuality, it’s that either solution can overcome the main bottleneck to counterfactuality. After that, per Amdahl’s Law, there will still be other (weaker) bottlenecks to overcome, including e.g. keeping oneself focused on something important.
I don’t think the social thing ranks above “be able to think useful important thoughts at all”. (But maybe otherwise agree with the rest of your model as an important thing to think about)
[edit: hrm, “for smart people with a strong technical background” might be doing most of the work here”]
Plausibly going off into the woods decreases the median output while increasing the variance.
Why do you think this? When I try to think of concrete examples here, its all confounded by the relevant smart people having social circles not working on useful problems.
I also think that 2 becomes more true once the relevant smart person already wants to solve alignment, or otherwise is already barking up the right tree.
One need not go off into the woods indefinitely, though.
I don’t think I implied that John’s post implied that and I don’t think going into the woods non-indefinitely mitigates the thing I pointed out.
As a counterpoint to the “go off into the woods” strategy, Richard Hamming said the following in “You and Your Research”, describing his experience at Bell Labs:
Bell Labs certainly produced a lot of counterfactual research, Shannon’s information theory being the prime example. I suppose Bell Labs might have been well-described as a group that could maintain its own attention, though.
Bell Labs is actually my go-to example of a much-hyped research institution whose work was mostly not counterfactual; see e.g. here. Shannon’s information theory is the only major example I know of highly counterfactual research at Bell Labs. Most of the other commonly-cited advances, like e.g. transistors or communication satellites or cell phones, were clearly not highly counterfactual when we look at the relevant history: there were other groups racing to make the transistor, and the communication satellite and cell phones were both old ideas waiting on the underlying technology to make them practical.
That said, Hamming did sit right next to Shannon during the information theory days IIRC, so his words do carry substantial weight here.
solution 3 is to be an iconoclast and to feel comfortable pushing against the flow and to try to prove everyone else wrong.
Good idea, but… I would guess that basically everyone who knew me growing up would say that I’m exactly the right sort of person for that strategy. And yet, in practice, I still find it has not worked very well. My attention has in fact been unhelpfully steered by local memetic currents to a very large degree.
For instance, I do love proving everyone else wrong, but alas reversed stupidity is not intelligence. People mostly don’t argue against the high-counterfactuality important things, they ignore the high-counterfactuality important things. Trying to prove them wrong about the things they do argue about is just another way of having one’s attention steered by the prevailing memetic currents.
This is true, but I still can’t let go of the fact that this fact itself ought to be a blindingly obvious first-order bit that anyone who calls zerself anything like “aspiring rationalist” would be paying a good chunk of attention to, and yet this does not seem to be the case. Like, motions in the genre of
where XYZ could centrally be things like e.g. copium or subtly contemptuous indifference, do not seem to be at all common motions.
Accusing people in my head of not being numerate enough when this happens has helped, because then I don’t want to be a hypocrite. GPT4o or o1 are good at fermi estimates, making this even easier.
Note that it is not necessary for the social circle to share your beliefs, only to have a social norm that people express interest in each other’s work. Could be something like: once or twice in a week the people will come to a room and everyone will give a presentation about what they have achieved recently, and maybe the other people will provide some feedback (not in the form of “why don’t you do Y instead”, but with the assumption that X is a thing worth doing).
How would this model treat mathematicians working on hard open problems? P vs NP might be counter factual just because no one else is smart enough or has the right advantage to solve it. Insofar as central problems of a field have been identified but not solved, I’m not sure your model gives good advice.
I visited Mikhail Khovanov once in New York to give a seminar talk, and after it was all over and I was wandering around seeing the sights, he gave me a call and offered a long string of general advice on how to be the kind of person who does truly novel things (he’s famous for this, you can read about Khovanov homology). One thing he said was “look for things that aren’t there” haha. It’s actually very practical advice, which I think about often and attempt to live up to!
What else did he say? (I’d love to hear even the “obvious” things he said.)
I’m ashamed to say I don’t remember. That was the highlight. I think I have some notes on the conversation somewhere and I’ll try to remember to post here if I ever find it.
I can spell out the content of his Koan a little, if it wasn’t clear. It’s probably more like: look for things that are (not there). If you spend enough time in a particular landscape of ideas, you can (if you’re quiet and pay attention and aren’t busy jumping on bandwagons) get an idea of a hole, which you’re able to walk around but can’t directly see. In this way new ideas appear as something like residues from circumnavigating these holes. It’s my understanding that Khovanov homology was discovered like that, and this is not unusual in mathematics.
By the way, that’s partly why I think the prospect of AIs being creative mathematicians in the short term should not be discounted; if you see all the things you see all the holes.
For those who might not have noticed Dan’s clever double entendre: (Khovanov) homology is literally about counting/measuring holes in weird high-dimensional spaces—designing a new homology theory is in a very real sense about looking for holes that are not (yet) there.
Are there any examples yet, of homology or cohomology being applied to cognition, whether human or AI?
There’s plenty, including a line of work by Carina Curto, Katrin Hess and others that is taken seriously by a number of mathematically inclined neuroscience people (Tom Burns if he’s reading can comment further). As far as I know this kind of work is the closest to breaking through into the mainstream. At some level you can think of homology as a natural way of preserving information in noisy systems, for reasons similar to why (co)homology of tori was a useful way for Kitaev to formulate his surface code. Whether or not real brains/NNs have some emergent computation that makes use of this is a separate question, I’m not aware of really compelling evidence.
There is more speculative but definitely interesting work by Matilde Marcolli. I believe Manin has thought about this (because he’s thought about everything) and if you have twenty years to acquire the prerequisites (gamma spaces!) you can gaze into deep pools by reading that too.
No.
Topological data analysis comes closest, and there are some people who try to use it for ML, eg.
Though my understanding is this is used in interp, not so much because people necessarily expect deep connections to homology, but because its just another way to look for structure in your data.
TDA itself is also a relatively shallow tool too.
As someone who does both data analysis and algebraic topology, my take is that TDA showed promise but ultimately there’s something missing such that it’s not at full capacity. Either the formalism isn’t developed enough or it’s being consistently used on the wrong kinds of datasets. Which is kind of a shame, because it’s the kind of thing that should work beautifully and in some cases even does!
I thought it might be “look for things that might not even be there as hard as you would if they are there.” Then the koan form takes it closer to “the thereness of something just has little relevance on how hard you look for it.” But it needs to get closer to the “biological” part of your brain, where you’re not faking it with all your mental and bodily systems, like when your blood pressure rises from “truly believing” a lion is around the corner but wouldn’t if you “fake believe” it.
I imagine it’s something like “look for things that are notably absent, when you would expect them to have been found if there”?
Some things even withdraw. https://tsvibt.blogspot.com/2023/05/the-possible-shared-craft-of-deliberate.html#aside-on-withdrawal-and-the-leap https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html#withdrawal
Obvious point—I think a lot of this comes from the financial incentives. The more “out of the box” you go, the less sure you can be that there will be funding for your work.
Some of those that do this will be rewarded, but I suspect many won’t be.
As such, I think that funders can help more to encourage this sort of thing, if they want to.