[This is condensed and informalized from a much longer and more explicit comment which I’m not sure would have been worth wading through, which still seemed hazy in important ways, and which seemed like it needed me to open more boxes than I have energy for right now. This one still seems hazy, but hopefully it wears it more on its sleeve. I should also declare up front that I have a bunch of weird emotional warping around this topic; hopefully I’m working around enough of it for this to still be useful.]
I think you’re interpreting “this is not how human psychology works” in a noncentral way compared to how Bob Jacobs is likely to have meant it, or maybe asserting your examples of psychology working that way more as normative than as positive claims.
I have a completely different tack in mind: how do we know that the sort of mental maneuvers you describe don’t become harmful in their aggregate effects when too many people do them, or do them without coordinating enough, or something along those lines?
I would like to point out the following:
The person literally could not process what she said because it was so far from what he was expecting, and she felt foolish for saying it. Her injury then swelled up, even though it had already been a while since the break.
“felt foolish” together with the consequences looks like a description of an alief-based and alief-affecting social feedback mechanism. How safe is it for individuals to unilaterally train themselves out of such mechanisms? Some detachment from it seems to be part of emotional maturity, but that’s coupled with a lot of other mediating material. Further detachment seems to be part of various spiritual traditions—also coupled with even more mediating material. That’s not very promising for any implied “there’s no such thing as too much”.
More specifically, I would like to consider the possibility that many of the sort of false aliefs you’re talking about act more like restraining bolts imposed by the social cohesion subunit of a human mind, “because” humans are not safe under amplification with regard to social values. (And notably, “hypercompetent individuals are good for society” is by no means a universal more.)
Or: individual coherence and social cohesion seem to be at odds often enough for that to be a way for “not-winning due to being too coherent” to sneak in through crazy backdoors in the environment, absent unbounded handling-of-detachment resources which are not in evidence and at some point may be unimplementable within human bounds.
[… this was going to be more-edited, but I’ve accidentally hit Submit, and I don’t want to do too much frantic editing, so I’ve just cleaned up a few pieces. I think this is still just-about worth enough to leave up; I’ll try to come back to it later if it’s deemed worth talking about.]
I should also declare up front that I have a bunch of weird emotional warping around this topic; hopefully I’m working around enough of it for this to still be useful.]
This is a really cool declaration. It doesn’t bleed through in any obvious way, but thanks for letting me know and I’ll try to be cautious of what I say/how I say them. Lemme know if I’m bumping into anything or if there’s anything I could be doing differently to better accommodate.
I think you’re interpreting “this is not how human psychology works” in a noncentral way compared to how Bob Jacobs is likely to have meant it, or maybe asserting your examples of psychology working that way more as normative than as positive claims.
I’m not really sure what you mean here, but I can address what you say below. I’m not sure if it’s related?
“felt foolish” together with the consequences looks like a description of an alief-based and alief-affecting social feedback mechanism. How safe is it for individuals to unilaterally train themselves out of such mechanisms?
Depends on how you go about it and what type of risk you’re trying to avoid. When I first started playing with this stuff I taught someone how to “turn off” pain, and in her infinite wisdom she used this new ability to make it easier to be stubborn and run on a sprained ankle. There’s no foolproof solution to make this never happen (in my infinite wisdom I’ve done similar things even with the pain), but the way I go about it now is explicitly mindful of the risks and uses that to get more reliable results. With the swelling, for example, part of my indignant reaction was “it doesn’t have to swell up, I just won’t move it”.
When you’ve seen something happen with your own eyes multiple times, I think that’s beyond the level where you should be foolish for thinking that it might be possible. When you see that the thing that is stopping other people from doing it too is ignorance of the possibility rather than an objection that it shouldn’t be done, then “thinking it through and making your reasoned best guess” isn’t going to be right all the time, but according to your own best guess it will be right more often than the alternative.
Or: individual coherence and social cohesion seem to be at odds often enough for that to be a way for “not-winning due to being too coherent” to sneak in through crazy backdoors in the environment, absent unbounded handling-of-detachment resources which are not in evidence and at some point may be unimplementable within human bounds.
It seems that this bit is your main concern?
It can be a real concern. More than once I’ve had people express concern about how it has become harder to relate with their old friends after spending a lot of time with me. It’s not because of stuff like “I can consciously prevent a lot of swelling, and they don’t know how to engage with that” but rather because stuff like “it’s hard to be supportive of what I now see as clearly bad behavior that attempt to shirk reality to protect feelings and inevitably ends up hurting everyone involved”. In my experience, it’s a consequence of being able to see the problems in the group before being able to see what to do about it.
I don’t seem to have that problem anymore, and I think it’s because of the thought that I’ve put into figuring out how to actually change how people organize their minds. Saying “here, let me use math and statistics to show you why you’re definitely completely wrong” can work to smash through dumb ideas, but then even when you succeed you’re left with people seeing their old ideas (and therefore the ideas of the rest of their social circle) as “dumb” and hard to relate to. When you say “here, let me empathize and understand where you’re coming from, and then address it by showing how things look to me”, and go out of your way to make their former point of view understandable, then you no longer get this failure mode. On top of that, by showing them how to connect with people who hold very different (and often less well thought out) views than you, it gives them a model to follow that can make connecting with others easier. My friend in the above example, for instance, went from sort of a “socially awkward nerd” type to a someone who can turn that off and be really effective when she puts her mind to it. If someone is depressed and not even his siblings can get him to talk, he’ll still talk to her.
If there’s a group of people you want to be able to relate to effectively, you can’t just dissociate off into your own little world where you give no thought to their perspectives, but neither can you just melt in and let your own perspective become that social consensus, because if you don’t retain enough separation that you can at least have your own thoughts and think about whether they might be better and how best to merge them with the group, then you’re just shirking your leadership responsibilities, and if enough people do this the whole group can become detached from reality and led by whomever wants to command the mob. This doesn’t tend to lead to great things.
I’ve put a few cycles into trying to come up with a better way to point at the thing/model I’m thinking of. (I say “thing/model” because in the domain of social psychology especially, Strange Loops between a phenomenon and people’s models of the phenomenon cause them to not be that cleanly separable. Is there a word for that that I’m missing?) I haven’t gotten through much of it, but in the meantime, I’ve also just noticed that a recent second-level comment by Vaniver on their own “How alienated should you be?” post has description that seems to come from a similar observation/interpretation of the world to the part of mine I’m trying to point at, and the main post goes into more detail. So that may help. I think there is a streak of variants of this idea in LW already, and it’s possible that what I really want to do is go through the archives and find the best-aligned existing posts on the subject to link to…
I think I get the general idea of the thing you and Vaniver are gesturing at, but not what you’re trying to say about it in particular. I think I’m less concerned though, because I don’t see inter agent value differences and the resulting conflict as some fundamental inextricable part of the system.
Perhaps it makes sense to talk about the individual level first. I saw a comment recently where the person making it was sorta mocking the idea of psychological “defense mechanisms”, because “*obviously* evolution wouldn’t select for those who ‘defend’ from threats by sticking their heads in the sand!”—as if the problem of wireheading were as simple as competition between a “gene for wireheading” and a gene against. Evolution is going to select for genes that make people flinch away from injuring themselves with hot stoves. It’s also going to select for people who cauterize their wounds when necessary to keep from bleeding out. Designing an organism that does *both* is not trivial. If sensitivity to pain is too low, you get careless burns. If it’s too high, you get refusal to cauterize. You need *some* mechanism to distinguish between effective flinches and harmful flinches, and a way to enact mostly the former. “Defense mechanisms” arise not out of mysterious propagation of fitness reducing genes, but rather the lack of solution to the hard problem of separating the effective flinches from the ineffective—and sometimes even the easiest solution to these ineffective flinches is hacked together out of more flinches, such as screaming and biting down on a stick when having a wound cauterized, or choosing to take pain killers.
The solution of “simply noticing that the pain from cauterizing a serious bleed isn’t a *bad* thing and therefore not flinching from it” isn’t trivial. It’s *doable*, and to be aspired to, but there’s no such thing as “a gene for wise decisions” that is already “hard coded in DNA”.
Similarly, society is incoherent and fragmented and flinches and cooperates imperfectly. You get petty criminals and cronyism and censorship of thought and expression, and all sorts of terrible stuff. This isn’t proof of some sort of “selection for shittiness” any more than it is to notice individual incoherence and the resulting dysfunction. It’s not that coherence is impossible or undesirable, just that you’re fighting entropy to get there, and succeeding takes work.
The desire to eat marshmallows succeeds more if it can cooperate and willingly lose for five minutes until the second marshmallow comes. The individual succeeds more if they are capable of giving back to others as a means to foster cooperation. Sometimes the system is so dysfunctional that saying “no thanks, I can wait” will get you taken advantage of, and so the individually winning thing is impulsive selfishness. Even then, the guy failing to follow through on promises of second marshmallows likely isn’t winning by disincentivizing cooperation with him, and it’s likely more of a “his desire to not feel pain is winning, so he bleeds” sort of situation. Sometimes the system really is so dysfunctional that not only is it winning to take the first marshmallow, it’s also winning to renege on your promises to give the second. But for every time someone wins by shrinking the total pie and taking a bigger piece, there’s an allocation of the more cooperative pie that would give this would-be-defector more pie while still having more for everyone else too. And whoever can find these alternatives can get themselves more pie.
I don’t see negative sum conflict between the individual and society as *inevitable*, just difficult to avoid. It’s negotiation that is inevitable, and done poorly it brings lossy conflict. When Vaniver talks about society saying “shut up and be a cog”, I see a couple things happening simultaneously to one degree or another. One is a dysfunctional society hurting themselves by wasting individual potential that they could be profiting from, and would love to if only they could see how and implement it. The other is a society functioning more or less as intended and using “shut up and be a cog” as a shit test to filter out the leaders who don’t have what it takes to say “nah, I think I’ll trust myself and win more”, and lead effectively. Just like the burning pain, it’s there for a reason and how to calibrate it so that it gets overridden at only and all the right times is a bit of an empirical balancing act. It’s not perfect as is, but neither is it without function. The incentive for everyone to improve this balancing is still there, and selection on the big scale is for coherence.
And as a result, I don’t really feel myself being pulled between a conflict of “respect societies stupid beliefs/rules” and “care about other people”. I see people as a combination of *wanting* me to pass their shit tests and show them a better replacement for their stupid beliefs/rules, being afraid and unsure of what to do if I succeed, and selfishly trying to shrink the size of the pie so that they can keep what they think will be the bigger piece. As a result, it makes me want to rise to the occasion and help people face new and more accurate beliefs, and also to create common knowledge of defection when it happens and rub their noses in it to make it clear that those who work to make the pie smaller will get less pie. Sometimes it’s more rewarding and higher leverage to run off and gain some momentum by creating and then expanding a small bubble where things actually *work*, but there’s no reason to go from “I can’t yet be effective in the broader community because I can’t yet break out of their ‘cog’ mold for me, so I’m going to focus on the smaller community where I can” to “fuck them all”. There’s still plenty of value in reengaging when capable and pretending there isn’t isn’t that good functional thing we’re striving to do. It’s not like we can *actually* form a bubble and reject the outside world, because the outside world will still bring you pandemics and AI, and from even a selfish perspective there’s plenty of incentive to help things go well for everyone.
[This is condensed and informalized from a much longer and more explicit comment which I’m not sure would have been worth wading through, which still seemed hazy in important ways, and which seemed like it needed me to open more boxes than I have energy for right now. This one still seems hazy, but hopefully it wears it more on its sleeve. I should also declare up front that I have a bunch of weird emotional warping around this topic; hopefully I’m working around enough of it for this to still be useful.]
I think you’re interpreting “this is not how human psychology works” in a noncentral way compared to how Bob Jacobs is likely to have meant it, or maybe asserting your examples of psychology working that way more as normative than as positive claims.
I have a completely different tack in mind: how do we know that the sort of mental maneuvers you describe don’t become harmful in their aggregate effects when too many people do them, or do them without coordinating enough, or something along those lines?
I would like to point out the following:
“felt foolish” together with the consequences looks like a description of an alief-based and alief-affecting social feedback mechanism. How safe is it for individuals to unilaterally train themselves out of such mechanisms? Some detachment from it seems to be part of emotional maturity, but that’s coupled with a lot of other mediating material. Further detachment seems to be part of various spiritual traditions—also coupled with even more mediating material. That’s not very promising for any implied “there’s no such thing as too much”.
More specifically, I would like to consider the possibility that many of the sort of false aliefs you’re talking about act more like restraining bolts imposed by the social cohesion subunit of a human mind, “because” humans are not safe under amplification with regard to social values. (And notably, “hypercompetent individuals are good for society” is by no means a universal more.)
Or: individual coherence and social cohesion seem to be at odds often enough for that to be a way for “not-winning due to being too coherent” to sneak in through crazy backdoors in the environment, absent unbounded handling-of-detachment resources which are not in evidence and at some point may be unimplementable within human bounds.
[… this was going to be more-edited, but I’ve accidentally hit Submit, and I don’t want to do too much frantic editing, so I’ve just cleaned up a few pieces. I think this is still just-about worth enough to leave up; I’ll try to come back to it later if it’s deemed worth talking about.]
This is a really cool declaration. It doesn’t bleed through in any obvious way, but thanks for letting me know and I’ll try to be cautious of what I say/how I say them. Lemme know if I’m bumping into anything or if there’s anything I could be doing differently to better accommodate.
I’m not really sure what you mean here, but I can address what you say below. I’m not sure if it’s related?
Depends on how you go about it and what type of risk you’re trying to avoid. When I first started playing with this stuff I taught someone how to “turn off” pain, and in her infinite wisdom she used this new ability to make it easier to be stubborn and run on a sprained ankle. There’s no foolproof solution to make this never happen (in my infinite wisdom I’ve done similar things even with the pain), but the way I go about it now is explicitly mindful of the risks and uses that to get more reliable results. With the swelling, for example, part of my indignant reaction was “it doesn’t have to swell up, I just won’t move it”.
When you’ve seen something happen with your own eyes multiple times, I think that’s beyond the level where you should be foolish for thinking that it might be possible. When you see that the thing that is stopping other people from doing it too is ignorance of the possibility rather than an objection that it shouldn’t be done, then “thinking it through and making your reasoned best guess” isn’t going to be right all the time, but according to your own best guess it will be right more often than the alternative.
It seems that this bit is your main concern?
It can be a real concern. More than once I’ve had people express concern about how it has become harder to relate with their old friends after spending a lot of time with me. It’s not because of stuff like “I can consciously prevent a lot of swelling, and they don’t know how to engage with that” but rather because stuff like “it’s hard to be supportive of what I now see as clearly bad behavior that attempt to shirk reality to protect feelings and inevitably ends up hurting everyone involved”. In my experience, it’s a consequence of being able to see the problems in the group before being able to see what to do about it.
I don’t seem to have that problem anymore, and I think it’s because of the thought that I’ve put into figuring out how to actually change how people organize their minds. Saying “here, let me use math and statistics to show you why you’re definitely completely wrong” can work to smash through dumb ideas, but then even when you succeed you’re left with people seeing their old ideas (and therefore the ideas of the rest of their social circle) as “dumb” and hard to relate to. When you say “here, let me empathize and understand where you’re coming from, and then address it by showing how things look to me”, and go out of your way to make their former point of view understandable, then you no longer get this failure mode. On top of that, by showing them how to connect with people who hold very different (and often less well thought out) views than you, it gives them a model to follow that can make connecting with others easier. My friend in the above example, for instance, went from sort of a “socially awkward nerd” type to a someone who can turn that off and be really effective when she puts her mind to it. If someone is depressed and not even his siblings can get him to talk, he’ll still talk to her.
If there’s a group of people you want to be able to relate to effectively, you can’t just dissociate off into your own little world where you give no thought to their perspectives, but neither can you just melt in and let your own perspective become that social consensus, because if you don’t retain enough separation that you can at least have your own thoughts and think about whether they might be better and how best to merge them with the group, then you’re just shirking your leadership responsibilities, and if enough people do this the whole group can become detached from reality and led by whomever wants to command the mob. This doesn’t tend to lead to great things.
Does that address what you’re saying?
I’ve put a few cycles into trying to come up with a better way to point at the thing/model I’m thinking of. (I say “thing/model” because in the domain of social psychology especially, Strange Loops between a phenomenon and people’s models of the phenomenon cause them to not be that cleanly separable. Is there a word for that that I’m missing?) I haven’t gotten through much of it, but in the meantime, I’ve also just noticed that a recent second-level comment by Vaniver on their own “How alienated should you be?” post has description that seems to come from a similar observation/interpretation of the world to the part of mine I’m trying to point at, and the main post goes into more detail. So that may help. I think there is a streak of variants of this idea in LW already, and it’s possible that what I really want to do is go through the archives and find the best-aligned existing posts on the subject to link to…
I think I get the general idea of the thing you and Vaniver are gesturing at, but not what you’re trying to say about it in particular. I think I’m less concerned though, because I don’t see inter agent value differences and the resulting conflict as some fundamental inextricable part of the system.
Perhaps it makes sense to talk about the individual level first. I saw a comment recently where the person making it was sorta mocking the idea of psychological “defense mechanisms”, because “*obviously* evolution wouldn’t select for those who ‘defend’ from threats by sticking their heads in the sand!”—as if the problem of wireheading were as simple as competition between a “gene for wireheading” and a gene against. Evolution is going to select for genes that make people flinch away from injuring themselves with hot stoves. It’s also going to select for people who cauterize their wounds when necessary to keep from bleeding out. Designing an organism that does *both* is not trivial. If sensitivity to pain is too low, you get careless burns. If it’s too high, you get refusal to cauterize. You need *some* mechanism to distinguish between effective flinches and harmful flinches, and a way to enact mostly the former. “Defense mechanisms” arise not out of mysterious propagation of fitness reducing genes, but rather the lack of solution to the hard problem of separating the effective flinches from the ineffective—and sometimes even the easiest solution to these ineffective flinches is hacked together out of more flinches, such as screaming and biting down on a stick when having a wound cauterized, or choosing to take pain killers.
The solution of “simply noticing that the pain from cauterizing a serious bleed isn’t a *bad* thing and therefore not flinching from it” isn’t trivial. It’s *doable*, and to be aspired to, but there’s no such thing as “a gene for wise decisions” that is already “hard coded in DNA”.
Similarly, society is incoherent and fragmented and flinches and cooperates imperfectly. You get petty criminals and cronyism and censorship of thought and expression, and all sorts of terrible stuff. This isn’t proof of some sort of “selection for shittiness” any more than it is to notice individual incoherence and the resulting dysfunction. It’s not that coherence is impossible or undesirable, just that you’re fighting entropy to get there, and succeeding takes work.
The desire to eat marshmallows succeeds more if it can cooperate and willingly lose for five minutes until the second marshmallow comes. The individual succeeds more if they are capable of giving back to others as a means to foster cooperation. Sometimes the system is so dysfunctional that saying “no thanks, I can wait” will get you taken advantage of, and so the individually winning thing is impulsive selfishness. Even then, the guy failing to follow through on promises of second marshmallows likely isn’t winning by disincentivizing cooperation with him, and it’s likely more of a “his desire to not feel pain is winning, so he bleeds” sort of situation. Sometimes the system really is so dysfunctional that not only is it winning to take the first marshmallow, it’s also winning to renege on your promises to give the second. But for every time someone wins by shrinking the total pie and taking a bigger piece, there’s an allocation of the more cooperative pie that would give this would-be-defector more pie while still having more for everyone else too. And whoever can find these alternatives can get themselves more pie.
I don’t see negative sum conflict between the individual and society as *inevitable*, just difficult to avoid. It’s negotiation that is inevitable, and done poorly it brings lossy conflict. When Vaniver talks about society saying “shut up and be a cog”, I see a couple things happening simultaneously to one degree or another. One is a dysfunctional society hurting themselves by wasting individual potential that they could be profiting from, and would love to if only they could see how and implement it. The other is a society functioning more or less as intended and using “shut up and be a cog” as a shit test to filter out the leaders who don’t have what it takes to say “nah, I think I’ll trust myself and win more”, and lead effectively. Just like the burning pain, it’s there for a reason and how to calibrate it so that it gets overridden at only and all the right times is a bit of an empirical balancing act. It’s not perfect as is, but neither is it without function. The incentive for everyone to improve this balancing is still there, and selection on the big scale is for coherence.
And as a result, I don’t really feel myself being pulled between a conflict of “respect societies stupid beliefs/rules” and “care about other people”. I see people as a combination of *wanting* me to pass their shit tests and show them a better replacement for their stupid beliefs/rules, being afraid and unsure of what to do if I succeed, and selfishly trying to shrink the size of the pie so that they can keep what they think will be the bigger piece. As a result, it makes me want to rise to the occasion and help people face new and more accurate beliefs, and also to create common knowledge of defection when it happens and rub their noses in it to make it clear that those who work to make the pie smaller will get less pie. Sometimes it’s more rewarding and higher leverage to run off and gain some momentum by creating and then expanding a small bubble where things actually *work*, but there’s no reason to go from “I can’t yet be effective in the broader community because I can’t yet break out of their ‘cog’ mold for me, so I’m going to focus on the smaller community where I can” to “fuck them all”. There’s still plenty of value in reengaging when capable and pretending there isn’t isn’t that good functional thing we’re striving to do. It’s not like we can *actually* form a bubble and reject the outside world, because the outside world will still bring you pandemics and AI, and from even a selfish perspective there’s plenty of incentive to help things go well for everyone.