I think I get the general idea of the thing you and Vaniver are gesturing at, but not what you’re trying to say about it in particular. I think I’m less concerned though, because I don’t see inter agent value differences and the resulting conflict as some fundamental inextricable part of the system.
Perhaps it makes sense to talk about the individual level first. I saw a comment recently where the person making it was sorta mocking the idea of psychological “defense mechanisms”, because “*obviously* evolution wouldn’t select for those who ‘defend’ from threats by sticking their heads in the sand!”—as if the problem of wireheading were as simple as competition between a “gene for wireheading” and a gene against. Evolution is going to select for genes that make people flinch away from injuring themselves with hot stoves. It’s also going to select for people who cauterize their wounds when necessary to keep from bleeding out. Designing an organism that does *both* is not trivial. If sensitivity to pain is too low, you get careless burns. If it’s too high, you get refusal to cauterize. You need *some* mechanism to distinguish between effective flinches and harmful flinches, and a way to enact mostly the former. “Defense mechanisms” arise not out of mysterious propagation of fitness reducing genes, but rather the lack of solution to the hard problem of separating the effective flinches from the ineffective—and sometimes even the easiest solution to these ineffective flinches is hacked together out of more flinches, such as screaming and biting down on a stick when having a wound cauterized, or choosing to take pain killers.
The solution of “simply noticing that the pain from cauterizing a serious bleed isn’t a *bad* thing and therefore not flinching from it” isn’t trivial. It’s *doable*, and to be aspired to, but there’s no such thing as “a gene for wise decisions” that is already “hard coded in DNA”.
Similarly, society is incoherent and fragmented and flinches and cooperates imperfectly. You get petty criminals and cronyism and censorship of thought and expression, and all sorts of terrible stuff. This isn’t proof of some sort of “selection for shittiness” any more than it is to notice individual incoherence and the resulting dysfunction. It’s not that coherence is impossible or undesirable, just that you’re fighting entropy to get there, and succeeding takes work.
The desire to eat marshmallows succeeds more if it can cooperate and willingly lose for five minutes until the second marshmallow comes. The individual succeeds more if they are capable of giving back to others as a means to foster cooperation. Sometimes the system is so dysfunctional that saying “no thanks, I can wait” will get you taken advantage of, and so the individually winning thing is impulsive selfishness. Even then, the guy failing to follow through on promises of second marshmallows likely isn’t winning by disincentivizing cooperation with him, and it’s likely more of a “his desire to not feel pain is winning, so he bleeds” sort of situation. Sometimes the system really is so dysfunctional that not only is it winning to take the first marshmallow, it’s also winning to renege on your promises to give the second. But for every time someone wins by shrinking the total pie and taking a bigger piece, there’s an allocation of the more cooperative pie that would give this would-be-defector more pie while still having more for everyone else too. And whoever can find these alternatives can get themselves more pie.
I don’t see negative sum conflict between the individual and society as *inevitable*, just difficult to avoid. It’s negotiation that is inevitable, and done poorly it brings lossy conflict. When Vaniver talks about society saying “shut up and be a cog”, I see a couple things happening simultaneously to one degree or another. One is a dysfunctional society hurting themselves by wasting individual potential that they could be profiting from, and would love to if only they could see how and implement it. The other is a society functioning more or less as intended and using “shut up and be a cog” as a shit test to filter out the leaders who don’t have what it takes to say “nah, I think I’ll trust myself and win more”, and lead effectively. Just like the burning pain, it’s there for a reason and how to calibrate it so that it gets overridden at only and all the right times is a bit of an empirical balancing act. It’s not perfect as is, but neither is it without function. The incentive for everyone to improve this balancing is still there, and selection on the big scale is for coherence.
And as a result, I don’t really feel myself being pulled between a conflict of “respect societies stupid beliefs/rules” and “care about other people”. I see people as a combination of *wanting* me to pass their shit tests and show them a better replacement for their stupid beliefs/rules, being afraid and unsure of what to do if I succeed, and selfishly trying to shrink the size of the pie so that they can keep what they think will be the bigger piece. As a result, it makes me want to rise to the occasion and help people face new and more accurate beliefs, and also to create common knowledge of defection when it happens and rub their noses in it to make it clear that those who work to make the pie smaller will get less pie. Sometimes it’s more rewarding and higher leverage to run off and gain some momentum by creating and then expanding a small bubble where things actually *work*, but there’s no reason to go from “I can’t yet be effective in the broader community because I can’t yet break out of their ‘cog’ mold for me, so I’m going to focus on the smaller community where I can” to “fuck them all”. There’s still plenty of value in reengaging when capable and pretending there isn’t isn’t that good functional thing we’re striving to do. It’s not like we can *actually* form a bubble and reject the outside world, because the outside world will still bring you pandemics and AI, and from even a selfish perspective there’s plenty of incentive to help things go well for everyone.
I think I get the general idea of the thing you and Vaniver are gesturing at, but not what you’re trying to say about it in particular. I think I’m less concerned though, because I don’t see inter agent value differences and the resulting conflict as some fundamental inextricable part of the system.
Perhaps it makes sense to talk about the individual level first. I saw a comment recently where the person making it was sorta mocking the idea of psychological “defense mechanisms”, because “*obviously* evolution wouldn’t select for those who ‘defend’ from threats by sticking their heads in the sand!”—as if the problem of wireheading were as simple as competition between a “gene for wireheading” and a gene against. Evolution is going to select for genes that make people flinch away from injuring themselves with hot stoves. It’s also going to select for people who cauterize their wounds when necessary to keep from bleeding out. Designing an organism that does *both* is not trivial. If sensitivity to pain is too low, you get careless burns. If it’s too high, you get refusal to cauterize. You need *some* mechanism to distinguish between effective flinches and harmful flinches, and a way to enact mostly the former. “Defense mechanisms” arise not out of mysterious propagation of fitness reducing genes, but rather the lack of solution to the hard problem of separating the effective flinches from the ineffective—and sometimes even the easiest solution to these ineffective flinches is hacked together out of more flinches, such as screaming and biting down on a stick when having a wound cauterized, or choosing to take pain killers.
The solution of “simply noticing that the pain from cauterizing a serious bleed isn’t a *bad* thing and therefore not flinching from it” isn’t trivial. It’s *doable*, and to be aspired to, but there’s no such thing as “a gene for wise decisions” that is already “hard coded in DNA”.
Similarly, society is incoherent and fragmented and flinches and cooperates imperfectly. You get petty criminals and cronyism and censorship of thought and expression, and all sorts of terrible stuff. This isn’t proof of some sort of “selection for shittiness” any more than it is to notice individual incoherence and the resulting dysfunction. It’s not that coherence is impossible or undesirable, just that you’re fighting entropy to get there, and succeeding takes work.
The desire to eat marshmallows succeeds more if it can cooperate and willingly lose for five minutes until the second marshmallow comes. The individual succeeds more if they are capable of giving back to others as a means to foster cooperation. Sometimes the system is so dysfunctional that saying “no thanks, I can wait” will get you taken advantage of, and so the individually winning thing is impulsive selfishness. Even then, the guy failing to follow through on promises of second marshmallows likely isn’t winning by disincentivizing cooperation with him, and it’s likely more of a “his desire to not feel pain is winning, so he bleeds” sort of situation. Sometimes the system really is so dysfunctional that not only is it winning to take the first marshmallow, it’s also winning to renege on your promises to give the second. But for every time someone wins by shrinking the total pie and taking a bigger piece, there’s an allocation of the more cooperative pie that would give this would-be-defector more pie while still having more for everyone else too. And whoever can find these alternatives can get themselves more pie.
I don’t see negative sum conflict between the individual and society as *inevitable*, just difficult to avoid. It’s negotiation that is inevitable, and done poorly it brings lossy conflict. When Vaniver talks about society saying “shut up and be a cog”, I see a couple things happening simultaneously to one degree or another. One is a dysfunctional society hurting themselves by wasting individual potential that they could be profiting from, and would love to if only they could see how and implement it. The other is a society functioning more or less as intended and using “shut up and be a cog” as a shit test to filter out the leaders who don’t have what it takes to say “nah, I think I’ll trust myself and win more”, and lead effectively. Just like the burning pain, it’s there for a reason and how to calibrate it so that it gets overridden at only and all the right times is a bit of an empirical balancing act. It’s not perfect as is, but neither is it without function. The incentive for everyone to improve this balancing is still there, and selection on the big scale is for coherence.
And as a result, I don’t really feel myself being pulled between a conflict of “respect societies stupid beliefs/rules” and “care about other people”. I see people as a combination of *wanting* me to pass their shit tests and show them a better replacement for their stupid beliefs/rules, being afraid and unsure of what to do if I succeed, and selfishly trying to shrink the size of the pie so that they can keep what they think will be the bigger piece. As a result, it makes me want to rise to the occasion and help people face new and more accurate beliefs, and also to create common knowledge of defection when it happens and rub their noses in it to make it clear that those who work to make the pie smaller will get less pie. Sometimes it’s more rewarding and higher leverage to run off and gain some momentum by creating and then expanding a small bubble where things actually *work*, but there’s no reason to go from “I can’t yet be effective in the broader community because I can’t yet break out of their ‘cog’ mold for me, so I’m going to focus on the smaller community where I can” to “fuck them all”. There’s still plenty of value in reengaging when capable and pretending there isn’t isn’t that good functional thing we’re striving to do. It’s not like we can *actually* form a bubble and reject the outside world, because the outside world will still bring you pandemics and AI, and from even a selfish perspective there’s plenty of incentive to help things go well for everyone.